Aligning Text-to-Image Diffusion Models to Human Preference by Classification

We greatly appreciate the reviewers' hard work and thank you for your valuable comments. We will address each concerns in the following.

---

Q: This works only focuses on DPO-based variants, and neglects many existing work in the domain. On the one hand, we have inference-time methods, which attempt to align T2I models by steering the generative paths through optimization of the attention layers of the score network

A: We believe this concern arises from a misunderstanding of the comparison scope. As discussed in "Large Language Models Post-training: Surveying Techniques from Alignment to Reasoning," generative modeling methods can be broadly categorized into pre-training and post-training. Pre-training focuses on building general-purpose models from large datasets, while post-training adapts these models to specific downstream tasks.

Due to their distinct goals, evaluation strategies for these two categories differ. Pre-training methods are typically compared across architecture types, datasets, or training paradigms. In contrast, post-training methods—like ours—should be evaluated using the same foundation model and training setup, with only the loss objective varying to ensure a fair comparison.

Most works cited by Reviewer RW4E, except [2,3], focus on pre-training. For instance, [6] proposes an autoregressive model as a new foundation model and compares its generative capabilities with other pre-training approaches. By contrast, our ABC loss is a post-training method applied to a fixed foundation model. Analogously, comparing a fine-tuned GPT-3 with GPT-4o would not inform the quality of the fine-tuning method applied to GPT-3.

Additionally, [2] is an inference-time method without training, and [3] is self-supervised, whereas DPO-based methods—including ours—require supervised training. According to our tests conducted during the rebuttal period, [2] and [3] consistently underperform all DPO-based methods on the same base model. Given their fundamentally different setups, comparing their raw performance to ours is not meaningful for assessing the quality of our approach.

---

Q: How would your approach perform with more natural prompts than the ones you used in your evaluation? Would it be possible to consider using T2I-CompBench++ for your evaluation?

A: This is a valuable suggestion. However, implementing it poses several technical challenges. T2I-CompBench++ is specifically designed to test compositional generation—handling multiple objects, attributes, and relationships—which makes it more suited for evaluating foundation models developed through pre-training.

In contrast, post-training methods like ours require a well-defined downstream task and a corresponding dataset. The standard procedure involves fine-tuning a pre-trained foundation model using a specific loss and evaluating the resulting checkpoint against either the original model or other post-trained variants.

In our case, we currently lack a task-specific dataset aligned with the compositional goals of T2I-CompBench++. Therefore, although T2I-CompBench++ is a promising benchmark, it is not directly applicable under our current post-training setup. Currently, we make a comporise which take the checkpoints trained on Pick-a-Pic to conduct this evaultion. Due to the maximum length limitation, we have included the corresponding data in our response to Reviewer qFcD.

---

Q: If recent autoregressive generative models are becoming the new standard for image generation, how relevant do you consider alignment approaches for diffusion models in general, and DPO-based fine-tuning approaches in particular, will be?

A: Both autoregressive generative models and diffusion models are promising approaches for image generation. The DPO (Differentiable Prompt Optimization) method was initially proposed for aligning autoregressive language models, and Diffusion-DPO adapts this technique for use with diffusion models. The primary contribution of our paper is that we are the first to demonstrate that the alignment task can be transformed into a classification task. Reviewer RW4E can verify that this discovery does not depend on the model type. Both autoregressive and diffusion models can benefit from this finding.

Since Reviewer RW4E has a particular interest in the application of the ABC loss to autoregressive generative models, we believe it would be a great idea to prepare a separate paper discussing how to apply the ABC loss to autoregressive models.

---

Q: What are the key advantages of DPO-based alignment methods such as ABC when compared to other fine-tuning methods from the literature (e.g., you cite DPOK as reference [14], but do not even refer to it in the main), or other inference-time methods?

A: Both DPO and SFT methods have demonstrated their value in training large language models. SFT-based methods, such as DPOK, require a reward model to fine-tune the diffusion model. However, training a good reward model requires additional time and dataset resources. In contrast, DPO-based methods bypass the need for a reward model when fine-tuning the diffusion model.

Regarding inference-time methods, the running time for inference is significantly longer compared to SFT and DPO-based methods. In general, these three approaches represent different pathways toward achieving better results, and at this stage, it is difficult to predict which will ultimately prevail. This paper focuses solely on demonstrating that the alignment task can be transformed into a classification task through our ABC loss. We do not address whether DPO-based methods will outperform SFT and inference-based methods, as this is beyond the scope of our current work.

---

Q: In sec 4.1, line 116, you rely on a pretty strong assumption. I realize your work is not the only one requiring such an assumption, but it would be interesting to hear your opinion on how a DPO-based framework such as ABC could cope with the more realistic case of multiple images being aligned to a single prompt.

A: Thank you for your question. We acknowledge the limitation of this assumption. However, we could not avoid it in the current environment for two reasons:

1. The current preference datasets are built on this assumption.
2. The evaluation networks are also designed based on this assumption, as they are trained on the preference dataset.

Without new preference datasets and evaluation criteria, it is difficult for methods that handle multiple preferences to demonstrate their superiority and gain acceptance in top conferences like NeurIPS.

Since we have established the connection between alignment and classification, the assumption that "each text prompt corresponds to a single aligned image" can be interpreted as a multi-class classification task, where each sample is assumed to belong to only one class. In contrast, Reviewer RW4E's preference for a scenario where one prompt corresponds to multiple images can be interpreted as a multi-label classification task, where each sample may belong to multiple classes. In the future, we plan to explore the multi-label classification perspective to address this issue.

---

Q: In Theorem 2, lines 142-143, this assumption is pretty strong. Can you comment on what would it be necessary to do if we relax such an assumption?

A: We apologize for any confusion caused to Reviewer RW4E. In our paper, Theorem 2 is used to illustrate that classification performance directly determines alignment performance. To demonstrate this, we assume that we have a total of $N$ text prompts $\mathrm{y}_i$​, each corresponding to an aligned image $\pmb{x}\_{\mathrm{y}\_i}$​​. We believe it is reasonable to assume that the prior distribution for both the prompt $\mathrm{y}_i$​ and the image $\pmb{x}\_{\mathrm{y}\_i}$​​ follows a uniform distribution. If we were to assume otherwise, we would have to acknowledge that certain labels and images are inherently more special than others. However, we currently do not have a compelling reason to support this assumption.

---

Q: The proposed method, in principle, can be applied to other generative models, such as rectified flows. The evaluation is based on “older” models such as SD1.5 and SDXL, whereas the current best practice is to use FLUX or SD3, which use a different generative modeling paradigm (still related to diffusion somehow), and that are known to be more aligned than SD1.5 and SDXL.

A: Applying DPO-like methods to flow models is not an easy task. Flow models rely on a deterministic generative process based on Ordinary Differential Equations (ODEs), meaning they cannot sample stochastically during inference. In contrast, DPO relies on stochastic sampling to explore the environment, learning by trying different actions and improving based on rewards. This need for stochasticity conflicts with the deterministic nature of flow matching models. To apply DPO to flow models, one would need to conduct an ODE-to-SDE conversion, which transforms a deterministic ODE into an equivalent Stochastic Differential Equation (SDE) that matches the original model’s marginal distribution at all timesteps, thus enabling statistical sampling.

We recently discovered a concurrent work, "Flow-GRPO: Training Flow Matching Models via Online RL", which claims to be the first method to apply GRPO to flow models in order to address this problem. By employing Flow-GRPO, our approach could potentially be applied to models like FLUX or SD3. However, this paper was released on May 8, 2025, on arXiv, and the code was released on May 15, 2025. As a result, we were not aware of this work before submitting our paper.

---

Q: Could you report absolute metrics rather than percentage of wins in Table 1?

A: Thank you for the suggestion. Due to length limits, we report the absolute metrics in Table 2 of our response to Reviewer PucN, including appended results for DPOK and D3PO.

Dear Reviewer RW4E,

We sincerely apologize for the delayed response, as we needed additional time to carefully consider how to conduct a fair comparison. We also acknowledge that our initial rebuttal did not provide sufficient evidence to fully address your concerns. We would like to take this opportunity to clarify why the comparisons you suggested were not included. Our decision was based on three main reasons: (1) the referenced methods belong to different methodological categories; (2) cross-category comparisons raise concerns regarding fairness and interpretability; and (3) space limitations prevented us from including additional results.

For the first reason, we note that A&E [1] is a training-free approach designed to improve inference-time performance without modifying model weights. In contrast, methods [3], [4], and ours are fine-tuning-based and rely on data-driven checkpoint updates. Even among fine-tuning methods, the assumptions differ significantly: MITUNE [3] generates pseudo-labeled data automatically using the base model, while method [4] assumes access to human-annotated data and sufficient resources to train a reward model, which is then used to guide the training of the generation model. In resource-constrained environments—especially those with limited GPU memory—method [4] becomes impractical, making DPO-style methods like ours the only viable option. Furthermore, the training data used also differs: DPO is trained on preference triplets (prompt, preferred image, less preferred image), whereas [4] uses prompt-image pairs without explicit preference signals. These differences in both input requirements and training objectives further complicate direct comparison.

For the second reason, even when technically possible, fair comparisons remain challenging. Fine-tuning methods yield new checkpoints that are evaluated under a shared sampling strategy, while training-free methods like A&E [1] modify the sampling process itself, operating on a fixed checkpoint. As such, the natural baselines for A&E [1] are other sampling strategies (e.g., DDPM, DDIM), not fine-tuned models.

To aid your evaluation, we report the performance of our fine-tuned checkpoint under both DDPM and A&E [1]’s sampling strategy, along with the performance of the original (unmodified) SDXL checkpoint under the same conditions. Note that A&E [1] requires manually specified subject tokens to guide inference—information that standard DDPM cannot leverage—further limiting the validity of direct comparisons.

We also compare our fine-tuned model (ABC) against MITUNE [3] under the same DDPM sampling protocol:

We further provide a comparison on T2I-CompBench++, showing that ABC consistently outperforms MITUNE across most dimensions:

Overall, our method consistently outperforms MITUNE across a wide range of metrics. We attribute this to two main factors: (1) our dataset includes human-annotated preferences, whereas MITUNE relies on synthetic labels generated by the base model; and (2) our proposed ABC loss is more effective than the pseudo-labeling strategy employed by MITUNE.

We acknowledge that on T2I-CompBench++, ABC underperforms MITUNE on two specific scores. We attribute this to differences in the training datasets: the dataset used for ABC was not specifically designed for the T2I-CompBench++ task, whereas the dataset used in MITUNE was optimized to maximize point-wise mutual information—that is, to emphasize the difference between conditional and unconditional matching scores. While this design benefits pixel-level fidelity, it may lead to weaker semantic representation and alignment, which in turn affects performance on certain benchmarks.

Please feel free to reach out if you have any further questions. We sincerely appreciate your thoughtful feedback.

Best regards,

Authors

We greatly appreciate the reviewers' hard work and thank you for your valuable comments. We will address each concerns in the following.

---

Q: Importance of data augmentation is unclear.

A: We apologize for the confusion regarding data augmentation in the preference data. Each dataset item is a triplet $(\mathrm{y}, \pmb{x}^+\_{\mathrm{y}}, \pmb{x}^-\_{\mathrm{y}})$, where $\mathrm{y}$ denotes the prompt, and $\pmb{x}^+\_{\mathrm{y}}$ and $\pmb{x}^-\_{\mathrm{y}}$ represent the positive and negative images corresponding to the prompt $\mathrm{y}$, respectively.

In this paper, we propose replacing the alignment loss (DPO loss) with a classification loss (ABC loss). Since the classification loss requires a label for each image, we need to transform the triplet $(\mathrm{y}, \pmb{x}^+\_{\mathrm{y}}, \pmb{x}^-\_{\mathrm{y}})$, where the two images share the same label $\mathrm{y}$, into a form where each image has its own label. Specifically, we define $\mathrm{y}^+$ as the original prompt $\mathrm{y}$ and construct $\mathrm{y}^-$ by appending “The image that aligns less with human preferences” to $\mathrm{y}$. This reformulation transforms each preference tuple $(\mathrm{y}, \pmb{x}^+_{\mathrm{y}}, \pmb{x}^-\_{\mathrm{y}})$ into two supervised examples: $(\mathrm{y}^+, \pmb{x}^+\_{\mathrm{y}})$ and $(\mathrm{y}^-, \pmb{x}^-\_{\mathrm{y}})$. This is clarified in lines 173 to 176 of the paper.

---

Q: Real-world preference data is quite noisy and its unclear why a formulation dependent on such strong assumptions is able to outperform other methods like Diffusion-KTO. Additionally, the data augmentation strategy is questionable as even though an image is less-preferred in a single triplet it may generally be more preferred on aggregate. It is unclear why this strategy works with noisy real-world data.

A: Yes, this is a great question. Preference data is inherently noisy, making it difficult to establish a clear principle for determining whether one image is preferred over another. Current diffusion alignment methods do not take this issue into account. To address this problem, we believe it is necessary to build a new dataset that provides preference comparison results across different dimensions. However, current alignment methods are typically trained on datasets like Pick-a-Pic, which assumes that one image is better than another with respect to the given prompt.

We do not directly address this issue in the current paper, but we present a promising approach to tackle it. Specifically, we show that the alignment task can be transformed into a classification problem, and classification in noisy environments has been extensively studied. It may therefore be possible to adapt existing methods from noisy classification to address the noisy alignment problem.

As for the noise-resistant properties of our method, they are easy to interpret. The well-known ImageNet classification dataset contains incorrect labels, yet research has shown that simply using cross-entropy loss to train the network still yields state-of-the-art performance, without the need for special handling of mislabeled data. We attribute this noise-resistant property to the classification loss. In this paper, we transform the preference data into a classification form. As a result, even if the data is noisy, the network retains the ability to resist the effects of noise.

---

Q: Statistical significance of results. To fully validate the performance of ABC, statistical significance of these results should be provided, e.g. via confidence intervals.

A: Thank you for raising this important point. We agree that providing statistical significance is important for validating the reported win rates. While our experiments followed the standard protocols used in prior work, we now include win rates along with variance intervals.

Specifically, we compute these intervals by discarding the top 5% of deviations from the mean, resulting in a range that captures 95% of the scores. This provides a robust estimate of variability and offers a more informative view of our method’s performance.

Table 1. Win rate on PartiPrompts (P) and HPS (H) benchmarks for SD1.5 and SDXL-based models.

---

Q: Lack of user study details. The details of the user study are largely omitted in the paper, outside of 2 lines in the checklist. Was this a blind user study? How many annotators were assigned per comparison?

A: Thank you for your question. We conducted a user study to compare the proposed ABC method with several baseline approaches. Specifically, we randomly sampled 100 prompts from the PartiPrompts dataset and another 100 prompts from the HPSv2 benchmark. For each prompt, we generated five images using five different methods.

Participants were shown five images per prompt (one from each method) and asked to answer three questions:

1. Which image is your overall preferred choice?
2. Which image is more visually attractive?
3. Which image better matches the text description?

To minimize position bias, the order of images was randomized for each prompt. Each method’s final score was computed as a weighted sum of its win rates under the three criteria, with weights of 30% for general preference, 30% for visual appeal, and 40% for prompt alignment.

The study was conducted as a blind evaluation. Annotators were not informed about which method generated each image. We recruited participants from our research group, comprising approximately 100 students, and collected a total of 82 valid responses.

We hope this clarification provides a more complete picture of our user study design and evaluation protocol.

---

Q: Can the authors report the average score (within some confidence interval) to help quantify the improvement of ABC over other methods?

A: We thank the reviewer for the valuable suggestion. In addition to win-rate comparisons, we now report the absolute scores along with the setting in the Table 1.

Table 2. Absolute scores on the PartiPrompts (P) and HPS (H) benchmarks for SD1.5 and SDXL-based models.

We greatly appreciate the reviewers' hard work and thank you for your valuable comments. We will address each concerns in the following.

---

Q: The paper is not easy to read, particularly the theorem sections. It would be helpful if the authors could provide informal interpretations or explanations in simpler terms to improve accessibility.

A: We apologize that the theorem sections may have made the paper difficult to follow. We briefly interpret them here. Theorem 1 proves that the AM-Softmax loss is upper bounded by the Diffusion-DPO loss. In other words, minimizing the Diffusion-DPO loss for better alignment will also reduce the AM-Softmax loss, leading to improved classification performance. Simply put, better alignment leads to better classification. Conversely, Theorem 2 shows that, under certain conditions, improved classification leads to better alignment. Together, these two theorems reveal a strong connection between classification and alignment, forming the theoretical foundation of our approach, which replaces the DPO loss with the ABC loss for alignment tasks. We first establish a connection between alignment and classification. This idea is conceptually similar to the ICLR 2025 paper "On a Connection Between Imitation Learning and RLHF," which draws a connection between Reinforcement Learning from Human Feedback (RLHF) and Imitation Learning.

---

Q: Due to the large number of notations, the paper is difficult to follow. Including a table summarizing all notations and their meanings would greatly enhance readability.

A: We apologize for the oversight in summarizing the notations. Below is a table that summarizes all the notations used in the paper:

PS：

1. $x_0$ can refer to any clean image, while $x_y^+$, $x_y^-$ refer specifically to images aligned/misaligned with a prompt $y$.
2. For simplicity, the timestep $t$ is sometimes omitted in notation when it is clear from context, eg. $\epsilon_\mathrm{ali}(x,y)$.
3. For simplicity, the prompt $y$ is sometimes omitted in notation , eg. $\iota({x}^-, {x}^+, {y})$.

---

Q: Comparisons of training time and analyses of the effect of dataset size are important. More extensive ablation studies exploring different aspects could further strengthen the paper.

A: Thank you for the suggestion. We believe it will significantly improve our paper. Due to the page limitations of NeurIPS, we initially restricted the training dataset to Pick-a-Pic v2, published in "Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation," which is commonly adopted by current diffusion alignment methods. Additionally, given the current research focus on designing various alignment losses, the training overhead for these methods is minimal. As a result, the training time for different alignment methods remains nearly the same for any given stable diffusion model. This helps explain why we did not include an ablation study comparing training times or analyzing the effect of dataset size.

---

We sincerely apologize for the inconvenience. Following is the supplementary response for Reviewer RW4E, provided here due to the rebuttal length limitation.

---

Q: How would your approach perform with more natural prompts than the ones you used in your evaluation? Would it be possible to consider using T2I-CompBench++ for your evaluation?

A: Following the official setup, we tested on 8,000 prompts in total, covering: attribute binding, relationships, numeracy, and complex compositions. We report the results in the tables below. Higher scores indicate better alignment with the intended composition.