Aligning Diffusion Models by Optimizing Human Utility

Thank you for the positive review! We are excited to hear that you found our method innovative and significant to improving text-to-image models. We are also happy to hear that you appreciated our comprehensive experiments and overall presentation. Please find our responses below.

A more thorough examination of how different biases in the feedback data affect the aligned model would be valuable.

To understand the effect of aligning with Pick-a-Pic v2, which is known to contain some NSFW content, we run a CLIP-based NSFW safety checker on images generated using test prompts from Pick-a-Pic v2 and HPSv2. For Pick-a-Pic prompts, 5.4% of Diffusion-KTO generations are marked NSFW, and 4.4% of SDv1-5 generations are marked NSFW. For HPSv2 prompts, which are safer, 1.3% of Diffusion-KTO generations are marked NSFW, and 1.0% of SD v1-5 generations are marked NSFW. Overall, training on the Pick-a-Pic dataset leads to a marginal increase in NSFW content. We observe similar trends for Diffusion-DPO, which aligns with the same preference distribution (5.8% NSFW on Pick-a-Pick & 1.3% NSFW on HPSv2). We would like to emphasize that our method is agnostic to the choice of preference dataset, as long as the data can be converted into binary per-sample feedback. We used Pick-a-Pic because of its size and to fairly compare with related works.

In some sense, the method itself is not novel... similar methods from the NLP research can potentially be applied to this setting in the future, e.g. BCO or SimPO etc.

Regarding BCO and SimPO, we agree that these works show promising results in the NLP domain. However we find it infeasible to implement and evaluate these methods for text-to-image generation in such a short rebuttal period. We will investigate the effectiveness of these approaches in future works. Please see the “Novel Contributions of Diffusion-KTO” of our main rebuttal for further details regarding novelty.

Including a wider range of diffusion models, as they would better demonstrate the generalizability of the approach.

We provide results using two different models (SD v1-5 and SD v2-1) in our main paper and Appendix. These results highlight the generality of Diffusion-KTO for various text-to-image diffusion models. We agree that it would be interesting to see how our method works for other state-of-the-art models. However, since recent models use more complicated architectures (e.g., Multi-Modal Diffusion Transformer of SD v3.0) and significantly more parameters, it is not feasible to complete these experiments within a week. We leave these endeavors for future work.

While the paper demonstrates the potential for personalizing models to individual preferences, this is only shown through synthetic experiments.

While binary preferences are easier to collect than paired preferences, curating high-quality personalized preference data is still expensive. We explored the possibility of scraping likes and dislikes from websites such as Artstation, but this does not comply with their terms of service. Additionally, it is hard to evaluate whether the model can effectively align with personalized preferences due to the diverse nature of human interests. In contrast, synthetic experiments are more controllable with measurable quantitative metrics such as aesthetic scores (Appendix C). For these reasons, we left the task of curating a high-quality personalized feedback dataset for future works.

Did you use the trainers from the diffusers library?

We refer the reviewer to our sample code in supplementary material for implementation details. We do not use the trainers.

How does the effectiveness of Diffusion-KTO vary with different types of prompts?

Below, we provide a per-style score breakdown using the prompts and their associated styles (Animation, Concept-art, Painting, Photo) in the HPSv2 test set. Across these metrics, our model performs best for "painting" and "concept-art" styles. We attribute this to our training data. Since Pick-a-Pic prompts are written by users, it will reflect their biases, e.g., a bias towards artistic content. Such biases are also noted by the authors of HPSv2 who state "However, a significant portion of the prompts in the database is biased towards certain styles. For instance, around 15.0% of the prompts in DiffusionDB include the name “Greg Rutkowski”, 28.5% include “artstation”." We also observe that different metrics prefer different styles. For example, the "photos" style has the highest PickScore but the lowest ImageReward. With this in mind, we would like to underscore that our method, Diffusion-KTO, is agnostic to the preference distribution (as long as feedback is per-sample and binary) and, training on different, less biased preference data could avoid such discrepancies.

Have you considered extending the method to incorporate other types of feedback beyond binary signals?

While we have considered exploring a variety of human feedback, this work, Diffusion-KTO, focuses specifically on binary feedback. For continuous feedback, we show in Appendix C an example where Diffusion-KTO can effectively align with the preference distribution after the continuous feedback signal is thresholded into a binary one. Textual feedback is non-trivial to implement in such a short rebuttal period, and we leave this for future work.

Thank you for your review! We are glad that you appreciated the contributions and comprehensive experiments presented in this paper.

Regarding the concerns on specific formulations, we have provided a detailed review of our formulation in the main rebuttal that should address these concerns. The relevant parts are summarized as follows:

It's not clear why prop 1 from the DKTO paper still applies.

We assume you are referring to prop1 from the D3PO (not DKTO) paper. It is sufficient to establish this relation under the policy optimization objective. The “substitution” is not wrong because KTO defines its implicit reward function $r_\theta(X_0,C)=\beta\log(\pi_{\theta}(X_0|C)/\pi_{ref}(X_0|C))$ in an axiomatic approach (Definition 3.4 of KTO paper).

KTO justifies such definition through a “classic prospect theory experiment”, and the fact that $r_\theta$ is in the same equivalence class as the human-preference-induced reward function in (eq 2) at optimal policy $\pi^*_\theta$ under RLHF objective (eq 2). In prospect theory, $r_\theta$ "amounts to the dollar amount assigned to each outcome." However, the KTO formulation does not imply that the KTO objective is strictly equivalent to the RLHF objective in a similar way as DPO. There is no "substitution" in KTO formulation.

By "applying the relation $Q^∗(a, s)$" in sec 4.1, we meant that we adopt the implied quality function definition $Q_\theta(X_{t-1},X_{t},C)=\beta\log(\pi_{\theta}(X_{t-1},X_{t},C)/\pi_{ref}(X_{t-1},X_{t},C)$ based on the relation $Q^*(X_{t-1},X_{t},C)= \beta\log(\pi^*_\theta(X_{t-1}|X_{t},C)/\pi_{ref}(X_{t-1}|X_{t},C))$, as well as the connection between $Q^*$ and the RLHF objective established by D3PO. We provide further details in the main rebuttal. If you have any additional questions, feel free to let us know and we are happy to answer them in discussion period.

Secondly, the substitution $Q_{ref}=\beta D_{KL}$ is not very well motivated.

The reference point of KTO is defined as $z_0=E_D[r_\theta(X_0,C)]$, which is the expected reward over some distribution $D$ of $(X_0,C)$. This definition stems from the assumption that "rather than having just one dispreferred generation serve as the reference point z0", "humans judge the quality of [generation] in relation to all possible input-output pairs."

Similarly, we define $Q_{ref}$ as $E_{D'}[Q_\theta(X_{t-1},X_{t},C)]$ over some distribution $D'$ of $X_{t-1},X_{t},C,t$. KTO set $D$ to be a uniform distribution over the input dataset, and the reference point simplifies to the expected KL divergence. Following an identical derivation, we can establish that $Q_{ref}$ in Diffusion-KTO formulation simplifies to the expected KL divergence.

We provide further details in the main rebuttal. We will add these details in future versions and apologize for any confusion caused by this omission.

Note: In the above discussion, we replace the term $l(y)$, the normalization factor in the definition of $r_\theta(X_0,C)$, with a constant $\beta$ following Eq(6) of KTO paper for simplicity. We also replace the notation $x,y$ to $C,X_0$ for consistency with notations of diffusion models.

Thank you for your review! We are happy to hear that you find our problem area important and that you appreciate the presentation of our work and our experimental results. Please see our responses below.

The proposed algorithm is almost a simple application of KTO to diffusion models.

Please see the “Novel Contributions of Diffusion-KTO” of our main rebuttal for our response.

The paper has to be more self-contained.

In regards to the specific concerns raised (i.e. the substitution of $Q$ and definition of $\pi^*$, and $Q_{ref}$), we provide a detailed review of our Diffusion-KTO formulation with clarifications for these definitions and choices in the main rebuttal (Clarification of Formulation). We hope these discussions can resolve your concerns. If you have any additional questions, please let us know and we would be happy to answer them in the discussion period.

The number of human subjects and human responses are not disclosed.

We are sorry for this oversight. We collected 300 human responses. The 95% confidence interval is 65.6%-74.8% for the win-rate against DPO and 73.6%-82.9% for the win-rate against SDv1-5. Both results are significant, with p-value < 0.001.

Thank you for your positive review! We are excited to hear that you appreciate the importance of our problem statement, the strength of our experimental results, and the presentation of our method. Please find our responses below.

The novelty is somewhat limited

Please see the “Novel Contributions of Diffusion-KTO” of our main rebuttal for our response.

Can you include an example without the use of a preference pair dataset.

We provide results when using two protocols for sampling at the population level of our data. We tried a) for each prompt in the dataset, incorporate both winner and loser separately as training data and b) for each prompt in the dataset, keep either the winner or loser as training data. Empirically, we find no significant differences between the performance of these two protocols. This effect is also seen by the authors of the original KTO paper. Specifically, we find the second protocol (b) performs slightly better on Aesthetics (+0.02), PickScore (+0.04), HPS (+0.002), CLIP (+0.13), and ImageReward (+0.05). These differences, apart from ImageReward, are within the error bound of estimates. The second setup (b) could be marginally better than the first setup (a) as it may remove some relative noise within paired preference data. In summary, we did not train with paired preferences and, we find that incorporating both the winner and loser separately or incorporating either the winner or loser into our training data generally leads to no significant difference in performance.

Additionally, we would also like to point out that we tried two toy experiments using binary preferences in Appendix C. In these experiments, we created two synthetic datasets with binary labels by a) using a red filter and setting red-filtered images as desirable and non-filtered ones as undesirable, and b) thresholding the LAION aesthetic score and labeling samples with high scores as desirable and samples with low scores as undesirable. Results show that Diffusion-KTO can effectively align to binary preferences at the user level.

Thank you for recognizing our strong results on Pick-a-Pic. Regarding the results on HPS and PartiPrompts, these datasets have significantly more prompts than Pick-a-Pic, which leads to a smaller margin of error. For example, on HPS, all automatic win-rates above 51.4% have a p-value < 0.00024, and on PartiPrompts all automatic win rates above 51.4% have a p-value < 0.00729, both of which are below the threshold for statistical significance (p=0.05).

We would also like to point out that our human evaluation is conducted on a subset of HPS prompts. The 95% confidence interval is 65.6%-74.8% for the human-evaluated win-rate against DPO and 73.6%-82.9% for the human-evaluated win-rate against SDv1-5. Both results are significant, with p-value < 0.001. These experiments reaffirm a key observation made by the authors of KTO when using KTO for LLMs: "margin between KTO and DPO is even bigger in human evaluations than it is in automated LLM-as-a-judge evaluations", because "maximizing preference likelihood does not mean one is maximizing human utility" [1]. We believe these results provide a key insight for aligning diffusion models with human feedback and are valuable for the community.

If you have any further questions, feel free to let us know in the discussion period.

[1] Ethayarajh, Kawin, et al. "KTO: Model alignment as prospect theoretic optimization." (2024).

The number of test prompts is 500 for Pick-a-Pick, 3210 in HPS and 1632 in PartiPrompts. We generate 5 images per prompt. For human evaluation, we collected 300 responses on randomly sampled HPS prompts.

We agree that the "practical significance" of the improvement is very important, which is why we performed the human evaluation. Since human evaluations show considerable improvement over baselines, we argue that our improvement is "practically significant," because the automatic metrics are mere surrogates for human preference (many of these metrics such as PickScore and ImageReward use reward models trained on human preference data).

To further investigate the significance of our improvements, we compare the automatic win-rate of our method against SD v1.5 versus the automatic win rate of Diffusion-DPO against SD v1.5 on HPS, the largest of three datasets. The results are shown in table below

It has been well established the improvements of Diffusion-DPO are of "practical significance". Compared with Diffusion-DPO, Diffusion-KTO achieves "significantly" higher win-rates against baseline on Aesthetic (+9.7%), PickScore (+6.4%), Image Reward (+13.0%), and is comparable on HPS (+0.1%) and CLIP (-0.5%). Given these results, we argue that we are our method also achieves an improvement of "practical significance".

In summary, we believe that "practical significance" is best measured by human evaluation, which show decisive advantages of Diffusion-KTO. Additionally, we also note that when compared with baseline methods, Diffusion-KTO achieves a larger gap than Diffusion-DPO, which is known to have significant improvements.

To avoid further confusions, we would like to clarify that all occurrences of the term “significance” in the above discussion refers to “practical significance” mentioned in previous comments from AC.

If you have additional questions or would like to see additional experiment results, don't hesitate letting us know.

i think even the Diffusion-DPO work can then be viewed in a similar vein, it just used the method from NLP for the text-to-image model. And to a certain extent, this is a valid concern. I suppose the the KTO method is more practical as it does not require paired data but rather text-image pairs that are deemed good to bad which is more realistic.

There is no redeeming in terms of memory / compute usage either.

Additional Human Evaluation Results

First, we would like to highlight that human evaluation shows Diffusion-KTO has a 70.2% win rate against Diffusion-DPO. We also conducted additional human evaluation of Diffusion-KTO vs SFT, which results in a win rate of 72.3% in favor of Diffusion-KTO.

Further discussion on Automatic Metric

In additional to head-to-head comparison provided in Appendix Table 3, we additionally incorporate the SFT baseline in the results (Method X vs SDv1.5) from previous comment on the HPSv2 test set.

Compared with SDv1.5, Diffusion-KTO achieves a significantly higher win rate than the SFT and Diffusion-DPO baselines. For example, Diffusion-KTO has a lead of +9.7 on Aesthetic, +6.4 on PickScore, and +13.0 on ImageReward when compared with DPO, and a lead of +3.9 on Aesthetic, +9.8 on PickScore, and +8.6 on ImageReward compared with SFT.

In addition, we would argue that results presented in Appendix Table 3 and the table above are indeed "practically significant". For example, while it may seem that a win rate of 54.0 on ImageReward (Ours vs SFT, appendix Table 3) in a head-to-head comparison is not "significant", comparing (Ours vs SDv1.5) and (SFT vs SDv1.5), we can see that Diffusion-KTO expands the margin of improvement by +8.6. Comparing with the neutral baseline (0.5 win rate), we expand the gain from (+13.4) of "SFT vs SDv1.5" to a gain of (+22.0) of "Ours vs SDv1.5", or 164% when evaluated by ImageReward. The gains demonstrated by Diffusion-KTO indicate undoubtedly a significant improvement.

In general, the community has not yet established a clear numerical threshold for what constitutes an automatic win rate of 'practical significance' in text-to-image diffusion models, as these are relatively new metrics. This is why it is important to consider the results provided by human evaluations. In other fields, such as image classification and object detection, even a 1–2-point gain in Top1-Acc or mAP is considered a meaningful improvement. If the AC is suggesting that we adopt a specific numerical threshold to determine the 'practical significance' of the improvements presented in Appendix Table 3 and Table 4, we would appreciate clarification on what that threshold should be to facilitate further discussions. Otherwise, we maintain that: 1) human evaluation has demonstrated that our improvement is of 'practical significance,' and 2) Diffusion-KTO provides a more significant margin of improvement (Table above) when compared to SFT or Diffusion-DPO.

Note: we discovered an editorial oversight in the original manuscript: the columns of the tables are mislabeled. The column label "ImageReward" and "HPS v2" should be swapped for all win rate tables. This has been corrected in the tables we provide above.

To reviewer MiLc and AC, We agree that there has been some miscommunication and appreciate your efforts to clarify your questions and comments. To clarify, we have provided direct comparison with SFT baselines.

1. We have provided automatic win-rates (Diffusion-KTO vs SFT, Diffusion-KTO vs Diffusion-DPO, etc) in Table 1, Appendix Tables 3 and 4.

2. In the comments above, we provided an additional human evaluation which is a direct comparison between images generated by Diffusion-KTO and the SFT baseline using the same HPSv2 prompts. In this human evaluation against SFT, Diffusion-KTO had a win-rate of 72.3%.

3. Further, we also provided qualitative results in Figure 5, which is a "side by side" comparison of images generated using the same prompts from various baselines.

To our best understanding, we think the discussion is not about whether certain results are missing. Rather, its about how to interpret the results.

We thanks reviewer MiLc's insight and hope the above information is helpful. If you belive additional forms of "direct comparisons" other than (1) (2) (3) can facilitate the discussion, or there are better metrics to measure the "degree of improvement", we are happy to provide those as well.

Conceptually, we would also point out that a key difference between Diffusion-KTO and SFT is that Diffusion-KTO learns to "avoid" bad generations. While this property may be hard to measure directly, in Figure 6 we show that in the 2D Gaussian example, Diffusion-KTO and SFT are both able to fit the distribution of "desirable data", but Diffusion-KTO uniquely avoids "undesirable data".

Q: Is such a small improvement practically significant?

Response 1: In a hypothetical of 50.5%-53.5, the answer is yes. We apologize for the confusion in adressing this question. Without any prior literature or community consensus to establish a quantative threshold of "practical significance", we argue that all results that are statistically significant are of "practical use", as it implies "an average improvement". In fact, it is precisely because of the large amount of test data, which leads to low margin of errors, that the ML community consider even 1 or 2 point gain in benchmarks such as ImageNet-1K classification as "significant". We believe, this discussion is outside the scope of our paper and is a broader problem within the ML community in general. To highlight the aspect of "average improvements", we additionally provide average scores on HPS v2 below.

Response 2: We strongly disagree with the AC's premise of "small" improvements

We would like to respectfully mention, that we have previously inquired about a clear numerical threshold for "practical significance," but this has not been specified. Without a defined standard, it is unfair to justify dismissing our improvements as insignificant. We would like to point out that we have also presented human evaluation results comparing Diffusion-KTO and SFT side-by-side and found that human evaluators prefer Diffusion-KTO generations 72.3% of the time. We believe this 70%+ win-rate is practically significant considering human evaluation is the best measurement of how well models are aligned with human preferences. We kindly request the AC to provide a clear threshold and give justifications for such threshold, otherwise, we stand that our results are significant particulary, practically significant.

We would like to also point out that the win rate is more thoroughly studied in the context of LLM. At the moment, 54% win rate is the win rate between LLama3.1-405B-instruct versus LLama3.1-70B-instruct on lmsys tournament. We recognize that the task is different, however given the lack of related studies in the context of text-to-image generation, we think this information may be useful.

Response 3: Discussions on specific "in-significant" numbers.

Looking back at the discussion, the AC noted his/her reservations based on some low numbers in Table 3 and Table 4, but we think our strong results against SFT in Table 1 have been largely marginalized. We apologize but as we mentioned above, our stance is that it is unreasonable to expect Diffusion-KTO to exhibit a large numeric gap on all metrics, as such are rare cases for most works. We believe that all results should be looked at holistically, based on the following principles.

1. Human evaluation should outweigh automatic ones, as this is the direct objective of preference optimization and human-utility optimization.
2. Among all numerical metrics, PickScore is the most significant one as it is trained to reflect the human preference exhibited in the Pick-a-Pick dataset, and is most suitable to evaluate the performance of training algorithms. While other metrics are also trained on preference data, differences in dataset curation may lead to different biases in these reward models.

In this regard, we argue that human evaluation and the results on Pick-a-Pick test set in Table 1 are the best measurements to compare the effectiveness of optimization algorithms, which the AC has kindly acknowledged: "clearly higher winning rates based on Pick-a-Pick test prompts".

As requested, to facilitate the discussion, we listed the error bars of 95% confidence interval as below. We observe that all win rate has a margin of error <2%.

Win rate on Pick-a-Pick Test Set (2500 generations)

Win rate on HPS v2 Test Set (16050 generations)

Win rate on PartiPrompts (8160 generations)

Q:How do you sample prompts for Fig 5. To highlight the advantage of Diffusion-KTO in real-world scenarios, we create prompts based on user prompts shared over the internet. In particular, we heavily referenced Playground.AI, where users share generated images alongside their prompts. While these prompts are manually written, we believe they reflect typical use cases in real-world scenarios and is generally similar to the "good" samples in the Pick-a-Pic dataset, which is also written by human labelers. To further highlight the fairness of our method, we report the automatic win rate of 20 randomly selected prompts and generate 5 images per prompt. The prompts are copied from the front page of playground AI on Aug 10, 2024. An example prompt would be

Photorealistic wet-erase marker portrait depicts Cleopatra with a hybrid aesthetic of a bikini and Galadriel's queenly regalia,....

The results are as follows

For simplicity, we adopt an overall win rate in the above calculation. X > Y if and only if amongst 5 metrics, (Aesthetic ,PickScore ,ImageReward , CLIP ,HPS), at least 3 are better. Contrary to AC's statement, "the likelihood of (Diffusion-KTO) outperforming all others", in the distribution of user-specified prompt sampled from Playground AI, is 49.9%, which is not small. This is because the win rates are highly correlated with each other and not independent. For example, if Diffusion-KTO beats SFT and DPO on a prompt, it is likely that it will beat SDv1.5 as well.

Q: Have you adequately utilized negative-labeled examples in SFT. We request further clarification on how to adequately utilize negatives in SFT as it is not in the default formulation. We tried to naively add a negative term to the L2 loss to "steer clear of poor-performing regions" using the loss $\mathbb{E}[y(x_t)\lVert \epsilon_\theta(x_t)-\epsilon\rVert^2]$ where $y(x_t)\in\{+1,-1\}$ is the label. The model diverge to infinity as the loss on positive samples has range (0,+inf), but the loss on negative samples has range (-inf,0). This results in the model pushing the loss to -inf. In fact, it is precisely because this form of naive contrastive SFT does not work, that the community has resorted to alternative approaches such as RL, Inverse RL, and more recently DPO.

In summary, we want to apologize for the confusion during this discussion and we hope in this response to have cleared some of the questions by providing detailed information.

Hi,

We are glad that the AC recognize these advantages. We sincerely apologize for any potential misunderstanding and miscommunications. We might have wrongly assumed the discussion around "practical significance" was based on perceived "substantial differences in numbers" as a criterion in lieu of "statistical significance", a stance we found to be not objectively grounded. We are glad that this is not the case, and the confusions have been cleared away.

Q:Further Clarification on Fig 5

For Figure 5, the images are not chosen by specific seeds, we just randomly generate an image per prompt. However, the prompt, which is listed on the left most column, are not randomly selected from a dataset. They are manually written ones based on practical use cases on Playground AI, a website where users share their prompts. Typically, these prompts are highly specific. Our observations show that given these highly specific prompt, (we have provided an example above), the images generated by each model will have small variations. Hence, the difference between models is fairly consistent across generations. We provided additional analysis of these prompts in the previous comment. Please feel free to let us know if you have any additional questions.

Q: Reproducibility of Human Evaluation

We agree that this is a limitation for human evaluations. We hope future works from the community can establish an open and reproducible ranking based on human evaluation like lmsys for LLMs. However, at the moment, we have provided a detailed description of the human evaluation protocol in appendix A.2, alongside designs to mitigate systematic biases (e.g. "not given any information about which methods are being compared and the order of methods (left or right) is randomized"). The statistic confidence which we provided in previous comments and our rebuttal to reviewer 7SUT in this case should be a good measurement for reproducibility.

Q:It's important to consider that the method's advantages might be less pronounced under domain shifts.

We agree with this observation. Then, it is critical to consider what is the distribution of prompts that best reflect a "practical use", as this is a key focus of this discussion.

Consider the common use case of generating people as a case study. The PartiPrompts "People" category contains prompts such as "a woman with long hair", "a boy going to school".

On HPS v2, examples of prompts involving people are "Blond-haired girl depicted in anime style", "A cyberpunk woman on a motorbike drives away down a street while wearing sunglasses"

Pick-a-Pick contains a mix of different styles and even in different languages. Examples are "A woman’s wrist with a thin golden bracelet, romantic city lights in the background", "Eighteen year-old girl, pale skin, smooth skin, black jacket, ...... Chris Bachalo, Belén Orteg".

Most prompts on Playground AI are similar to longer ones in Pick-a-Pick, such as "Transparent-skinned creature showing complex anatomical structures intertwined with botanical patterns,... Lynn Maycroft's work, cinema."

Unlike the previous three academic datasets, these are prompts used by actual paid subscriber who are willing to pay 15-35 USD per month for T2I models. The authors believe this distribution best illustrate the "practical usecase", which is why we use this for our qualitative results in Fig5. While we also made our best attempt to provide some quantitative analysis in previous comments, various engineering and legal complications of large-scale web-scrapping prevents us from scaling these analyses to a full-sized benchmark.

In summary, we agree that "advantages might be less pronounced under domain shifts", but we consider this a natural consequence of training on Pick-a-Pick. Given that KTO only requires binary preferences, it is easier to collect such preference data on the intended distribution (e.g. PartiPrompts). The authors hold that results presented in this paper have met the burden of "substantiate the empirical advantages". It can be reasonably expected that KTO should show similar improvements with proper preference data on any distribution of prompts.

The main contribution of Diffusion-KTO is a general and effective optimization algorithm, not the actual model trained on Pick-a-Pick dataset. The latter is an experiment to study the effectiveness of the former. We do not claim this specific model weight has stronger generalization capability on unseen concepts, but rather, it can reasonably generalize to "practical usecases" as discussed above. In fact, we believe it is unrealistic to expect any model (including SFT,DPO) that is trained on Pick-a-Pick prompts to generalize to prompts such as "element", "concurrent lines" (which are concepts uniquely present in PartiPrompts). It is even debatable if human preferences make sense for such prompts. We appreciate the AC's detailed feedback and discussion.