Dual Caption Preference Optimization for Diffusion Models

We greatly value the reviewer’s insightful feedback on our paper and look forward to engaging in a productive discussion, as there is still significant time left in the discussion period. Below, we offer a detailed response to your comments:

W1: The issues of conflict distribution and irrelevant prompts seem like two aspects of the same problem—both involve a single prompt (C) corresponding to two different images, which can lead to unstable optimization. Therefore, I think they could be consolidated into a single issue.

Reply to W1: We appreciate the reviewer’s observation, but conflict distribution and irrelevant prompts address distinct challenges.

Conflict Distribution refers to the similarity between preferred ($x^w$) and less preferred ($x^l$) images, quantified using the correlation between images ($x$) and captions ($z$) via CLIPscore. High similarity can make it harder to distinguish preferences during optimization.

Irrelevant Prompts highlight misalignment between the prompt ($c$) and the less preferred image ($x^l$). Prompts often include details relevant to $x^w$ but not $x^l$, which hinders the model's ability to learn effectively.

For example, even when $x^w$ and $x^l$ have similar CLIPscores, a prompt $c$ may still lack relevance for $x^l$, introducing additional optimization challenges. These two issues complement but do not fully overlap, as both similarity and prompt relevance impact performance differently.

W2: When comparing generated images, the improvements achieved by the proposed method could be highlighted more clearly; otherwise, it’s often not immediately obvious, as in Figure 1.

Reply to W2: Thank you for this comment. We have updated Figure 1 in the revised version. Additionally, the reviewer can find more qualitative examples across different benchmarks in Appendix F.

W3: In fact, the explanations of conflict distribution and irrelevant prompts in the abstract and introduction are quite obscure and difficult to understand. I had to reread these sections several times, only gaining clarity after reading the methods section. This part may need reorganization.

Reply to W3: Thank you for your suggestion. We are in the process of re-organizing this section and will add a note here when it is completed on our end.

Dear Reviewer Y9en,

As the deadline is nearing, we wanted to gently follow up on our recent submission. Your feedback is highly valuable to us, and we would appreciate any updates or further guidance you might have regarding our revisions and responses.

Thank you for your time and consideration.

W3: In fact, the explanations of conflict distribution and irrelevant prompts in the abstract and introduction are quite obscure and difficult to understand. I had to reread these sections several times, only gaining clarity after reading the methods section. This part may need reorganization.

Reply W3: Following your suggestions, we have added two additional lines to clarify the explanation of the irrelevant prompts issue in the abstract (L16–L18) and introduction (L101–L107).

We hope our explanations have clarified any unclear aspects of our proposed method. Your questions and feedback have been invaluable, and we sincerely appreciate the opportunity to address your concerns. If you feel that we have adequately resolved your queries and you are satisfied with the discussion, we would be grateful if you could consider revisiting the rating of our work.

We sincerely appreciate your time and thoughtful review!

In our general comments, we provided both theoretical and experimental motivation for our work, along with a series of proofs demonstrating the effectiveness of DCPO compared to diffusion-DPO. As the rebuttal period approaches its conclusion, we would like to respectfully ask if you have any remaining concerns regarding the motivation behind our work, the effectiveness of our method (including the additional results provided), or any other aspects of our submission. We have been more than happy to address any remaining or new concerns during the discussion period.

If our responses have adequately addressed your concerns, we kindly ask you to consider reevaluating our work in light of the updated information.

Thank you once again for your valuable feedback!

W4: The LAION-2B and MSCOCO datasets are widely regarded benchmarks for image generation tasks, yet they are not discussed or evaluated within this study. The absence of experiments or comparisons involving LAION-2B raises questions about DCPO’s general applicability.

Reply to W4: LAION and MSCOCO are not equipped with preference data by design, and there is no publicly annotated version of them. Hence, we are unable to perform experiments on them. Our practice is well aligned with existing works such as [1], [2]. Still, for image generation purposes, we evaluated DCPO and Diffusion-DPO on the FID metric to provide a broader perspective on their performance, and DCPO consistently outperformed Diffusion-DPO.

[1] Diffusion-DPO: https://arxiv.org/abs/2311.12908

[2] MaPO: https://arxiv.org/abs/2406.06424

We sincerely thank the reviewer for their insightful feedback on our paper. We look forward to engaging in a constructive discussion, as there is still plenty of time remaining in the discussion period. Below, we present a detailed response to your comments:

W1: The proposed method depends on the caption quality. The quality of generated captions significantly affects performance, and challenges remain in creating effective captions for less preferred images without straying out-of-distribution.

Reply to W1: One of the key experiments in our paper investigates how the quality of generated captions influences the alignment performance of diffusion models. We first define high-quality captions as those with a higher correlation between image $x$ and caption $z$. In Table 5 in Appendix B, we reported the average CLIP score as a standard correlation metric.

The results in Table 5 in Appendix B indicate that captions generated by LLaVA have better quality, evidenced by a higher average CLIP score compared to Emu2. Specifically, the difference in correlation scores between preferred and less preferred captions is approximately 4 for LLaVA and 3 for Emu2, both of which are larger than the original prompt (~2, as explained in Lines 340–357). However, despite the higher caption quality of LLaVA, the performance of DCPO-c-Emu2 remains comparable to DCPO-c-LLaVA, as shown in Tables 1 and 2.

This suggests that while caption quality plays a role, it is not the primary determinant of performance. Instead, the critical factor is the difference in correlation between captions and the preferred and less preferred images. Evidence for this conclusion arises from the superior performance of DCPO-h, which achieves the largest correlation distance among DCPO-c and DCPO-p techniques, as shown in Figure 4. This supports the hypothesis that maximizing the correlation difference, rather than caption quality alone, is one of the keys to optimizing alignment performance.

W2: While DCPO demonstrates quantitative improvements across several metrics, the qualitative results (e.g., Figure 1) indicate that the visual distinctions between images generated by DCPO and baseline methods are not significant. This subtle difference may limit the perceived impact of DCPO in practical applications.

Reply to W2: Thank you for this comment. We have updated Figure 1 in the revised version. Additionally, the reviewer can find more qualitative examples across different benchmarks in Appendix F.

W3: The DCPO has limited generalizability compared to real-world large-scale datasets. Although leveraging preferred and non-preferred images is a novel approach for enhancing diffusion models, high-quality, large-scale datasets from real-world settings often provide stronger improvements in model performance. This reliance on real-world data diminishes the relative advantage of DCPO, potentially limiting the distinctiveness of its contributions in scenarios where comprehensive datasets are available.

Reply to W3: To the best of our knowledge, real-world datasets do not have a preference structure. Preference data requires selecting preferred and less preferred images from multiple images generated for the same prompt based on human judgment. While real-world datasets like MSCOCO and LAION-2B are valuable, they are not suitable for preference optimization due to the lack of this structure.

Preference datasets are specifically used in post-training, a step following pre-training, to enhance the performance of pre-trained models, as outlined in Diffusion-DPO. For preference optimization, a dataset D ={$c, x^w, x^l$} is required, where $x^w$ and $x^l$ represent the preferred and less preferred images for the same prompt $c$. This specialized data structure is essential for methods like DCPO to achieve their objectives.

To demonstrate DCPO's generalizability, we fine-tuned Stable Diffusion 2.1 using Diffusion-DPO and DCPO on another high-quality preference dataset - Rapidata Image Generation Preference Dataset (RIGPD) [1]. The table below shows that DCPO variants consistently outperform Diffusion-DPO across multiple benchmarks, including Geneval, Pickscore, HPSv2.1, ImageReward, and CLIPscore.

These results highlight DCPO's versatility and superior performance across benchmarks.

[1] Rapidata Image Generation Preference Dataset (RIGPD): https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3

Dear Reviewer CPj9,

As the deadline is nearing, we wanted to gently follow up on our recent submission. Your feedback is highly valuable to us, and we would appreciate any updates or further guidance you might have regarding our revisions and responses.

Thank you for your time and consideration.

We sincerely appreciate your time and thoughtful review!

In our general comments, we provided both theoretical and experimental motivation for our work, along with a series of proofs demonstrating the effectiveness of DCPO compared to diffusion-DPO. As the rebuttal period approaches its conclusion, we would like to respectfully ask if you have any remaining concerns regarding the motivation behind our work, the effectiveness of our method (including the additional results provided), or any other aspects of our submission. We have been more than happy to address any remaining or new concerns during the discussion period.

If our responses have adequately addressed your concerns, we kindly ask you to consider reevaluating our work in light of the updated information.

Thank you once again for your valuable feedback!

W3: From the description of L175-L180, the irrelevant problem is hardly a problem either. It is an inherently part of the objective in Eqn (1), where one way of minimizing Eqn(1) is to decrease $log(p(xi|zl))$, which makes the model less likely to generate the less preferred image. So to me this is a desired behavior instead of a problem.

Reply to W3: We observed that the prompt $c$ often does not serve as a meaningful caption for less preferred images (see Figure 2 in the main text and Figure 10 in Appendix C) because it may contain irrelevant information. We refer to this as the "irrelevant prompt" problem. The key question is: Why is an irrelevant prompt problematic?

U-Net predicts the added noise for both preferred and less preferred images conditioned on a prompt, as shown in Equation 2. If the prompt is irrelevant for the less preferred images, U-Net’s ability to make accurate predictions becomes limited. This issue is supported by findings in the DALL-E 3 paper [1], which demonstrates that more relevant captions improve U-Net’s performance in predicting noise. Building on this, we generated relevant captions for the less preferred images using a captioning method that conditions the original prompt.

During direct preference optimization, $\log P(x^w|c)$ increases while $\log P(x^l|c)$ decreases, where $x^w$ and $x^l$ represent the preferred and less preferred images generated for the same prompt $c$. Theoretically, if a prompt $c_1$ has a stronger correlation with both the preferred and less preferred images, the optimization process is more effective, resulting in better model performance. Intuitively, a diffusion model better understands why humans prefer image $x^w$. Because the prompt $c_1$ contains more relevant information. Thus, prompts with high correlation between $x^w$ and $x^l$ can influence $\log P(x|c)$ more effectively during optimization. The results in Table 4 provide strong evidence supporting this explanation.

Inspired by this behavior, we hypothesized that generating a more suitable caption for less preferred images, distinct from the original prompt, helps decrease the likelihood of less preferred images during optimization. This aligns with our findings and further validates the importance of addressing the irrelevant prompt issue.

[1] DALL-E 3: https://cdn.openai.com/papers/dall-e-3.pdf

W4: By changing the captions, authors changed a prefer/less prefer pair into two separate samples. In this sense it is no longer the original DPO problem, yet there is no clear connection between the original DPO formulation and the new problem e.g. is the new one an upper-bound of the original so minimizing the new problem potentially minimize the original one? or why solving the new problem will necessarily give better results than original DPO?

Reply to W4: In the DCPO framework, we modify the original prompt by replacing it with a caption generated by a captioning model $Q_\phi$. This modification enables better alignment with the less preferred images while maintaining the DCPO in the original task space.

To establish the equivalence with the original Diffusion-DPO problem, we theoretically demonstrate that Diffusion-DPO provides a lower bound of the DCPO loss function. In the DCPO framework, the caption $z$ is generated by the captioning model $Q_\phi$ conditioned on the image $x$ and the original prompt $c$, i.e., $z \sim Q_\phi(z \mid x, c)$. Importantly, in scenarios where $Q_\phi$ generates a caption $z$ that closely matches the original prompt $c$, we effectively have $z \simeq c$. In this case, substituting $z$ for $c$ in the DCPO loss function produces the original Diffusion-DPO loss function (refer to Equation 9 in Appendix A).

This theoretical equivalence ensures that minimizing the DCPO loss function aligns with minimizing the Diffusion-DPO loss. Moreover, the modification introduced by $Q_\phi$ helps address challenges arising from irrelevant prompts in the original framework, which we have shown to improve alignment and performance on various benchmarks. Thus, solving the DCPO problem not only retains the original Diffusion-DPO framework’s objectives but also provides additional flexibility for addressing inherent limitations, leading to better results.

We sincerely thank the reviewer for their thoughtful feedback and the time dedicated to evaluating our paper. We hope our responses below address the concerns raised and support a potential reconsideration of the score.

W1: The motivation behind the proposed approach is not clear to me. For the conflict distribution challenge, when the distribution overlap becomes larger, the dataset is proposing a harder problem for the model to optimize, but it isn't necessary an issue as long as the two distributions are not identical.

Reply to W1: Suppose we aim to optimize a policy model $p_\theta$, such as a large language model by preference optimization methods. For this, we require a dataset $D$ that includes both preferred ($x^w$) and less preferred ($x^l$) responses generated for the same prompt $c$. The standard method involves generating multiple responses for a single input, ranking these responses through human evaluation or automated Judger models (e.g., GPT-4o), and selecting the best response as preferred and the least desirable as less preferred. This methodology is outlined in the OpenAI paper on Reinforcement Learning with Human Feedback (RLHF) [1].

Also, in commonly generated preference datasets such as UltraFeedback-binarized [2][3] and UltraFeedback-PairRM [4][5], a similar approach is utilized with RLHF. However, we observed that existing vision-based preference datasets, such as Pick-a-Picv2, demonstrate a high correlation between preferred and less-preferred images—a phenomenon we refer to as "conflict distribution."

Our hypothesis is that increasing the divergence between the correlation distributions of preferred and less preferred images can enhance the performance of diffusion models optimized using direct preference optimization methods. By addressing this "conflict distribution," our approach aims to make the optimization process more effective and improve overall model performance.

[1] RLHF: https://arxiv.org/pdf/2203.02155

[2] UltraFeedback Binarized: https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback\_binarized

[3] Zephyr: https://arxiv.org/abs/2310.16944

[4] UltraFeedback PairRM: https://huggingface.co/datasets/princeton-nlp/llama3-ultrafeedback

[5] SimPO: https://arxiv.org/abs/2405.14734

W2: When the diffusion models's quality gets better, the two distribution will inevitably become more and more similar, as both preferred and less preferred images from an optimized model will be closer to real human preference. So it's more of the nature of the task itself, unless the task is defined differently.

Reply to W2: The goal of preference optimization is to train a diffusion model to distinguish between preferred and less preferred images generated for the same prompt, where the less preferred image exhibits some differences from the preferred image. Theoretically, if the preferred ($x^w$) and less preferred ($x^l$) images are too similar, the values of $\log P(x^w|c)$ and $\log P(x^l|c)$ will converge, leading to a loss value close to zero. This would hinder the model's ability to effectively learn the preferred distribution.

Moreover, during optimization, the diffusion model is designed to increase the likelihood of $x^w$ while decreasing the likelihood of $x^l$. However, when there is a high degree of similarity between preferred and less preferred images, the model may become confused, struggling to differentiate between the two distributions effectively.

To address the potential convergence of the preferred and less preferred distributions as models improve, we propose an alternative evaluation framework. Instead of assigning binary preference scores to image pairs (e.g., $I_g = 1$ and $I_b = 0$), we suggest adopting a soft-scoring system that captures varying degrees of alignment with human preference across multiple images. For example, in a set of four generated images, scores could indicate preference levels such as {$I_1: 0.7, I_2: 0.5, I_3: 0.1, I_4: 0.6$}, where $I_1$ represents the most preferred image and $I_3$ the least preferred.

Using the FLUX model, we tested this framework on 100 samples, generating $I_1$ and $I_2$ from different seeds. The average difference across 100 samples was 0.4, demonstrating that current models like FLUX still struggle to produce multiple well-aligned images from the same prompt, further supporting the relevance of our proposed framework.

W5 The change of captions made the problem closer to the KTO problem referenced in the paper, where text-image data are labeled by like and dislike binary labels. Please describe the connection and difference between the modified problem represented by the new data and the KTO problem formulation above.

Reply to W5: Assume a preference dataset D ={$c, x^w, x^l$}, where $x^w$ and $x^l$ represent the preferred and less preferred images for the prompt $c$. The Diffusion-KTO [1] hypothesizes optimizing a diffusion model using only a single preference label based on whether an image $x$ is suitable or unsuitable for a given prompt $c$. Thus, Diffusion-KTO utilizes a dataset D ={$c, x$}, where $x$ is a generated image corresponding to prompt $c$.

This hypothesis is fundamentally different from ours. While Diffusion-KTO focuses on binary preferences (like/dislike) for individual image-prompt pairs, our approach involves paired preferences. We do not claim that having two preferences is problematic; rather, we observed that using the same prompt $c$ for both preferred and less preferred images may not be ideal. To address this, we propose optimizing a diffusion model using a dataset D ={$z^w, z^l, x^w, x^l$}, where $z^w$ and $z^l$ are captions generated by a captioning model $Q_\phi$ for the preferred and less preferred images with respect to the original prompt, respectively.

However, we compared Diffusion-KTO on various benchmarks, and the results are presented in the following table. This comparison highlights how the differences in problem formulation influence the model's performance and further emphasizes the distinct nature of our proposed approach relative to Diffusion-KTO.

[1] Diffusion-KTO: https://arxiv.org/abs/2404.04465

W6: It is great to conduct extensive experiments on SD 2.1, but the paper will be stronger if there are experiment results on other diffusion models, even if the experiments are not as complete as on SD 2.1.

Reply to W6: In response to the reviewer's suggestion, we conducted additional experiments to evaluate the performance of DCPO on the SDXL model. Because SDXL is a large model, we perform LoRA fine-tuning with minimal hyper-parameter search.

The results presented in the following table demonstrate that DCPO outperforms Diffusion-DPO on metrics such as Pickscore, Geneval, HPSv2, and CLIPscore. Additionally, DCPO achieves performance comparable to Diffusion-DPO on ImageReward. These findings highlight the generalizability and effectiveness of DCPO across different diffusion models.

Q1: Despite the experiments suggests the proposed approach is better, it is unclear to me why this would be the case, any proofs or intuitions will help reader better understand it.

Reply to Q1: The Diffusion-DPO objective is designed for preference optimization using a dataset D ={$c, x^w, x^l$}, where $x^w$ and $x^l$ represent preferred and less preferred images for a given prompt $c$. DCPO enhances this framework by replacing the prompt $c$ with a caption $z$, generated by a captioning model $Q_\phi$, which provides more relevant and contextually aligned information about each image. This adjustment improves performance by enabling better contextual conditioning, where $z^l$ captures specific details relevant to $x^l$ that $c$ might lack. It also reduces noise by eliminating irrelevant details in $c$, supporting the model in distinguishing $x^w$ from $x^l$ more effectively. Additionally, conditioning on $z^l$ simplifies dependencies, ensuring the model focuses entirely on the necessary information for preference evaluation. These improvements result in better preference optimization, as demonstrated by the stronger correlation between captions and their corresponding images compared to the original prompts.

Q2: Several papers appear multiple times in the References section, please dedupe.

Reply to Q2: Thank you for pointing this out! We’ve fixed the issue in the updated version.

Dear Reviewer f715,

As the deadline is nearing, we wanted to gently follow up on our recent submission. Your feedback is highly valuable to us, and we would appreciate any updates or further guidance you might have regarding our revisions and responses.

Thank you for your time and consideration.

We sincerely appreciate your valuable time and thoughtful review.

Regarding your concern about using different captions for the preferred and less preferred images, we would like to clarify that our approach aligns with prior work. As a side note, we have found that a recent paper accepted at ICML proposes DOVE, a loss function in a design similar to our DCPO, to address the limitations of the DPO loss function in language models. Similarly to our method, DOVE generates distinct instruction inputs for the less preferred examples, resulting in different inputs for both the preferred and the less preferred cases. We kindly refer you to [1], and its Figure 1 and Equation 2, for reference.

As the rebuttal period is near its conclusion, we respectfully ask if you have any further concerns regarding the motivation behind our work, the effectiveness of our method (including the additional results provided) or any other aspects of our submission. We have been more than happy to address any remaining questions or new ones during the remaining discussion period.

Thank you again for your constructive feedback!

[1] DOVE: https://openreview.net/pdf?id=AzMnkF0jRT

Thank you for your valuable feedback. We have thoughtfully addressed your comments below and look forward to discussing any remaining concerns during the discussion period to facilitate a positive re-evaluation of the score.

W1: My biggest concern is about the generalization of the approach method in the development of diffusion models. For example, in Figure 2, it is easy to distinguish the preferred and less-preferred image as the latter one even does not align with the original prompt. What if the model's development is already beyond the alignment stage? The current positive/negative samples are only about alignment, what about more advanced difference if both have enough alignment?

Reply to W1: Thank you for your question. In this study, we follow the prevailing paradigm in preference data, where for a given text prompt $P$, an image $I_g$ is deemed preferred, while an image $I_b$ is less favored. We first observe that even leading models, such as FLUX, currently struggle to generate multiple aligned images from the same prompt. This current limitation highlights the significant potential for improvement in producing images with robust alignment.

Assuming future advancements close this alignment gap, we propose an alternative evaluation framework: rather than assigning binary preference scores to image pairs (i.e., $I_g = 1$ and $I_b = 0$), we envision a soft-scoring system that captures varying degrees of alignment with human preference across multiple images. For instance, if four images are generated, they could be ranked with scores indicating preference levels, such as {$I_1 : 0.7, I_2 : 0.5, I_3 : 0.1, I_4 : 0.6$}, where $I_1$ is the most preferred and $I_3$ the least preferred.

To test this approach, we generated 100 samples $I_1$ and $I_2$ using different seeds with FLUX and asked two volunteers to assign soft scores. The observed average difference of 0.4 demonstrates that current models like FLUX face challenges in producing multiple well-aligned images from the same prompt, emphasizing the importance of our proposed framework.

W2: Line 188-189, could you explain more details on how to get the preferred and less-preferred images? Human annotation?

Reply to W2: In our experiments, we use the Pick-a-Pic v2 dataset, which is human annotated. This dataset is constructed where, for a given prompt, 2 images are generated; the user then chooses the preferred image between the two.

• For more details, please refer to https://stability.ai/research/pick-a-pic.

W3: In terms of GPT-4o evaluation, does it matter for showing the images together or showing them separately? And how about the order of showing them to GPT-4o if showing separately?

Reply to W3: We follow standard practice, as in Diffusion-DPO [1] and MaPO [2], by showing two images side by side for comparison. To address positional bias in GPT-4o's evaluations, we alternate the positions of the images across different criteria (explained in Section 3).

The results below show that DCPO consistently achieves better performance than Diffusion-DPO, even when positional bias is accounted for.

This demonstrates that DCPO provides more reliable results under unbiased conditions.

W4: It would be beneficial to highlight the difference between medium and strong permutation. Do we have a way to quantify the difference between them? Are they controllable generated? Why do we need medium permutations? Would weak/strong be enough?

Reply to W4: In our study, perturbations are controllably generated using DIPPER, a model that allows configuration via a 'lexicon diversity' parameter, which ranges from 1 to 100. Lexicon diversity measures the likelihood of a perturbation introducing alternative expressions or synonyms. By setting this parameter to 40, 60, and 80, we create weak, medium, and strong perturbations, respectively.

Qualitatively, weak perturbations replace certain words in the original caption with synonyms, medium perturbations combine synonym swapping with the reordering of words, and strong perturbations paraphrase the caption into entirely different sentence structures.

Quantitatively, Figure 4 illustrates the differences across these levels. Based on the CLIPScore distribution of the original captions (turquoise histogram in (a)), we observe that stronger perturbations shift the CLIPScores further left. Medium perturbations (c) remain closer to the original captions in terms of mean and standard deviation compared to strong perturbations (d), demonstrating that medium perturbations achieve a balance between preserving the original meaning and introducing diversity.

[1] Diffusion-DPO: https://arxiv.org/abs/2311.12908

[2] MaPO: https://arxiv.org/abs/2406.06424

Dear Reviewer tyap,

As the deadline is nearing, we wanted to gently follow up on our recent submission. Your feedback is highly valuable to us, and we would appreciate any updates or further guidance you might have regarding our revisions and responses.

Thank you for your time and consideration.

Really appreciate authors' efforts on the clarifications.

Hypothesis 1: not sure what does "improve" mean. I can see that a more relevant caption $z_l$ will increase $\logP(x_l|c)$ to $\logP(x_l|z_l)$ which increase the loss. So it to some degree sets an upper bound to the original loss, but it is unclear why optimizing this upper bound would be necessarily better than optimizing the original loss.

Overall I'd hold a neutral position if this paper simply states that through experiments they found generating $z_l$ and $z_w$ , and use them in DPO instead of $c$ yields better performance. It's quite unfortunate that so far none of the motivations sounds reasonable to me. It seems other reviewers are also confused.

continue of Proof 3:

The residual for $R_\theta$ is proportional to the misalignment between $Q(\mathbf{c})$ and $\boldsymbol{\epsilon}^w$:

Replacing $\mathbf{c}$ with $\mathbf{z}^w$ (higher $Q$) results in a larger proportional reduction:

The reference model's residual $R_{\text{ref}}$ depends weakly on $Q(\mathbf{c})$, as it is fixed or less adaptable:

where $Q_{\text{ref}}(\mathbf{c})$ is less sensitive to changes in $\mathbf{c}$.

Thus, the proportional improvement in $R_\theta$ due to $\Delta Q$ is significantly larger than for $R_{\text{ref}}$.

The preferred difference term is:

As $R_\theta$ decreases significantly more than $R_{\text{ref}}$, the gap $R_\theta - R_{\text{ref}}$ becomes larger, increasing $\Delta_{\text{preferred}}$:

The learnable model $\boldsymbol{\epsilon}_\theta$ benefits more from the improved conditioning $\mathbf{z}^w$

because of its adaptability and training dynamics. This results in a larger reduction in $R_\theta$ compared to $R_{\text{ref}}$. Mathematically, the relative rate of decrease:

which ensures that $\Delta_{\text{preferred}}$ also increases, hence improving the optimization process in $L(\theta)$ and helping the model distinguish predictions on preferred and less preferred image-captions more effectively.

We hope our responses have addressed the follow-up concerns over our motivations and methodology raised by the reviewers. All of the formal proofs in these 4 parts will become part of the finalized version of our paper.