PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference

Reviewer G8Cx

Q1. Differences from Other Diffusion Alignment

Our work is significantly different from existing diffusion alignment work. While some related works exist in the area of text-to-image tasks involving human preference, our work is the first to align diffusion-based image inpainting with human preference through reinforcement learning (RL). Initially, our proposal of the human preference inpainting dataset enables the feasibility of this task. Without such a dataset that includes high-quality labels based on human preference, undertaking inpainting alignment would be impossible. Technically, we propose a reward accuracy-aware weight strategy into the RL process to further accelerate the training process and boost performance, which has also not been investigated in the mentioned works. We will add further illustrations about the works related to diffusion model alignment with human preference.

Q2. Experimental Comparisons with your Mentioned Methods

Moreover, to further resolve the reviewer's concern, we have experimentally compared our method with your mentioned methods. Specifically, we simply summarize the implementation of each method.

(1) Human Preference Score (ICCV 2023)[1] learns a negative prompt to map the diffusion process to low-quality samples. Then, in the inference process, the negative sample is utilized in the classifier-free guidance~(CFG) to push the generation trajectory away from low-quality samples.

(2) ImageReward (NeurIPS 2023)[2] trains a reward model and then applies the reward model as a loss metric to end-to-end optimize the diffusion model accompanied by a reconstruction loss. We also conduct the ablation study on reward training strategy in Table 3 in our paper. Our method employs a regression-driven training strategy, while ImageReward (NeurIPS 2023) [2] a classification-drive strategy.

(3) DPOK (NeurIPS 2023) [3] simultaneously optimizes the whole trajectory of a reverse diffusion process and utilizes the KL divergence to panel the regularization, avoiding a large distribution shift.

(4) D3PO (CVPR 2024) [4] adopts the RL strategy from direct performance optimization (DPO)[5], directly optimizes the model on the reward labeling data to minimize the probability of low-quality samples and increase the probability of high-quality samples.

The experimental results shown in the table below further validate the advantage of the proposed method.

Moreover, the authors will provide further discussion in the final version of the paper.

User Study

First, we clarify that the labeling of our dataset is provided by a professional data labeling company, with all annotators being professionals trained in similar tasks. The reward scoring is highly accurate and capable of assessing the results of inpainting under criteria based on human preference. We also provide some examples scoring by our reward model in Fig S4 in the uploaded one-page PDF file, the assessment of inpainted images is closely aligned with human preference. Moreover, to alleviate your concerns, we have also carried out a user study to evaluate our superiority. We have randomly selected about 130 groups of results and conducted a user study involving 10 users, as detailed below. The WinRate map and the diagram of the user study platform can be found in Fig. S3 and Fig. S5 of the uploaded one-page PDF file.

Dear Reviewer G8Cx,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We understand that you may be reviewing multiple papers and have a busy schedule. In our previous response, we made sure to address your remaining concerns directly and thoroughly. We eagerly await your further feedback on our responses.

Best regards,

The Authors

Reviewer yfx6

Q1. Error of Reward Model & Amplification Factor

We make statistics of reward estimation errors, and the results are shown in Fig. S1 of the uploaded one-page PDF file. Here we make a table to briefly show the results. Although the proportion of very large error samples is not large, the incremental performance of our method may lie in a more suitable choice of amplification function, as evidenced by the table in our response to the Q2 from the reviewer Hftn.

Sorry for causing the confusion of the reward range. Actually, the x-axis of Fig. 2 of the manuscript is normalized. The range of $||z||V^{-1}$ lies approximately between 1.5 and 14. Thus, the range of the amplification factor is approximately between exp(0)=1 and exp(0.625)=1.86.

Q2. Reward Model

We have experimentally validated the consistency between the proposed method and the BERT score [1]. As shown in the Fig. S2 in the uploaded one-page PDF file, the deeper color indicates the larger errors in the BERT score. We can see that there are plenty of samples whose BERT scores are unrelated to human labeling even contradictory, shown as the upper left or lower right regions. We also provide some examples scoring by our reward model in Fig S4 in the uploaded one-page PDF file, the assessment of inpainted images is closely aligned with human preference. Thus, our proposed human-labeled dataset is necessary for the alignment of inpainting tasks. Note that our dataset is labeled by professional data annotation company. We will add more discussion with [1] in the final version. Thanks for the advice.

[1] Black, Kevin, et al. "Training diffusion models with reinforcement learning." arXiv preprint arXiv:2305.13301 (2023).

Dear Reviewer yfx6,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript and your favorable recommendation. We understand that you may be reviewing multiple papers and have a busy schedule. In our previous response, we made sure to address your remaining concerns directly and thoroughly. We eagerly await your further feedback on our responses.

Best regards,

The Authors

Reviewer Hftn

Q1. Differences from T2I Methods

We confirm that our method is NOT simply applying the text-to-image alignment scheme to image inpainting. The technical novelty of the proposed method primarily lies in modeling reward accuracy and adaptively controlling the regularization reward strength, which has not been investigated by T2I methods before. Moreover, the ablation studies presented in the right part of Table 3 in the main paper validate the superiority of our design, which can achieve a +106% speed acceleration while maintaining high performance (even increasing WinRate by up to 1.7%). We will add further discussion comparing our method with current T2I methods. More importantly, we want to note that this is the first time to explore the diffusion model human preference alignment problem on the task of image inpainting. To enable the feasibility of this task, we built the first dataset, where inpainted images are labeled with human preference through a professional data annotation company. Without such a dataset, it is almost impossible to conduct this work. We believe it will advance this field. Our dataset holds significant research value and application scenarios in several related areas, including image quality assessment tasks and other tasks involving human preference. We also provide some examples scored by our reward model in Fig S4 in the uploaded one-page PDF file, the assessment of inpainted images is closely aligned with human preference.

Q2. More Explanations of Eq. (11)

Selection of Hyper-parameters. The critical thing for pasteurization is the region of the weight factor $\gamma$. In the left of Table~3 (e) and (f) of our manuscript, we experimentally validate the two sets of $k=0.05,b=0.7$ and $k=0.065,b=0.9$. The experimental results show that our selection is better than the other settings. Moreover, please refer to our response to the third question, where we detailedly investigate the different hyper-parameters and weighting functions.

Implementation details. We first prepare the matrix $\mathbf{V}^{-1}$ by calculating $\mathbf{V} = \mathbf{Z}^T\mathbf{Z} + \lambda \mathbf{I}$, where $\mathbf{Z}$ represents the concatenation of feature embeddings before the last MLP layer of the reward model, encompassing all training dataset samples. During the diffusion training process, for each sample, we obtain a feature $z$ from the reward model. We then calculate $\gamma$, which serves as the weight factor for the final RL loss, adaptively adjusting the magnitude of the gradient. Thanks for the comments; we will include the corresponding content in the final version.

Other Functions & Ablation Study. To further resolve your concern, we have applied another linear function to parameterize $\gamma$ as table below. Specifically, $\gamma = -1.9*\| \boldsymbol{z}\|_{\mathbf{V}^{-1}} + 0.06$. The experimental results indicate that the exponential function provides the best regularization effect. In contrast, the linear function and static constant do not fully exploit the regularization effect of the reward upper boundary.

Q3. Weight Range

Since RL requires a reward value to guide the training direction, we must combine scores to decide whether to push the diffusion model away from the reconstruction sample or bring them closer together. We determine these weights using our scoring scheme. Specifically, the first two scores focus on partial aspects, such as structure and texture, respectively. The third score, however, reflects the overall impression, which is more comprehensive. Therefore, it is reasonable to assign lower weights to the first two metrics and a higher weight to the final metric. Additionally, to further address your concern, we had the labeling supplier company directly rank these samples to validate the consistency between our weighted combination of scores and the human labeling rank. Specifically, the rank of several reconstructions by both human experts and previous label weighting schemes. Then, we calculate the consistency between the human ranking and the weighted score ranking. The experimental results are shown in the following table. The proposed weighting combination scheme is validated by its high consistency with the human scoring scheme. The scoring system is shown in Fig. S6 of the uploaded one-page PDF file.

Q4. Meaning of the Metric "Rank"

“Rank” in Table 2 indicates an average order (from the best to the worst) of all metrics. Thus, a lower rank indicates better performance.

Q5. Dataset Details

Note that the labeling is provided by a professional data annotation company. All annotators are professionals trained in similar tasks. The annotation time for each pair (3 samples) averages about 2 minutes. A total of 24 annotators were employed. We will provide more details in the final version.

Q6. Limitations

We have indeed discussed some limitations in the final paragraph of the conclusion section.

Dear Reviewer Hftn,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript and your favorable recommendation. We understand that you may be reviewing multiple papers and have a busy schedule. In our previous response, we made sure to address your remaining concerns directly and thoroughly. We eagerly await your further feedback on our responses.

Best regards,

The Authors

Reviewer 85gD

Q1. Difference between the Proposed Method, Common Reinforcement Learning, and Supervised Fine-tuning

In the following, We first clarify that our method is NOT a straightforward practice of the common reinforcement learning on diffusion model alignment. Then, we address your confusion about supervised fine-tuning (i.e., directly tuning on high-quality data) and reinforcement learning-based tuning.

(1) Not the same task as text alignment RL task. While some related works exist in the area of text-to-image tasks involving human preference, our work is the first to introduce reinforcement learning (RL)-based alignment into the task of image inpainting. Initially, our proposal of the human preference inpainting dataset enabled the feasibility of this task. Without such a dataset that includes high-quality labels based on human preference, undertaking inpainting alignment would be impossible. Note that our dataset is annotated by professional data annotation company. The technical novelty of the proposed method, compared to conventional RL, primarily lies in our modeling of reward accuracy and the adaptive control of reward regularization strength using the reward error upper bound, as detailed in Secs. 3.2 and 3.3 of our paper.

(2) Supervised fine-tuning cannot address our task. We first clarify some experimental results that our baseline model (the pretrained diffusion model) has already trained on plenty of GT samples for image inpainting, which is even better than our high-preference samples. However, the limited performance of the pretrained model, with comparison of Runway and Ours in Tables 1 and 2 of our paper, indicates the supervised fine-tuning model is hard to accurately and effectively modifying model generation on the perspective of preference alignment.

Moreover, to further address your concerns, we have experimentally validated the neural network performance using fine-tuning based methods.

where (a) signifies the high-quality subset generated from the original samples and selected by our reward labels; (b) represents the reconstructed images by the fine-tuned model; (c) corresponds to the fine-tuned model's performance on the validation dataset; (d) indicates the generated samples by the RL-based model evaluated on the validation dataset. We observe that the fine-tuned model performs not well on the training set and shows even more undesirable performance on the validation set. In contrast, the proposed method accurately aligns with human preferences.

Q2. Comparisons with Additional inpainting Methods

We have experimentally compared with your mentioned papers. For all these methods, we assessed performance using their publicly released models. We utilized our test dataset to comprehensively evaluate their performance. As shown in the table below, our method significantly outperforms all the compared methods.

Moreover, we also tested the Winrate compared with our baseline model (Runway). Our method also greatly surpass all compared methods, which validates the effectiveness of the proposed RL-based alignment scheme.

Q3. Size of Training Dataset

We believe you have a misunderstanding here. We clarify that the dataset is primarily used to train the reward model rather than the diffusion model. With an accurate reward model, we can easily train diffusion models on scaled datasets by allocating more prompts. We empirically validate that the dataset size is sufficient for training a reward model. Specifically, as shown below, we train the reward model with 10K, 20K, 30K, and 50K data samples, respectively.

We observe that the accuracy of the reward model becomes saturated as the size of the training dataset increases. Therefore, we believe that 50K data samples are sufficient for training an accurate reward model. Additionally, the proposed method has already outperformed SOTA, validating its effectiveness.

Dear Reviewer 85gD,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We understand that you may be reviewing multiple papers and have a busy schedule. In our previous response, we made sure to address your remaining concerns directly and thoroughly. We eagerly await your further feedback on our responses.

Best regards,

The Authors