UniFL: Improve Latent Diffusion Model via Unified Feedback Learning

Thanks for your kind words about our good writing, sufficient experiments, and the effectiveness of our method. We would like to answer the proposed questions in the following:

1. Minor Typos: Thanks for the suggestion, we will modify these places in our revised version.
2. Selection of hinge coefficient $\alpha_d$: Please refer to the Author Rebuttal part and Fig.4 of the global rebuttal PDF for more details.
3. Impact of different aesthetics during generation: The impact of different aesthetic reward models in the generated images can be clearly observed in Fig.3 in our main paper. As can be seen, with the integration of color aesthetic rewards, the colors of the images generated by UniFL are more harmonious and natural. This enhancement is particularly pronounced in the acceleration scenario, where LCM and Turbo exhibit color deterioration characterized by darker, grayer, and vaguer, contrasting with our method's maintenance of a lively color palette. The effects of the detail and layout reward model are also notable. For instance, when considering the prompt A bloody may cocktail, the image generated by UniFL showcases intricate background details while maintaining a well-balanced layout with the foreground, resulting in a visually appealing composition. Furthermore, the lighting aesthetic reward ensures the lighting with an atmosphere in the image, as opposed to the flat center background lighting observed in alternative methods.
4. Effectiveness of User Study: Due to the subjective nature of the image quality judgment, the most effective way to evaluate the performance of a model in the context of text-to-image generation so far is still the user evaluation. This is also the standard practice of various classical text-to-image works such as SDXL[1], SDXL-Turbo[2], etc. Moreover, to increase reliability, we involve a considerable number of users(10 users) in our quality evaluation study, which we believe can validate our conclusion relatively accurately.
5. Acceleration ablation with DPO/ImageReward: We did not choose to compare the accelerated version of our method with the SDXL-IR and SDXL-DPO as they are not the methods tailored for inference acceleration instead of generation quality optimization. There is no few-steps optimized version of SDXL of ImageReward and DPO for fair comparison with our method of inference acceleration. Even though, the unaccelerated version SDXL optimized via our approach still displays more superior image quality than these methods as evidenced by our experiments.

We hope our rebuttal can address your concerns.

[1] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

[2] SDXL-Turbo: Adversarial Diffusion Distillation

Thanks for your kind words about our well-motivated method, sufficient experiments, and good writing. We would like to answer the proposed questions in the following:

1. Visualization on T2I alignment after acceleration: We visualize the text-to-image(T2I) alignment performance of SDXL accelerated via our method on the widely used benchmark prompt set - DrawBench, and provide some results in Fig.3 of the global rebuttal PDF. It clearly demonstrates that, even with a reduced number of inference steps, our approach maintains superior text-image consistency under various circumstances after acceleration, including attribute binding, object counting, and counterfactual prompt inputs.
2. Explanation of the unexpected generation: It should be noted that although it is not what the user expected, there is nothing wrong with the model to generate this black tie, as the input prompt does not explicitly specify that a tie is undesired and all the stuff mentioned in the prompt(e.g. cream-colored labradoodle, glasses, black beret. etc) has already correctly generated. In other words, it is free for the model to generate additional items in addition to the input prompt specified. One possible reason for the model to generate the black tie in this case is the data bias residing in the training samples. That is, there are vast training examples of teachers lecturing at a blackboard, where most teachers in these examples all wear ties. Such co-occurrence bias makes the model easily generate an extra tie when comes to the character in front of the blackboard. A possible solution is to input an additional negative prompt to suppress such undesired behavior.

We hope our rebuttal can address your concerns.

Thanks for your kind words about our writing and contribution to the feedback dataset curation. We would like to answer the proposed questions in the following:

1. Comparison with ReFL: There are significant differences between our method and ReFL. Specifically: 1) Given the input prompt, ReFL starts with pure noise to obtain the denoised image and then imposes the reward scoring. This limits their usage to the reward models pre-trained on the collected preference data for the reward guidance. In contrast, we incorporate an extra image condition for noise initialization and use the perceptual label of this condition image for feedback supervision on the denoised image. Such formulation allows us to leverage the prior knowledge of a wide range of existing perceptual models to perform feedback fine-tuning. 2) In ReFL, a single coarse-grained preference reward model is applied, which may encounter the rewarding conflict, resulting in insufficient fine-tuning. As a comparison, we develop multiple decoupled aesthetic reward models, which enable more effective fine-grained aesthetic preference alignment. 3) A core finding exhibited in ReFL is that it can only be applied to later denoising time steps during fine-tuning with the reward model as the reward model cannot reward correctly in the early steps. By contrast, we introduce adversarial feedback supervision, which effectively removes this limitation on optimization steps, allowing us to apply feedback fine-tuning to any denoised time step, thereby achieving model inference acceleration. Overall, these novel designs make our approach distinct from ReFL and yield a more flexible and effective feedback fine-tuning framework for LDM.
2. Fine-tune concepts limitation in PeFL: On the one hand, although we apply the SOLO instance segmentation model trained on the COCO dataset(80 categories), we observe that PeFL exhibits exceptional generalization capability and the generation performance of many concepts not shown in the COCO dataset is also boosted significantly as shown in Fig.2 of the global rebuttal PDF. We believe that the LDM can be guided to learn general and reasonable structure generation via PeFL optimization. On the other hand, the proposed perceptual feedback learning is a very general module, and it is very straightforward to replace the close-set model SOLO with other open-set instance segmentation models such as Ground-SAM to achieve further improvement for the concepts in the wild(e.g. concepts in LAION dataset).
3. About the performance in Tab.1: We argue that our method demonstrates notable improvements compared to other approaches as outlined in Tab.1. It is essential to emphasize that different metrics possess unique properties, and absolute enhancement should not be the sole criterion for evaluation. For instance, the inherent limitations in the fine text-to-image alignment of the CLIP model make it challenging to achieve significant advancements in CLIP scores as it cannot capture the nuanced content change. Consequently, even with tailored text-to-image alignment feedback fine-tuning, ImageReward achieves a mere 0.004 CLIP score improvement with SD1.5. Therefore, it is more reasonable to asses the relative improvements. Specifically, UniFL obtains 2$\times$ CLIP score improvement over SD1.5 base(0.005 vs 0.01) than the second best method(i.e. SD15-DS), which showcases the superiority of our method. There is also a similar case of other metrics, in which UniFL obtains relatively more notable improvement upon the base model. Moreover, our method also displays an obvious advantage over other methods by the extensive user study.
4. About multiple perceptual aspects optimization with PeFL: We show the results of multi-aspect optimization on style and structure through PeFL in Fig.1 of the global rebuttal PDF. As can be seen, incorporating these two distinct objectives does not hurt the effectiveness of each other. Take the prompt a baby Swan, graffiti as an example, integrating the style optimization via PeFL upon the base model successfully aligns the image with the target style. Further integrating the structure optimization objective retains the correct style and exhibits more complete structural details(e.g. the feet of the Swan) simultaneously.

We hope our rebuttal can address your concerns.

Thanks for your kind words about the design of perceptual feedback learning and decoupled feedback learning in our method. We would like to answer the proposed questions in the following:

1. Clarification on unified design: We claim our method is a unified design as all the modules are seamlessly integrated under the unified feedback learning framework. Specifically, under this framework, we developed different reward mechanisms (namely perceptual reward, decoupled reward, and adversarial reward), which are effectively combined to mitigate the problems of low fidelity, poor appeal, and inefficient inference of existing LDMs simultaneously at the first time via the unified rewarding then fine-tuning paradigm. In contrast, existing works exhibit high heterogeneity in the way(e.g. changing network structure, or altering the sampling scheduler) to address these defects of LDM, making it challenging for these methods to be well combined to achieve comprehensive improvement.
2. Additional image condition: We emphasize that the introduced additional condition of PeFL refers to the image content incorporated into the noise initialization, instead of the extra control image input like in ControlNet. Specifically, we highlight the perceptual feedback learning(PeFL) with additional image input in Fig.1(left) via the blue solid line. As depicted in Fig.1, this image is first injected noise after encoded(the injection process is not illustrated in Fig.1 for clarity, but detailed in Algo.1), then sent to the diffusion model for denoising, the denoised image is then supervised by the perceptual label (instance/semantic seg mask) of the input image. As a comparison, below the input image for PeFL, the decoupled feedback learning starts with pure noise as indicated in the yellow solid line.
3. More instantiation of PeFL: We have presented two more case studies of perceptual feedback learning with other types of perceptual models, including the semantic segmentation model(i.e. DeepLab-V3) and the style extraction model(i.e. VGG-16) in the Appendix attached to our main paper. The results(Fig.8 and Tab.2 for style, Fig.9 and Fig.10 for layout) demonstrate that PeFL can effectively exploit the prior image knowledge embedded in these expert models to boost the performance of style generation(increasing the style response rate) and semantic layout optimization(more preferred layout).
4. Loss formulation in Eq.5: We choose the hinge loss instead of the winner-loser loss in Eq.5, as it describes the reward-tuning procedure, wherein there are no winner and loser samples for pair-wise loss calculation. Specifically, in this process, a sample is generated from the pure noise given a prompt and then scored via the already well-trained reward models. Eq.5 then encourages the generated image to gain a higher reward score, and the hinge coefficient helps to avoid the unbounded increasing reward, preventing the LDM overfit to the reward model. The winner-loser loss is only used in the reward model training stage, where the reward model is trained to increase the score of the preferred sample while lowering the unpreferred one, aligning with the human preference.
5. Preference data annotations: We provide some examples of the collected annotated preference data in Sec.B.1(Fig.11) in the Appendix of our main paper.
6. More details of active prompt selection: We provide more details of the semantic-based prompt filter and nearest neighbor prompt compression during active prompt selection in Sec. B.2 in the Appendix of the main paper. In short, the semantic-based prompt filter selects the prompts owning rich semantic meaning by the ruled-based filtering, while the nearest neighbor prompt compression is designed to suppress the redundant prompts by checking the semantic similarity in the embedding space to ensure diversity.
7. Selection of hinge coefficient $\alpha_d$: Please refer to the Author Rebuttal part and Fig.4 of the global rebuttal PDF for more details.
8. Analysis on Adavasarial Feedback Learning: We provide a detailed analysis of the effect of adversarial feedback learning on acceleration in our ablation study section (L316). In summary, incorporating adversarial feedback plays two roles: (1) With the diffusion model and the adversarial reward model updated adversarially, the diffusion model is not prone to overfitting. The synchronously updated discriminator forces the diffusion model to evolve continuously, enjoying the reward guidance for a longer duration; (2) The denoised images under the low inference steps are forced to be clearer via the strong adversarial objective and thus can be correctly rewarded by the aesthetic reward model. Given these two benefits, the generation quality of low-step images steadily improved via the reward guidance, achieving inference acceleration.

We hope our rebuttal can address your concerns.

Dear Reviewers,

As the discussion phase is ending in one day, it would be helpful if you could respond to the authors' rebuttal if you haven’t already. This would allow the authors to address any remaining concerns.

Thanks,

AC