FABRIC: Personalizing Diffusion Models with Iterative Feedback

We thank the reviewer for the accurate summary of our method and for the insightful and constructive comments. In the following, we would like to offer our perspective on some of the points raised.

• Limited Expansion of Distribution

  It is true that FABRIC does not necessarily expand the conditional distribution of generated images, especially when the feedback images are sampled from the same model, but this is generally the case for interventions that improve quality. Indeed, in order to improve quality it is necessary to constrain the distribution by emphasizing desirable and cutting out undesirable parts. Still, it is possible to reintroduce some diversity. For example, by adding feedback images from an external corpus in addition to the ones from the current generation, one can add more diverse images, guiding the generative process in new directions. This can even be automated by retrieving images from the external corpus that are similar to the prompt or existing feedback images (e.g. using CLIP similarity). Additionally, by adding random alphanumeric characters to the prompt it is possible to artificially increase the diversity of the generations [1]. We have added clarifying comments about this and the following two points to Section 6 of the manuscript.

• Feedback Loop Limitation

  Similar to the previous point, this is more a limitation of our experimental setup rather than a fundamental limitation of FABRIC. Indeed, it is possible to move the balance of the exploration-exploitation trade-off in the direction of the former with very simple extensions to FABRIC such as the addition of a retrieval corpus. More sophisticated methods might also take the prompt into consideration, by rephrasing the prompt for increased diversity or by iterating on the prompt and taking prior feedback into account, but we leave exploration of this issue to future work.

• Diversity Collapse

  We agree with the reviewer that in our experimental setup FABRIC clearly suffers from diversity collapse. However, we would like to point out that this is not necessarily the case in a general setup. In order to automate the process of evaluation, we were only giving images from the ones generated during previous rounds as feedback images. There, naturally (and hopefully), the process has to converge to a result very similar to the target image. In practice, the user can select feedback based on a variety of criteria, hence producing more diverse feedback images and (anecdotally) more diverse results. Additionally, we would like to emphasize that of all the methods we investigated, FABRIC was able to elicit the highest increase in quality metrics for the lowest decrease in diversity (see Section 4.3 on the quality-diversity trade-off), comparing favorably to both finetuning and LoRA training.

• Binary Feedback System

  The binary nature of feedback is to a large degree imposed by the binary nature of CFG. Still, scalar ratings can be approximated with minor modifications to FABRIC: One could choose different feedback strengths based on the rating, with the strongest rating receiving the highest feedback strength and the weakest rating having a feedback strength of 0. While we did not evaluate this extension, it is easy to implement on the application side. The main reason why this was excluded from the evaluation is because it goes against the original motivation of considering sparse binary feedback, in addition to making the experiment design more complicated and comparison to baselines more challenging. We added a comment to Section 6 in order to clarify this.

• Lack of Detailed Feedback

  Even though we did not evaluate it experimentally, it is possible to provide a textual description in order to steer the feature extraction process. Namely, we use the null prompt for extracting attention features from the feedback images in our experiments. However, one may use an arbitrary prompt for each feedback image, describing for instance the specific aspects they like/dislike. This would significantly complicate the experiment design, as one would have to find a way to automatically generate textual descriptions of the feedback. In addition, it again goes against the aim of reducing the amount of prompt engineering required to achieve desired results by just shifting the prompting to a different stage of the process. We also note that attention masking is another simple extension that could be used to constrain the feedback to certain parts of the image (similar to inpainting with Stable Diffusion). We have added a clarifying comment about this to Section 6.

References

[1] Deckers et al. 2023. Manipulating Embeddings of Stable Diffusion Prompts

We thank the reviewer for the excellent summary of our contributions and for the concise, constructive feedback. We agree that certain sections of the writing could be improved and have uploaded a revised version of the paper. In particular, the methods section has been majorly overhauled, hopefully addressing your concerns. We refer to the general comment for a more detailed description of the changes that have been made.

We thank the reviewer for the valuable feedback and suggestions to improve our work. In the following, we would like to address your concerns to the best of our ability and answer any open questions.

• Limited technical novelty

  While we agree that the core conditioning mechanism isn’t novel and has been proposed by the authors of ControlNet [1], the combination with weighted attention and CFG makes for a very versatile and flexible algorithm even beyond the scenarios that were evaluated experimentally. For example, real-valued ratings can be incorporated by varying the feedback strength according to the magnitude of the rating and using it as positive/negative feedback depending on the sign. Textual descriptions as well as masking of certain parts of the image can also be supported by varying the prompt used for extracting attention features or by masking out certain keys and values.

  However, both of these extensions go against the core motivation, which is to alleviate the need for prompt engineering by making the best possible use of sparse binary feedback. In addition, it would have further complicated experiment design, making the results harder to reproduce and more difficult for future work to compare against. This was somewhat unclear from the writing and we have added clarifying comments to Section 6.

• Lack of human rating in a paper focused on iterative human feedback

  We agree with the reviewer that studying human interaction with the system would certainly be insightful. Doing a user study was considered, but we ultimately decided against it due to the challenges involved with the design and execution of such a study. To illustrate: A naive study design would simply let the user try out the system and subsequently have them rate their satisfaction and the perceived quality of generated images. This, however, leaves many variables uncontrolled. To name a few: How much of the perceived quality comes from the base model as opposed to the improvements made by FABRIC? How much does generation time (which is noticeably higher for FABRIC) impact user satisfaction and perceived quality? How much does the user interface influence the results? How big is the learning effect from using the system and how much skill is involved in achieving desirable results?

  As a consequence, if not executed properly, the results of such a study would amount to little more than anecdotal evidence (which we now have added to Section 4, Experiments) and might prove difficult to reproduce due to the number of variables that can impact the final rating. Considering all of this, rather than drawing conclusions from potentially weak evidence, we believe that it is appropriate to leave the empirical analysis of human interaction with FABRIC for future work.

• Missing discussion of prior work

  These two papers, prompt2prompt and instruct-pix2pix, are indeed related to our work, as they use similar techniques but have different goals. We thank the reviewer for pointing this out and have added a paragraph on image editing to Section 5 (Related Work).

• Comparing HPS and PickScore

  The reviewer is correct that HPS and PickScore are very similar and solve the same task, which is human preference estimation. In fact, in the early phases of the project, we were using HPS as the main evaluation metric but decided to replace it with PickScore when that was published since it demonstrated superior accuracy thanks to a larger training set (see the table below), which makes for a more accurate human proxy. We continue using the HPS LoRA as a baseline model explicitly trained to maximize human preference, which is ultimately also what PickScore measures. We note that FABRIC outperforms HPS LoRA whether measured by HPS or PickScore (early experiments used HPS).

  Method	#prompts	#choices	LoRA of SD available
  HPS	25k	25k	yes
  PickScore	38k	584k	no

• Systematic comparison of related work

  We like the idea of adding a systematic comparison of methods which incorporate human feedback in the generation process of diffusion models, even beyond just the techniques using attention-injection, but unfortunately we couldn’t find space to include it in the paper. Instead, we’ll just attach it here and possibly add it to the Appendix if it is considered a valuable addition. The table is provided in the next comment due to the character limit.

Finally, we would like to ask for clarification why the reviewer, despite rating the soundness and presentation of the work as good and the contribution as excellent, ultimately deems the paper to be below the acceptance threshold. Are there significant flaws that prevent acceptance of the paper in its current state but that could be addressed or is there a more fundamental issue?

References

[1] https://github.com/Mikubill/sd-webui-controlnet/discussions/1236