FineStyle: Fine-grained Controllable Style Personalization for Text-to-image Models

Thank you for the insightful comments and suggestions! We appreciate them and will make the necessary revisions to ensure our work is presented with better accuracy and completeness.

W1: As detailed in Section 3 of our paper, Muse employs a cascaded design of generative modules. FineStyle only adapts the lowest resolution model, which operates at a resolution of 256x256 and comprises 3B parameters. This model features a masked generative image transformer that serves as the image decoder and a VQGAN for converting images from pixel to latent space and vice versa. The VQGAN encodes a 256x256 image into a sequence of 16x16 latent visual tokens for the image decoder to consume.

W2: Thank you for your valuable feedback regarding the citation of related works. We acknowledge the oversight in not sufficiently referencing prior studies that have utilized parameter-efficient fine-tuning in the domain of text-to-image models. Unlike the typical application of LoRA layers across each transformer block of the image decoder, our method employs a singular main LoRA layer but modifies it with distinct biases for each transformer block. This adaptation not only reduces the number of trainable parameters but also aims to mitigate potential overfitting issues, a critical aspect in maintaining model generalizability. We will carefully revise the relevant sections of our paper to include these references and articulate the different design choices of our adapter compared to LoRA.

W3: See Figure 2 of the attached PDF. We showcase some results from SD3.

W4: Although iterative human feedback can enhance style fine-tuning performance by expanding the training dataset with human-annotated synthetic images, this approach is cumbersome, risky, and time-consuming. The limitations of human feedback include: Human intervention during the training process is necessary to curate a set of synthetic images, which incurs significant labor costs. The effectiveness of using synthetic data for iterative human feedback depends critically on the quality of the synthetic images produced by the style fine-tuned model. If the images do not meet a certain standard, performance may deteriorate. In practice, even the most advanced text-to-image models can struggle to accurately replicate the compositions of certain reference style images. The time required for labeling through human feedback is contingent on the quality of the synthetic training data; poorer quality requires more time. Regarding timing, StyleDrop, with one round of iterative human feedback, takes at least three times as long as FineStyle with additional labor costs.

Q1: The answer to checklist point 14 should have been Yes. We apologize for the mistake. For checklist point 15, given that the paper is approved for submission, I assume that the inclusion of human preference study is also approved.

Q2: As demonstrated in Figure 3(d), using cross-attention weights of the pre-trained model as segmentation map might not be accurate like the bottom right corner outliers. We do notice that it can get worse for non-natural domains, possibly resulting in reduced fine-grained controllability. We expect to explore the potential of fine-tuning a pre-trained model to improve its cross-attention alignment between visual and textual tokens corresponding to the same concept as future works, which could land a better foundation to answer this question.

Q3: This is a valid point, since outputs of T5-xxl are fed right into cross-attention layers modified by our kv adapter. However, the downside is T5-xxl actually contains more parameters than Muse image decoder, 4.6B v.s. 3B. Tuning the full text-encoder is computation prohibitive for most users. We would like to study light-weighted adapter for text-encoders in future works.

Q4: FineStyle can effectively process simple style images based on straightforward descriptions. For example, the second and third rows of Figure 12 show the styles “rainbow color flowing design” and “bookshelf in watercolor painting”, which are simple concept compositions.

Q5: See Author Rebuttal and Figure 3 of the attached PDF.

Q6: We tried using a style descriptor, e.g. “in watercolor painting style.”, as a prior-preservation anchor. It can mitigate concept collapsing to some extent, but doesn’t show concrete evidence of improving fine-grained style controllability. Unlike prior-preservation loss that requires auxiliary training data, FineStyle does not require any other image but the one style reference image.

Q7: Correct, the FineStyle adapter operates as a standalone layer with its parameters. It is designed to be dynamically attached or detached from the base model as needed.

Q8: We apologize for the confusion with the use of the double negative for listing “tree WITHOUT christmas decoration” under unwanted concepts. We will rephrase this to be listed as “bare tree” under unwanted concepts. The first example is NOT a failure as based on the synthesis prompt we expect generated trees to contain a form of christmas decorations. In contrast, StyleDrop mode collapses to the exact pine tree of the reference image.

Q9: We don’t use negative prompting during inference. The "Unwanted concept" column in Figure 4 is designed to demonstrate concepts that are implicitly inferred from the synthesis prompt. Our preliminary experimental findings have shown that negative prompting at inference time does not effectively counter mode collapse, which actually prompts us to seek an efficient fine-tuning strategy that can better disentangling style and visual elements of a subject.

I acknowledge your reply and truly appreciate the efforts you took to explain each and every point in detail. I truly respect your willingness to make your work open so that the entire community can benefit from it. I think it's safe to say that it was because of open-source codebases like LDM [1], the diffusion community, in particular, progressed very rapidly over the past few years. So, your commitment towards open-source is quite endearing and I am sure it will be helpful.

I have nothing further to add and I offer the authors my best wishes.

References

[1] https://github.com/CompVis/stable-diffusion

Thank you for your comments, particularly regarding the need for comparison with extra baselines. We appreciate the opportunity to add new baseline results and clarify the advantage of our method.

W1: See Figure 1 of the attached PDF. We include results of DreamStyler, StyleAligned, and IP-Adapter. To the best of our knowledge, there are no openly available codes for ControlStyle. We use SDXL as the base model for StyleAligned and IP-Adapter, while DreamStyler only supports SD 1.5. Both StyleAligned and IP-Adapter show the symptoms of content and structure leakage. The bay and the mountains keep appearing in the first row, even if not in the prompts. For StyleAligned, the bay water is fixed at the bottom left of the generated images. In the second row, IP-Adapter hallucinates mountains and the moon from the reference. In the third row, IP-Adapter almost copies the texture and shape of the house, which doesn’t look like an office building. StyleAligned correctly follows the semantics for the third and fourth rows but doesn’t quite follow the target styles. On the other hand, Dreamstyler fails in both semantic and style consistency and is of lower quality, which is partially due to the less powerful model SD 1.5.

W2: Our method utilizes detailed text descriptions to empower users to control fine-grained styles. This is a key distinction from methods like StyleAligned, IP-Adapter, etc., which interpret the styles of an image as a whole. Although crafting detailed descriptions may require an extra step, this provides users additional fine-grained control in text-to-image generation over how much granularity in a prompt they want rather than requiring exhausting every detail of a reference image. Thus, users can focus on which style aspects they wish to capture, making our method versatile to different user needs. Furthermore, detailed descriptions can also be beneficial in other methods requiring image inversion, such as StyleAligned. As in Figure 4 of the attached PDF, when a simple inversion prompt (a) is "a tall house in watercolor painting style.", gable roofs of the reference leak into synthesized images, which is uncommon for an office building. In contrast, when a detailed prompt (c) containing "gable roof," etc., is used., the problem is improved. We also use a control prompt (b) containing a "<random token sequence>" to make the length of the tokenized sequence the same as (c). This demonstrates it's "gable roof" in the detailed descriptions that improves the situation instead of any random, longer description.

W3: We acknowledge the discrepancies in certain results between StyleDrop and FineStyle in terms of style learning. In the Eiffel tower example of Figure 6, StyleDrop exhibits a darker blue tone, whereas FineStyle presents a lighter one. In fact, the reference image features a range of tones from dark to light, evident in various elements such as the bookshelf, plant leaves, pot, and laptop. This tone spectrum informs the lighter blue color in FineStyle. While StyleDrop tends to present a darker tone inheriting from the laptop, it does not capture the legit composition of the reference, particularly in the Eiffel tower example with recurring appearance of leaves.
As in the quantitative human evaluation of Table 2, although FineStyle scores lower on Style than StyleDrop, it has a significant edge over StyleDrop in Text and Structure/Common Sense scores. In our human study, we found that out of the image pairs where StyleDrop was deemed as having a superior style, 73% of the time raters found the FineStyle image to follow the prompt the best, 27% percent of the time both methods followed the prompt equally and 0% of the time did the StyleDrop image get rated as better at following the prompt. The same is found when looking at the axis of compositional structure and common sense of the generated images. Out of the image pairs where StyleDrop was deemed as having a superior style, 81% of the time raters preferred the FineStyle image for structure, 19% percent of the time both methods both had equally good structure and 0% of the time did the StyleDrop image get rated as better at structure and common sense. We think that StyleDrop sacrifices composition for style which can often create images that are far from the intended prompt. These findings underscore that FineStyle is a better method with balanced style learning and composition, with fine-grained style controllability.

W4: In our study, we recruited 14 participants, each of whom evaluated 24 image-text pairs of distinct styles. This resulted in a total of 336 evaluations, ensuring a robust dataset for evaluation. Regarding your observation about the sum of user preference proportions in Table 2 not equaling 1, the proportions indeed sum to 0.998, 0.999, and 0.999 across different comparisons. This slight discrepancy arises from the presentation without rounding the figures to maintain numerical precision. We recognize the need for clarity and will address this in the revised version of our paper, ensuring all totals are rounded appropriately.

W5: Regarding the inference speed, it is true that our method incorporates an additional 0.15M adapter parameters compared to StyleDrop. However, when considered in the context of Muse's overall 3B parameters, this increase is relatively minor. Consequently, FineStyle maintains an inference time that is comparable to that of StyleDrop, ensuring that the enhanced control over style elements does not come at the cost of efficiency.

Thank you for your insightful and detailed reviews. We appreciate the opportunity to address the weaknesses and questions raised and to clarify aspects that may have been unclear.

W1: The pre-training and human feedback datasets exhibit significant size disparities, which leads to differing methodologies in concept learning. Pre-training with a large dataset requires a vast amount of time and resources for a careful curation. On the contrary, style fine-tuning with human feedback requires significantly less time and resources to collect a few images of the same style yet of various contents to learn a model with a custom style. Nevertheless, this requires human intervention for every new style, which could be cumbersome. We believe that these two regimes (large once-for-all fine-tuning vs a few-shot case-by-case fine-tuning) exhibit a trade-off. In this paper, we aim to match the training efficiency and remedy the limitations of human feedback dataset with proposed data-scaling and concept-weighted loss strategy.

W2: In our approach, we not only scale up the single text-image pair to be multiple pairs with each one having a concept-oriented text, but also re-weight the loss computation by concept-oriented masking, detailed in line 152. This strategy boosts the learning of associations between text and image concepts by the gradient flows stemming from concept-oriented text and masking. Utilizing masks derived from cross-attention weights naturally complements the use of a kv adapter over a feature adapter. According to Table 1, the combination of data scaling and the kv adapter yields the highest clip text scores. Concept division is efficient in both time and computation. Upon completing the writing of a prompt, a user can quickly annotate desired concept words within the text, typically requiring only a few seconds. Subsequently, the concept mask is generated by a single, zero-mask pass through the Muse model, during which the attention weights corresponding to the annotated concepts are retrieved.

W3: Theoretically, data scaling with concept-oriented masking works better with kv adapter due to the intuitions and facts: concepts are activated during cross-attention between visual and textual tokens. Furthermore, as demonstrated in Table 1, the strategy achieves the best clip text scores.

W4: Concept data scaling and concept-oriented masking are designed to complement each other effectively, as discussed in the response to W2.

Q1: In practice, human feedback is undesirable due to human intervention requirements. While it can enhance style fine-tuning, it comes at a prohibitively high cost and a risk of performance deteriorating with low-quality synthetic data. For a fair comparison, we have excluded the human feedback component of StyleDrop, recognizing that including human feedback in our comparative analysis could obscure the individual contributions of each component. In this work, our goal is to develop an efficient method that facilitates improved concept learning without the need for costly human feedback.

Q2: For Figure 6, the reference image features a range of hues from blue to purple, evident in various elements such as the bookshelf, plant leaves, pot, and laptop. This spectrum informs the lighter blue color of the Eiffel Tower in the FineStyle. While StyleDrop tends to present a darker tone, it does not capture the legit composition of the reference, particularly in the Eiffel tower example with recurring appearance of leaves.
Regarding Figure 7, a similar principle applies. The oil painting style produced by FineStyle, though different, draws from the diverse stylistic cues within the reference. Theoretically, enhancing our style dataset with a broader range of subjects and finely annotated concepts could allow for more precise control over such discrepancies through tailored prompts. This suggests a need for improved methods in explicitly regulating each concept component in our KV adapter, which could address the issues you highlighted.

For W3: I hope the authors provide visualizations to prove that the kv adapter truly focuses on the conceptual content (as mentioned in their paper) rather than performance enhancements brought about by data augmentation methods and conceptual masks. I noticed in Figure 6 that the FineStyle generated an Eiffel Tower with a blue circle even without the prompt word 'circle'; and in Figure 4, parts of a mountain appeared. However, the authors have ignored this point and have not responded to my question.

For Q1: The authors claim that a 'fair comparison' is that StyleDrop does not use data scaling while FineStyle uses data scaling. I do not think such experimental results are convincing.

For Q2: The authors claim that the color differences in Figure 6 depend on the spectrum, but for the different styles in Figure 7, they state that a similar principle applies. How does the spectrum of color apply to clearly different styles? In my view, if the effectiveness of style transfer cannot be ensured, then the significance of the proposed method is limited. Theoretically, StyleDrop could achieve fine-grained control and maintain a better standard of style transfer by more precise and carefully designed human feedback and prompts.

For Q1: In fact, the Iterative Training with Feedback in StyleDrop is another form of data scaling. As the only baseline method compared in the paper, training without feedback is highly unreasonable and leads to unfair assessments. Moreover, incorporating Iterative feedback training does not require much cost. I do not understand why the authors are unwilling to provide the corresponding experimental results.

For W3 & Q2: The authors also acknowledge their in-sufficient experiments (visual analysis for KV adapter) and inferior style transfer results. Why the authors claim in rebuttal, Q2 that 'Theoretically, enhancing our style dataset with a broader range of subjects and finely annotated concepts could allow for more precise control over such discrepancies through tailored prompts,' but then do not provide actual improved visual results? This also makes me doubt their responses.

In the rebuttal, the authors only repeatedly emphasize the contributions mentioned in their paper once and once again, but they do not address the issues I raised, which only needs very low costs and is important for this paper. Overall, I will decrease my rating.

We thank the reviewer for your additional clarification and comments and we address them as follows.

W3 (1): This work demonstrated that the KV adapter is more effective in performance but lacks visualization to prove that the KV adapter truly focuses on the conceptual content.

Sorry indeed it was our oversight and we did not address this question as requested clearly enough; our previous answer mostly focused on emphasizing the performance improvement and our intuition. We do not claim in the paper that the performance enhancement from the KV adapter necessarily has concrete causal relations with the attention weight changes of conceptual contents. Our current observation shows that there could be a strong connection but we can’t be certain that is true in all cases; but we are certain about the performance enhancement and also our intuition. We thank the reviewer for this suggestion and will try to add further ablation and visualization comparisons in the revision to study and clarify whether our KV adapter could also be a robust and visually interpretable approach besides its performance and intuition.

W3 (2) in Figure 6 that the FineStyle generated an Eiffel Tower with a blue circle even without the prompt word 'circle'; and in Figure 4, parts of a mountain appeared.

The comments regarding Figure 6 was a misunderstanding, likely due to our terse caption for Figure 6: the prompt word “circle” is actually part of the style descriptor, so the blue circle generated with Eiffel Tower is expected. We assumed the readers will refer to Figure 3 for the full prompt when checking Figure 6, and the style description is “in flat cartoon vector art inside a light blue circle on a white background.” Due to space limits, we only include incremental changes in prompt as “….eiffel tower…”. We will include these full prompts in Figure 6 in our revision to clarify this.

As for figure 4, while there seems to be a small green corner region that could be viewed either as a tree or a green forest/mountain, at viewers’ discretion, we believe it is very different from the light grayish-brown stony mountain in the original reference image, which was generated by StyleDrop. We hope the reviewer will agree with us that this could be considered as a successful example of our method, or at least an imperfect but effective example :-)

Q1: The authors claim that a 'fair comparison' is that StyleDrop does not use data scaling while FineStyle uses data scaling. The reviewer does not think such experimental results are convincing.

We did mention “fair comparison” but that was only referring to the part that we have decided to exclude the human feedback component of StyleDrop so we can better understand the difference between the two methods. For better context, when conducting this work, we discussed iterative human feedback with the authors of StyleDrop. They recognize the limitations of iterative human feedback in practice: expensive labor cost, extra annotation time, dependency on quality of synthetic images and risk of performance deteriorating due to human selection bias. The last point is also noted at the end of Sec. 3.3 of StyleDrop paper. These limitations actually motivated us to seek a way of improving fine-grained control without iterative human feedback.

As for whether it is fair to compare FineStyle (with data scaling) and StyleDrop (without data scaling), we beg to differ with the reviewer. Data scaling along with concept-oriented masking are part of the contributions of our proposed work, we compared with StyleDrop as a baseline and did not add data scaling to this baseline because that was not part of their original method. It is reasonable to assume that data scaling will also help StyleDrop and combining our proposed work with StyleDrop can potentially produce a StyleDrop V2 with fine-grained control without the need of iterative human feedback. In theory, we can also add iterative human feedback to both FineStyle and StyleDrop under controlled environment to further study the capacity of FineStyle, and we intend to explore this in our future revisions.

(to be continued)

Q2: The authors claim that the color differences in Figure 6 depend on the spectrum, but for the different styles in Figure 7, they state that a similar principle applies. How does the spectrum of color apply to clearly different styles? In my view, if the effectiveness of style transfer cannot be ensured, then the significance of the proposed method is limited. Theoretically, StyleDrop could achieve fine-grained control and maintain a better standard of style transfer by more precise and carefully designed human feedback and prompts.

By “similar principle”, we did not mean the spectrum of color apply to clearly different styles. We simply meant that similar to the example in Figure 6 that a range of different colors contributed to the fine-grained control in FineStyle , the reference image in Figure 7 features a range of fine-grained style elements (e.g. different colors, texture, shape, and other stylistic cues) and the oil painting image produced by FineStyle also draw from the diverse stylistic cues within the reference.

We understand the concern from the reviewer that it could appear to some viewers (maybe including the reviewer) that the global style seems better captured in the StyleDrop than in FineStyle in the oil painting example and hence it could imply a limitation of FineStyle in terms of style transfer compared to StyleDrop. To our defense, the perception of what is a good style transfer is quite subjective here, especially when FineStyle is focused on fine-grained style elements and StyleDrop is focused on global styles in the reference image. While our proposed method is not perfect, we believe it clearly achieved its goal of enabling fine-grained controllability with proper style preservation. However, we also acknowledge the insight from the reviewer that we could potentially further improve our method to better preserve the global styles if that is what users may desire, and we leave this to future extensions of this work.

Finally, while we beg to differ that StyleDrop is stronger than FineStyle in style transfer for the reasons we mentioned above, we agree that both StyleDrop and our FineStyle could theoretically achieve better fine-grained control with additional iterative human feedback and carefully designed prompts. This, however, again goes back to part of the motivation of our proposed work that we want to gain fine-grained control without the need for iterative human feedback, which has a set of limitations we discussed above when addressing Q1.

Overall, we thank the reviewer for sharing their insight to help us to improve this work and we hope we have clarified some of the misunderstandings and we will adopt the valuable feedback from the reviewer in our revision.

We are extremely disappointed that the reviewer VFWr has completely ignored our contributions and rebuttals while we politely and professionally pointed out the misunderstandings from the reviewer's side, including they misjudged the prompt in Figure 6 etc., and the their reading of the Figure 6 and Figure 4 etc. (which were not even so important from the beginning, because our blue is lighter than StyleDrop then our generation is worse? the prompt word "circle" was there so generating it in Figure 6 is absolutely reasonable; the small green corner of trees in the background is not the brown rocky mountain in the reference and our results are simply effective ...)

Also we want to emphasize that we never acknowledged our style transfer is inferior and that which one is better is just the difference of subjective evaluation between the reviewer VFWr and the authors, which again support our claim that iterative human feedback could be problematic as pointed out by the original StyleDrop authors. And we hope the other reviewers and AC will examine this and provide their subjective evaluation and preference as well while clearly the reviewer VFWr's subjective evaluation is not in favor of this technical work but more nitpicking on the subtle visual difference from their own perspective with misunderstanding.

We also do not agree to the unreasonable standard from the review VFWr that we have to add these attention visualization to prove that our method works: our method apparently work and it showed performance improvement, effective visual output and we carefully explained our intuition behind it. While as we agreed we could add visualization to demonstrate this point in some additional ablation, it is neither really required to prove that our proposed method work (it works!) and nor rigorous (we believe the consensus in the ML community is that visualization generally is a nice tool to have but it is often less robust and can be easily cherry-picked and have strong bias).

Removing iterative human feedback training is desiderata from the original StyleDrop work and we proposed a valid approach to make it work. We are extremely disappointed at the reviewer's response in strongly downplaying our contributions, holding our work to unreasonable standards despite their misunderstanding while dismissing our improvement over a strong prior work StyleDrop.

We've shown our best intention in the last few responses to reviewer VFWr, but we also want to remind the reviewer VFWr that we believe they don't understand the NeurIPS review standards:
"3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and/or incompletely addressed ethical considerations." and "6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations."

We'll stop engaging with the reviewer VFWr. We've put in a significant amount of time and effort into this work and we are confident about its contributions and its applicability to users. We leave this indignant and frank feedback with the sincere hope that other reviewers and ACs will review the conversation above and judge the paper fairly in its own merit.

Thank you all! (including the reviewer VFWr for your reviewing effort, we provide this frank and fair feedback to everyone instead of just sending to AC with the hope to convey our point in the spirit that conversations in the ML community should be unbiased, professional and bi-directional. It is the only way that our community can move forward.)