StyleMaster: Towards Flexible Stylized Image Generation with Diffusion Models

The major problem of this paper is that the claims are not well justified. There is some gap between the motivation of addressing the issues in StyleAdapter and the proposed technical approach.

Both CLIP features and VGG descriptors are trained or designed to conduct image/object recognition. So their abilities to describe and represent complicated or fine-grained image styles are in question, which lead to the questions: 1) If CLIP features are aligned with image captions and not good to describe image styles, VGG descriptors may also suffer from the similar issues; 2) Is image style more related to local image patterns or global content? More in-depth discussions on how to define and describe style are required to justify the advantage of using VGG descriptors.

Conceptionally, if a method uses feature from more sources or removes some unrelated/negative tokens, it “may” improve the image generalization, but not necessarily. So the paper needs to explain and justify the proposed method can achieve this. For example, the paper needs to first justify “the global information can guide the model to better concentrate on style-related knowledge”, which is an assumption and shall not be used as a principle to derive the conclusion that so our approach can yield “better” results. Actually, the paper uses “better” many times where there are some logic gaps in those statements.

The embeddings of Z_{clip} and text embeddings Z_{cap} are no longer aligned after respective self attention operations. So it may not be that impactful to subtract/remove some unrelated/negative tokens as the paper claims.

The experiments show that the improvements are not consistent. There are 20 styles in the experiments, thus it is not clear how the proposed method performs for fine-grained style control.

1. The manuscript says, the use of the CLIP visual encoder for style embeddings emphasizes local patterns, but elaborating on this aspect could improve the manuscript. Specifically, (1) it’s unclear why local-only embeddings might be insufficient to replicate style, as I believe that local elements (e.g., brushstrokes) often receive primary focus from artists or viewers when evaluating specific artistic styles. (2) What if the CLIP visual encoder are used to model global features by embedding the entire image as a whole? I am not fully convinced of the need for an additional VGG network and Gram, though they are frequently used in style transfer. Is CLIP not sufficient for embedding global features?

2. The paper lacks a clear explanation and discussion on the necessity of semantic-aware features with BLIP in this model. Additionally, while this approach resembles the method in DreamStyler, the benefits or potential issues of omitting semantic-aware features remain unclear. Further discussion of how these features enhance overall performance, along with a comparison to DreamStyler, would clarify their contribution.

3. In model comparisons, there is a mix of models utilizing test-time optimization and those using pre-training approaches. Differentiating between these categories would improve clarity. Additionally, the N-shot experimental setup and procedure are somewhat ambiguous. I assume the initial (pre-trained) model is trained using the settings described in Section 5.1, but in the N-shot experiment, does further optimization occur using the N reference samples? If so, what approach is used?

1.The input style images should be put into Fig.1, 4, 5 and 6 for side-by-side comparsions.

2.The references of 2024 are missing in the part “Stylized image generation” of Sec.2. For example, [A-D] ...

[A] Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer, CVPR'24

[B] ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank, AAAI'24.

[C] DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations, CVPR'24

[D] ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptation, CVPR'24.

3.The details of style destroy are missing. How the input style image is processed in style destroy to derive the output?

4.The differences between results generated with or without style/disen loss are minor in Fig. 6 and Table.5.

1. The paper introduces several complex loss functions (e.g., Gram consistency loss and semantic disentanglement loss) and additional model components (e.g., multi-source embedding, dynamic adapter). While these designs are effective in improving performance metrics, they resemble "patches" added to the existing model, contributing to engineering complexity. The paper lacks an in-depth theoretical analysis to explain the interrelationships between these modules, relying instead on enhancing specific metrics with pre-existing components.
2. The multi-source style embedder and dynamic attention adapter largely extend existing techniques rather than introducing fundamentally new theoretical contributions. While effective in real-world applications, this approach may be perceived as lacking in novelty from a research perspective.
3. StyleMaster currently only supports image-based style conditioning. Extending this to other forms (e.g., text, video, or 3D data) would improve its versatility for broader stylization applications.
4. Due to the multi-source style embedding extraction, which involves patch-level transformers, the model is computationally intensive. This may limit its practicality in real-time or resource-constrained environments, particularly when handling larger or multiple style reference images.