Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models

Reply to weakness 1:

We provide visual comparisons between our method, RB-Modulation, and CSGO. The results can be accessed at the following links: https://drive.google.com/file/d/1jjAgViV9Z7MTrES0eMhaBPlpS2ZtyEVE/view?usp=sharing and https://drive.google.com/file/d/1zQQa2XHlTBspOJ1IvC6VKQnRxEuu2Lvc/view?usp=drive_link. We present several key observations:

1. The CSGO method may suffer from style degradation or loss of text fidelity, showing inferior performance compared to our method.
2. The RB-Modulation method proposes a framework that incorporates a style descriptor into a pre-trained diffusion model. With style descriptions, RB-Modulation can generate more satisfactory results than without them, preventing information leakage from the reference style and adhering more closely to the desired prompt.
3. As pointed out in the original paper, ''The inherent limitations of the style descriptor or diffusion model might propagate into our framework''. Thus, RB-Modulation may fail to preserve the style of the reference when the style description does not align well with the image reference. For example, RB-Modulation’s results may experience style degradation on abstract, Orphism, or realism art styles."
4. Building upon the impressive StyleShot and leveraging appropriately fewer conditions, our method successfully avoids content leakage while enhancing style, achieving better style transfer performance than recent competitive models.

Reply to weakness 2:

Sorry for the confusion; m^i should be set to 0 to discard the content-related elements.

Reply to weakness 3:

We apologize for the confusion. Figure 3(b) illustrates the Learning Paradigms introduced in Lines 305-318. In the Learning Paradigms, the style reference's image feature $e_1$ is subtracted by the content feature $\psi(e_1)$ or $\phi(e_2)$ to avoid the presence of content $c_2$. We will add an explanation for clarity. As pointed out in Line 361, the linear layers are trained through image reconstruction using mean squared error (MSE) loss to predict errors. The specific algorithm is provided in Algorithm 2 of Appendix A.3. The optimization objectives are illustrated in Lines 877 and 879 for the Text-Adapter (Baseline) and Image-Adapter (Ours), respectively. These optimization objectives are used to minimize the prediction error of noise in reconstructing style reference while maximizing the difference between the model conditioned on the style reference's content and that conditioned on the target prompt.

Reply to weakness 4:

The ground-truth condition data distribution ($q(x|c_1, c_2, c_3)$) defines images that exhibit high style similarity with the style reference while avoiding content from the style reference while maintaining high fidelity with the target text prompt. Thus, "smaller divergence" indicates that the result distribution generated by our proposed method is closer to the ground-truth distribution, showing better style alignment, text fidelity, or less content leakage.

Reply to weakness 5:

We ablate on cluster number $K$ in the text-driven style transfer based on the StyleBench dataset. We report the image alignment and text alignment results based on three different CLIP backbones in the following Table. It is shown that a smaller $K$, such as $K=2$ can lead to a slightly higher text alignment score since more content-related elements in the style reference are masked. Especially in the 3D model, Anime, and Baroque art style that contains more human-related images, smaller $K$ can lead to higher text alignment scores and more efficiently avoiding content leakage.

Reply to weakness 6:

We apologize for the confusion. We detailed this in Line 913. We perform binary classification on the generated images to differentiate between the reference's content object and the text prompt, computing the classification accuracy, which is referred to as the fidelity score. Therefore, the fidelity score primarily reflects the model’s ability to control text prompts.

Reply to weakness 7:

In all experiments, we do not use any style descriptions in the prompt. The only input difference between our method and the baseline model, IP-Adapter, is the content description for the style reference, which we simply use a common template: "person, animal, plant, or object in the foreground" in experiments on the StyleBench benchmark. That is, the proposed masking-based method does not require content knowledge of the image reference; instead, we leverage the CLIP text feature of a common template to identify the elements that need to be masked.

Reply to weakness 8:

Following the metric used in Gao et al., we report the image and text alignment scores alongside training steps in Figure 5(a). Image alignment refers to the cosine similarity between the CLIP embeddings of the generated images and the style reference images, while text alignment measures the cosine similarity between the CLIP embeddings of the generated images and the target text prompts.

Initially, the image alignment is high due to significant content leakage in the generated image. As the unwanted content decreases, the alignment between the generated image and the style reference gradually decreases.

[1] Junyao Gao, Yanchen Liu, Yanan Sun, Yinhao Tang, Yanhong Zeng, Kai Chen, and Cairong Zhao. Styleshot: A snapshot on any style. arXiv preprint arXiv:2407.01414, 2024.

Reply to weakness 9:

We asked 10 users from diverse backgrounds to evaluate the generated results in terms of text fidelity, content leakage, and style similarity, and to provide their overall preference for each composition of style reference and target text prompt, considering these three aspects. We sampled 50 generated images for each target text prompt and provided all results to them.

We sincerely thank you for your valuable comments and apologize for any confusion.

We have carefully incorporated all of your suggestions into our revised manuscript and have submitted it for your review. We have revised both the main manuscript and the appendix, with all changes marked in red for your convenience.

Further explanations are provided for replies 6 and 9.

For re 6: The binary classification is performed using the CLIP model, with the CLIP-H/14 as the image encoder. Specifically, we denote the cosine similarity between the CLIP image feature of the generated image and the CLIP text feature of reference's content object as $\frac{\langle e_2, e_g \rangle}{|e_2| \cdot |e_g|}$. Similarly, we denote the cosine similarity between the CLIP image feature of the generated image and the CLIP text feature of text prompt as $\frac{\langle e_3, e_g \rangle}{|e_3| \cdot |e_g|}$. If $\frac{\langle e_2, e_g \rangle}{|e_2| \cdot |e_g|} < \frac{\langle e_3, e_g \rangle}{|e_3| \cdot |e_g|}$, the generated image is considered correctly classified, meaning it contains the target content rather than the content of the style reference.

For re 9: Apologies for the confusion. In Section 4.1, the constructed dataset consists of 10 content objects (from CIFAR-10) and 21 image styles (11 for training and 10 for testing) for each content object, with 8 variations per style. This results in a total of $10 \times 11 \times 8 = 880$ style references. For each style reference, we perform style transfer for 5 target text prompts, with 4 generations per target text prompt, leading to $880 \times 4 = 3520$ generations per text prompt. We randomly sample 50 images from the 3520 generated images for each target text prompt. In total, this gives us $50 \times 5 = 250$ images from each method to evaluate. The same procedure is applied in the evaluation presented in Section 4.2. We have also included these details in Appendix A.4.2 for further clarity.

Reply to Weakness1:

We perform clustering on the element-wise product of $e_1$ and $e_2$. Compared to other clustering methods, such as clustering on the element-wise absolute difference between $e_1$ and $e_2$, our method achieves the highest energy score for content $c_2$ after masking elements in the high-mean cluster, as shown in Proposition 1. The high value of the energy function $\mathcal{E}(c_2, x_t)$ indicates $x_t$ exhibits a low likelihood of content $c_2$, leading to superior performance in content removal.

We conduct simulation experiments based on our constructed dataset (which we introduced in Sec 4.1) to demonstrate Proposition 1. Using the energy score proposed by Liu et al., we calculate the energy scores of the masked image features for two different masking approaches: one based on clustering the product of $e_1^i$ and $e_2^i$ ($e_1^i \cdot e_2^i, i\in\{1, \cdots d \}$) and the other based on clustering the absolute difference of $e_1^i$ and $e_2^i$ ($|e_1^i - e_2^i|, i\in\{1, \cdots d\}$). For both methods, we report the 0th, 25th, 50th, 75th, and 100th percentiles of the energy scores across various masking proportions. As shown in the table below, our method consistently generates higher energy scores when discriminating content $c_2$, confirming the results outlined in Proposition 1.

[1]Liu W, Wang X, Owens J, et al. Energy-based out-of-distribution detection[J]. Advances in neural information processing systems, 2020, 33: 21464-21475.

Reply to Weakness 2:

Compared to the vanilla model, the clustering process required by our method incurs a slight increase in GPU resource usage and inference time. We report the inference time and GPU usage when the test batch size is set to 1, as follows:

On the one hand, parallel computing can be employed to eliminate this additional time cost. On the other hand, instead of performing clustering to identify the masked elements, we can directly mask the top proportion (e.g., the top 5%) of the element-wise product of $e_1$ and $e_2$.

Reply to ``the contribution compared to IP-Adapter and InstantStyle is not significant'':

We uncover a critical yet under-explored principle: guiding with appropriately selected fewer conditions can efficiently avoid unwanted content flowing into the diffusion model, enhancing the style transfer performances. In this paper, we introduce two strategies to appropriately select fewer conditions, i.e., the training-free masking method and the training-based Image-Adapter (as illustrated in Lines 312-317 in the paper). We demonstrate the superiority of both the training-based and training-free methods theoretically and experimentally. Compared to the previous masking-based methods, we propose a novel marked-element selection method which masks (zero-out) the elements in the cluster with high means. This superiority of the proposed masking strategy in content removal is backed by Proposition 1.

Reply to `` Introducing a masking mechanism is effective has been demonstrated in previous studies'':

Although several studies have explored the effectiveness of masking mechanisms, our method differs from these approaches in several key aspects:

1. No coupled denoising processes: Our method avoids the need for two denoising processes, thus saving computational resources. For instance, the DIFFEDIT method requires two denoising processes—one conditioned on the query text and the other conditioned on a reference text. By contrasting the predictions of the two diffusion models, DIFFEDIT generates a mask that locates the regions needing editing to match the query text.

2. Masking in the latent space: Unlike DIFFEDIT, which operates on the pixel level to generate a mask highlighting the regions of the input image that need editing, our method performs masking in the latent space, bypassing pixel-level operations and patch-level manipulations.

3. Focus on content leakage in style transfer: While the MDT method introduces a latent masking scheme to enhance the DPMs' ability to learn contextual relations among object semantics in an image, it focuses on predicting randomly masked tokens from unmasked ones. In contrast, our method targets content leakage in style transfer. We mask feature elements that are related to unwanted content from the style reference, guided by clustering results on the element-wise product. The "From Test to Mask" method leverages the rich multi-modal knowledge embedded in diffusion models to perform segmentation. By comparing different correlation maps in the denoising U-Net, it generates the final segmentation mask.

Reply to `` the problem is not formally formulated'':

We apologize for the confusion. We provide the formulation of the problem for clarity:

In the context of style transfer, given a style reference $c_1$, the content of the style reference $c_2$, and the target text prompt $c_3$, the text-driven style transfer aims to generate a plausible target image by combining the content of the target text prompt with the style of the style reference, while ensuring that the unwanted content from the style reference does not transfer into the generated result.

Reply to `` the selection criteria are not clearly described'':

We apologize for the confusion. We introduced the selection criteria in Lines 212–231 in Section 3.1, as illustrated in Figure 3(a).

Reply to ``The paper does not show significant evidence to demonstrate its efficiency compared with the previous methods'':

We provide visual comparisons in Figures 6, 7, 10, and 11 to demonstrate the superiority of our method compared to previous approaches. In comparison to the baseline models, IP-Adapter and InstantStyle, we present generation results across various coefficient values in Figure 11 of Appendix A.5. These results highlight that both IP-Adapter and InstantStyle heavily rely on test-time coefficient tuning for style strength, requiring users to engage in a labour-intensive process to achieve a balanced synthesis between target content and style. Particularly in high-coefficient scenarios, both models experience significant content leakage from the style references and a loss of text fidelity. In contrast, our method produces satisfactory results even in high-coefficient settings.

Reply to ``The proposed method is only evaluated on CIFAR-10 dataset'':

We apologize for the confusion. The proposed method was not only evaluated on our newly constructed dataset based on the classes of CIFAR-10 (not a real CIFAR-10 dataset) but also on the benchmark dataset Stylebench, proposed in StyleShot, both on text-driven and image-driven style transfer.

Reply to Q1:

Thank you for your helpful suggestion! We will add a part to highlight the comparison between our method and the previous methods. For details about the comparison, please refer to the reply to weaknesses.

Reply to Q2:

Thank you for your helpful suggestion. We will provide a formal definition of the problem in the revised paper. As for how our method addresses this issue, we have detailed our approach in Lines 212-231. The effectiveness of our method is supported by the highest energy of $\mathcal{E}(c_2, x_t)$, achieved through the proposed masked element selection criteria, as shown in Proposition 1. This effectively reduces the likelihood of content in style reference, leading to superior performance in content removal. We also provide the simulation result in the reply to Reviewer gkxk, confirming the results in Proposition 1.

Reply to Q3:

We detailed the proposed method in Section 3.1. The proposed masking strategy to decouple the content and style characteristics was illustrated in Lines 212-231.

Reply to Q4:

The proposed masking strategy is detailed in Section 3.1. For the proposed masking-based method, we use the same selection criteria based on the K-means cluster and set the clustering number as 2.

Reply to Q5:

Compared to training-based methods, such as InST, the proposed masking-based method does not require a training process and instead manipulates the image condition of the IP-Adapter in a plug-and-play manner. By simply performing clustering on the element-wise product of $e_1 \cdot e_2$ based on IP-Adapter, our method effectively mitigates the content leakage issue. While the clustering process introduces a slight increase in GPU resource usage and inference time compared to the vanilla IP-Adapter model, it offers an efficient training-free solution for content removal.

Reply to Q6:

We perform experiments on our constructed dataset (not a real CIFAR-10 dataset), which uses content objects of CIFAR-10 but with various styles. The images are generated using the MACE code, and each image has a resolution of 512x512. All relevant details have been discussed in Lines 352-358 in the main paper. More importantly, we also evaluate our method on the benchmark dataset StyleBench and report the corresponding results in Figure 6 and Figure 12.

Reply to Q7:

The evaluation metrics, FID and LPIPS, are primarily used to assess the quality of generated images by measuring the similarity between real and generated images. However, a high similarity does not necessarily indicate low content leakage or high style similarity. Following your insightful suggestion, we report the image alignment and text alignment scores for our method and its counterpart, InstantStyle, using different CLIP classifiers, based on the StyleBench dataset. The results, shown in the table below, reveal that while InstantStyle achieves slightly higher text alignment, it significantly sacrifices image alignment with the style reference across various CLIP classifiers. As illustrated in Figure 6 and Figure 12, InstantStyle avoids content leakage at the cost of style degradation. By leveraging appropriately fewer conditions, our method achieves a better balance between target content and style, producing more effective style transfer results.

Reply to ``Demonstrate the divergence by comparing with InstantStyle rather than the method itself'':

In Theorem 1, we compare the proposed method, which masks certain elements of the image condition, with the vanilla method. It should be noted that the vanilla method, without the masking operation, uses conditions $c_1$, $c_2$, and $c_3$, forming a family of models that includes the InstantStyle method.

Reply to ``Why not only encode the background or non-object related style patches'':

It is true that we can encode the background or non-object-related style patches into the image encoder to prevent content leakage. To achieve this, we can use the GroundingDINO [1] and SAM [2] models to locate the non-object region in the image. However, compared to the proposed method, which performs masking in the latent space, masking patches in the image requires more computational resources and inference time.

Reply to `` the experiments showing more like photo as a reference image'':

In this work, we focus on the issue of content leakage in style transfer, an important challenge that has been explored by several recent studies [3,4]. We have included a comparison between our method and these studies [3,4], along with visual comparisons available at the following links: https://drive.google.com/file/d/1jjAgViV9Z7MTrES0eMhaBPlpS2ZtyEVE/view?usp=sharing and https://drive.google.com/file/d/1zQQa2XHlTBspOJ1IvC6VKQnRxEuu2Lvc/view?usp=drive_link. A detailed discussion can be found in Reply 1 to Reviewer nfUg, and the related points have been incorporated into the revised manuscript.

Here are further explanations of the comparison in this work:

1. To fully demonstrate the effectiveness of the proposed method in mitigating the leakage of various content from the image reference into the generated image, we create style references by combining 21 different styles and objects (from CIFAR-10). We analyze the experiment results in Section 4.1.

2. In Section 4.2, we also conducted experiments on the standard benchmark dataset, StyleBench, proposed in StyleShot, which includes 73 different styles, comprising both non-object and object style references. As shown in the 5th and 6th rows of Figure 6, when using non-object style references, both StyleDrop and StyleShot suffer from style degradation or loss of text fidelity. We provide additional results to compare the performance of our method with previous methods using non-object style references at https://drive.google.com/file/d/1ZBISv9HsiSyw5yNoHPOdHamN8kqapoxG/view?usp=sharing

Reply to `` Figure 8 compare the results with InstantSyle but not StyleShot'':

We provided visual comparisons between StyleShot and our method for image-driven style transfer in Figure 7. In Figure 8, we compare the proposed masking method with the feature subtraction approach of InstantStyle, using the same model configuration, including the StyleShot style encoder, random seed, guidance scale, and other parameters.

[1] Liu S, Zeng Z, Ren T, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection[J]. arXiv preprint arXiv:2303.05499, 2023.

[2] Ravi N, Gabeur V, Hu Y T, et al. Sam 2: Segment anything in images and videos[J]. arXiv preprint arXiv:2408.00714, 2024.

[3] Rout L, Chen Y, Ruiz N, et al. RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control[J]. arXiv preprint arXiv:2405.17401, 2024.

[4] Xing P, Wang H, Sun Y, et al. Csgo: Content-style composition in text-to-image generation[J]. arXiv preprint arXiv:2408.16766, 2024.

Reply to ``StyleShot's better results than those shown in the paper'':

Thanks for your suggestion! It is likely that the image generation results of the diffusion model exhibit high randomness. To account for this, we compare the style transfer results by reporting the average results of four generations for each combination of style reference and target text prompt.

We also provide multi-sample generalization results for each combination of style reference and target text prompt at https://drive.google.com/file/d/1XUCFhPFsrgD49uuQosNrU323zWxAqbbX/view?usp=drive_link. Content leakage and loss of text fidelity are marked for the one-to-one image generation results in Figure 20 of the revised manuscript. To avoid the influence of randomness, we ensure that all model configurations remain consistent, including the random seed, guidance seed, denoising steps, and other parameters. From the one-to-one comparison, we observe that our method significantly reduces content leakage and alleviates loss of text fidelity, consistently refining StyleShot’s results across all combinations.

Reply to `` Comparison with existing benchmark'':

In Section 4.2, we conducted experiments on the existing benchmark (the StyleBench dataset used in StyleShot) to demonstrate the effectiveness of our method. Visual comparisons were provided in Figure 8 and Figure 12 of the paper.

Reply to ``Questions on the visual results'':

The visual results in Figure 1 are based on the StyleBench dataset from StyleShot. Additional results are provided in Figure 6 and Figure 15 in the paper. Regarding the results in Figure 1, our method effectively avoids content leakage while exhibiting less style degradation compared to the previous method. In Figure 8(a), InstantStyle’s results tend to disrupt the style information extracted by StyleShot’s style encoder due to image-text misalignment. In Figure 8(b), InstantStyle also experiences style disruption, while our method alleviates this issue by using fewer appropriately-selected conditions.