Free-Lunch Color-Texture Disentanglement for Stylized Image Generation

We sincerely appreciate your constructive and insightful feedback. We are grateful for your recognition that the method is compatiblewith the community ecosystem and the performance is strong. Below, we provide detailed responses to each of your specific points.

W1(texture-related performance gain): In our method, the ability to represent textual information primarily comes from the pre-trained model (i.e., IPA and SD). However, IPA injects image features into all blocks of the network, which introduces semantic leakage from the reference image and results in poor text-image alignment. In contrast, InstantStyle only injects features into style-relevant blocks, effectively avoiding the semantic leakage issue inherent in the original IPA. We follow the InstantStyle injection strategy, which enables better adherence to textual semantics. As a result, the texture-related performance of our method aligns with that of InstantStyle, as shown by the CLIP-I scores in Table 1.

W2(supplement experiments with multiple CA layers): The purpose of employing multiple Cross-Attention (CA) layers in IPA is to ensure consistency in the identity of the input. Prior research has shown that different CA layers in IPA control different attributes—for example, some are more associated with content, while others relate more to style. Our decision to inject at the first decoder layer was initially inspired by InstantStyle. Experiments with multiple CA layers are provided below. Here, ‘b{idx0}a{idx1}’ denotes the ‘idx1’-th CA layer in the ‘idx0’-th decoder block of the denoising UNet. As shown in the table, injecting into additional CA layers beyond the style-relevant one (i.e., the first decoder layer) negatively affects color representation, despite improvements in texture metrics. Moreover, from the visualization results, we observed that the generated images tend to contain more semantic content from the texture reference image (similar to IPA’s results shown in Fig. 5), which is inconsistent with the goal of texture transfer. This phenomenon is consistent with the findings in InstantStyle.

When texture and color embeddings are injected solely into the style-related CA layer (i.e., the first decoder layer), the visual results show more desirable style transfer performance. We will include the corresponding visualization results in the supplementary material.

W3(clarification of TextureExtraction module): We apologize for any confusion. In our paper, the “SVD” mentioned in Table 2 and in the ablation experiments of Fig. 4 actually refers to the TextureExtraction module. We will revise the terminology accordingly in future versions.

W4(Ablation on γ and β): We conduct an ablation study on γ and β. As shown in the table below, when β increases beyond 1, the texture consistency metric (CLIP-I) improves and even surpasses that of InstantStyle (CLIP-I = 0.74), indicating enhanced texture alignment. However, this comes at the cost of decreased color consistency, as reflected by the MS-SWD and C-Hist scores. Therefore, to achieve a better balanced performance between texture and color consistency, we set the default values to β = 1 and γ = 0.003 in our main experiments.

We sincerely appreciate your constructive and insightful feedback. We are grateful for your recognition that the method is intuitive and the analysis is interesting, the design is well-motivated, and the paper is with good writing. Below, we provide detailed responses to each of your specific points.

Q1(clarification of the baseline): Your understanding is absolutely correct. We apologize for any inconvenience caused by the lack of clarity. In Fig.4(a), Our ‘Baseline’ denotes color-texture disentanglement with Image-Prompt Additivity only (i.e., without SVD and RegWCT). In the revised version, we will add clarifications regarding our ‘Baseline’ in the captions.

Q2&W1(experiments with the same reference image): According to your suggestion, we have added results where both texture and color are derived from the same reference image, using the same prompt as in R1.Q2. The comparative experimental results are presented in the table below. In any future versions, we plan to include these results in the supplementary materials. As shown by the experimental results, our method still achieves the best color consistency, which is consistent with the findings reported in the original paper.

In terms of texture consistency, our method achieves the best performance among all methods except for IP-Adapter. However, it is important to note that although the IP-Adapter achieves the highest numerical score for texture consistency, it introduces severe semantic leakage from the texture reference image (see Row 2 and Row 3 in Fig. 5), resulting in poor text-image alignment (CLIP: 0.260 in the table below), which fails to meet the requirements of stylization. In contrast, excluding the IP-Adapter, our method achieves the highest scores in color consistency (MS-SWD: 3.19), texture consistency (CLIP-I: 0.754), and text-image alignment (CLIP: 0.302).

In addition, it should be noted that the ability of our method to represent textual information primarily originates from the pre-trained model (i.e., IP-Adapter and InstantStyle). However, IP-Adapter injects image features into all blocks of the network, which leads to semantic leakage (see Row 2 and Row 3 in Fig. 5) from the reference image and results in poor text-image alignment . In contrast, InstantStyle only injects features into style-relevant blocks, effectively avoiding the semantic leakage issue present in the original IP-Adapter. We adopt the same injection strategy as InstantStyle, which allows for better adherence to the textual description rather than the semantics of the reference image. Given that our method and InstantStyle utilize the same style injection strategy, it is reasonable that their texture performance is comparable. Nevertheless, our main contribution lies in further improving color consistency while maintaining texture performance on par with InstantStyle, thereby achieving effective color-texture disentanglement.

Q3(ablation study of concatenation): Regarding the concatenated averaged embedding, it serves as the basis for SVD-based grayscale suppression, and therefore, SVD cannot be separated from the concatenation operation. The purpose of this concatenation is to enable the identification of the principal singular value(s) corresponding to grayscale information in the feature space after the SVD transformation.

Your understanding regarding the improvements in color consistency brought by SVD and RefWCT is correct. To achieve stylized image generation, we extract color information from the image-prompt and texture information from real images. When isolating color from the image prompt, two key challenges arise: (1) extracting color while minimizing the influence of texture from the image-prompt, and (2) removing color information from the real image to prevent it from interfering with the intended color guidance. The first challenge can be easily addressed by our proposed color-extraction method. To address the second challenge, we adopt SVD to suppress the color influence of the real image, enabling the extraction of cleaner texture features (as shown in Figure 2, Texture Extraction). Although SVD effectively reduces the impact of real-image colors on the final generation, it leads to only a slight degradation in texture fidelity, evidenced by a minor 0.01 drop in the CLIP-I score. Furthermore, your understanding of the role of the concatenated averaged embedding is also accurate. This embedding serves as the foundation for the SVD-based grayscale suppression. Therefore, SVD cannot be decoupled from the concatenation operation. The purpose of this concatenation is to identify the principal singular value(s) associated with grayscale information in the feature space after the SVD transformation.

To ensure better color consistency, we further introduce RegWCT, which enhances the traditional Whitening and Coloring Transform (WCT) by incorporating noise regularization in the latent space of diffusion models. This modification further improves color fidelity, again with only a minimal impact on image quality (a 0.01 drop in CLIP-I).

Minor points: Thank you very much for your suggestion. We will incorporate your feedback in our final revisions.

We sincerely appreciate your constructive and insightful feedback. We are grateful for your recognition that the method is applicable and addresses practical needs, the design is reasonable, and the generation is with good quality. Below, we provide detailed responses to each of your specific points.

W1&W2(texture consistency): The objective of our method is to explore tuning-free color-texture disentanglement using pre-trained models. In our approach, texture consistency is primarily inherited from the pre-trained models(InstantStyle), while our goal is to enhance color consistency with minimal loss of texture, thereby achieving a balance between color and texture attributes. To this end, both SVD and RegWCT are designed to address poor color consistency while minimizing texture loss as much as possible.

In terms of texture performance as reported in Table 1, our method achieves texture consistency second only to IP-Adapter and is comparable to InstantStyle (Our base model). However, it is important to note that although the IP-Adapter achieves the highest numerical score for texture consistency, it introduces severe semantic leakage from the texture reference image (see Row 2 and Row 3 in Fig. 5), resulting in poor text-image alignment (CLIP: 0.233), which fails to meet the requirements of stylization. In contrast, apart from the IP-Adapter, our method achieves the highest scores in color consistency (MS-SWD: 5.57), texture consistency (CLIP-I: 0.74), and text-image alignment (CLIP: 0.281). These results are also supported by the user study in Table 2 and Fig. 4. Additionally, following the suggestion of Reviewer R1.Q2, we have included performance comparisons under complex prompts, and the resulting trends are consistent with the above analysis. Overall, our method focuses on improving color consistency while maintaining texture consistency, and the experimental results demonstrate that our approach achieves a better balance between color and texture consistency.

We agree that our method shows stronger improvements in color accuracy, with slightly lower texture scores. However, it is important to note that the texture metrics used (e.g., CLIP-I, KID) remain sensitive to color content: modifying the color palette inevitably affects the greyscale values, which play a role in defining textures. Consequently, methods that transfer both color and texture from the same reference image—effectively ignoring the color image—benefit from aligned visual cues, making it easier to achieve higher texture scores. In contrast, our method explicitly disentangles color and texture sources and faithfully adapts to the color reference image. This necessary shift in greyscale structure may lead to a modest drop in texture metrics.

W3(other encoders): Our method is training-free and built upon IP-Adapter. Since IP-Adapter relies solely on the CLIP encoder as its vision encoder and does not utilize other encoders such as T5 or DINO from pre-trained models, conducting such an evaluation is not feasible. Furthermore, our method can be extended to any backbones as long as there is an IP-Adapter-like pretrained adapter for the backbone model, including Flux, SD3, SD3.5, etc.

Thank you for the rebuttal. Regarding W3, I understand the work is built on IP-Adapter and cannot be easily validated with other encoders. The authors could consider demonstrating the additive property of other encoders (e.g., DINO or T5) in the final version, to better support the feasibility of integrating diverse encoders into alternative architectures. Overall, I will maintain my current rating.

We sincerely appreciate your feedback. We are grateful for your recognition that the paper is well-written, the research motivation is clear, and our proposed method is interesting. Below, we provide detailed responses to each of your specific points.

W1&Q1(strong hypothesis in Eq.1): Our verification of the image-prompt additivity hypothesis in Eq. 1 is empirical rather than formally proven, and this approach is consistent with established practices in analyzing learned representations of T2I models[1,2]. Prior works such as SEGA [3] and ToMe [4] have demonstrated that CLIP-encoded embeddings exhibit semantic compositionality through additive and subtractive properties in textual spaces, providing foundation for similar behavior in image embedding spaces. Furthermore, the difference vectors between pairs of images with the same color (e.g., A − A* and B − B*) point in consistent directions (Fig. 3a), demonstrating the correctness of this subtraction-based color decoupling method. Besides, the consistent superior performance in color alignment metrics (MS-SWD, C-Hist, GPT-4 color scores) and successful disentanglement results shown in Fig. 4-Fig.6 also provide strong empirical evidence for the correctness of this hypothesis.

W1&Q3(correctness of Eq.3): Eq. 3 is inspired by WNNM[5], a method originally proposed for image denoising. WNNM assumes that most of the noise in a noisy image is concentrated in the lower-K singular values. Each singular value σ of an image patch is updated according to the formula: $\sigma \leftarrow \sigma - \frac{\lambda}{\sigma + \varepsilon}$

In this paper, we apply Eq. 3 to further suppress the gray color in grayscale images, while preserving only the texture information. Specifically, our goal is to decouple texture from the textural image and eliminate the influence of color information. We observe that directly using the Y channel (grayscale) of the textural image introduces a grayish color bias in the generated results (see Fig. 4(a), Baseline). To address this, we aim to remove the residual gray information from the grayscale image.

To achieve this, we concatenate the Y channel of the textural image with its mean grayscale intensity (i.e., the average gray value), forming a new embedding using Eq. 2. This embedding mainly captures the gray color information. Therefore, the top-K singular values in the constructed embedding matrix $\text{Emb}'_{\text{tx}}$ predominantly correspond to the color information we intend to suppress. At the same time, WNNM demonstrates that singular values carry clear physical meaning and should be weighted differently. Thus, we adopt Eq. 3 to ensure that components corresponding to larger singular values undergo stronger shrinkage, effectively removing the undesired gray color while retaining informative texture features.

W2&Q2 (complex prompts): To further evaluate performance with complex prompts, we conducted additional experiments by randomly sampling 10 complex prompts from DREAMBENCH++ [6], together with 10 color and 10 texture images from our dataset, resulting in 1,000 generated images for evaluation. The performance trend under long and complex prompts remains consistent with the results in our main paper (Table 1). Compared to other methods, our approach achieves the best disentanglement of color and texture, resulting in a more balanced performance in both color and texture consistency. Specifically, our method attains significantly higher color consistency, while other methods often exhibit strong color-texture entanglement, with color being overly influenced by the texture reference image. Although IP-Adapter achieves the highest numerical score for texture consistency, it suffers from severe semantic leakage from the texture reference image (see Row 2 and Row 3 in Fig. 5), leading to poor text-image alignment (CLIP score: 0.257), which is inadequate for stylization. In contrast, except for IP-Adapter, our method achieves the highest scores in both texture consistency and text-image alignment.

W3 (visual results in Fig. 5): Compared to the comparison methods, our approach offers best disentanglement of color and texture, achieving a more balanced performance in terms of color and texture consistency. As shown in Fig. 5, even with prompts providing detailed color descriptions from the reference image, baseline methods fail to accurately reproduce the intended colors. In contrast, our method achieves much higher color consistency and avoids the color-texture entanglement seen in other approaches, where color is overly influenced by the texture reference image. Notably, our method maintains strong texture preservation while improving color consistency. For instance, in Fig. 4 (last row), our generated "bear" image better matches the color of the “wave” reference, while other methods—such as StyleDrop, IP-Adapter, InstantStyle, DreamStyler, DEADiff, and CSGO—tend to inherit the colors of Van Gogh's Starry Night (the texture image). Although SDXL and Artist also produce colors closer to the “wave” image, their results are overly coarse. Similar trends appear in examples in Fig. 4, with additional results on color-texture disentanglement provided in Fig. S6 of the supplementary material.

Q4(stability of generating multiple images): To assess the stability of color and texture in repeated generations, we conducted additional experiments by using the data in W2&Q2. For each setting, the process was repeated five times. In each round, 1,000 images were generated, and the relevant metrics were computed. The results, summarized in the table below, show minimal variation in color and texture consistency across rounds, demonstrating the robustness and stability of our method.

Q5(support for multiple-image control): Our method supports simultaneous control over multiple colors and textures to generate images with mixed color and texture attributes. To ensure a clearer and more direct comparison of evaluation metrics, we use two color reference images and two texture reference images as inputs in each experiment. The following two tables summarize the results: the first table reports multi-color control using two color references and one texture reference, while the second table reports multi-texture control using two texture references and one color reference. Complex content prompts are sampled from Q2. For multi-color control, the comparison methods use concatenated textual prompts containing two color descriptions (generated by GPT-4o) to achieve multi-color control. Experimental results demonstrate that our method consistently outperforms others in color fidelity while maintaining competitive texture consistency. Notably, although IP-Adapter obtains the highest numerical score for texture consistency, it exhibits severe semantic leakage from the texture reference image (Row 2 and Row 3 in Fig. 5), resulting in poor text-image alignment (CLIP: 0.269), which is insufficient for effective stylization. In contrast, our method achieves a better balance between color and texture consistency, demonstrating superior disentanglement and control over color and texture throughout the stylization process.

For multi-texture control, since other methods do not support multiple texture reference images as input, we report results only for our method in this setting. The generated visualizations demonstrate that our approach effectively blends multiple color and texture attributes. In future versions, we will provide additional experimental results on the mixing of multiple colors and textures in the supplementary materials.

Paper formatting concerns: Thanks for pointing out our grammatical and logical errors in the original paper, we will correct them in any future versions.

[1] Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models (ECCV 2024)
[2] IP-Composer: Semantic Composition of Visual Concepts. (SIGGRAPH 2025)
[3] SEGA: Instructing Diffusion using Semantic Dimensions (NIPS 2023)
[4] ToMe: Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis (NIPS 2024)
[5] WNNM: Weighted Nuclear Norm Minimization with Application to Image Denoising (CVPR 2014)
[6] DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation (ICLR 2025)