CSGO: Content-Style Composition in Text-to-Image Generation

Q4：What are the primary differences between the proposed method and existing adapter variants, such as Ip-Adapter (Ye et al., 2023) and StyleAdapter (Wang et al., 2023), which leverage content-style pairs for adapter training?

A: None of them have usable pair data for training because of the difficulty of collecting images with the same set of styles in the previous work. Next, we will further describe the differences between CSGO and them. First of all, CSGO has a separate content control branch that can be utilized to more finely restrict the content of the original image. Therefore, CSGO can realize style transfer for image editing and style transfer for original image. However, IPAdapter and StyleAdapter only have single image control branch. For example, StyleAdapter can almost only control the content through text prompts, and lacks finer-grained control. Secondly, CSGO implements image-driven style transfer, text-driven stylized synthesis and text-editor-driven stylized synthesis at the same time. both IP-Adapter and StyleAdapter can only do text-driven stylized synthesis task. Finally, IPAdapter and StyleAdapter can only be trained by reconstruction way, i.e., the style image is the same as the target image, which leads to easy style leakage. The proposed CSGO utilizes the novel IMAGStyle dataset, where the control image and the target image only have the same style, which is more conducive to the learning of the style sampler. This further avoids leakage of style images.

Q5:These models are noted for their limited generalizability to style images that were not included in the training data. Does CSGO exhibit the same limitation? Please provide examples using style images that include random cartoon characters not present in the WikiArt dataset.

A: When building the IMAGStyle dataset, we collected not only some of the images from the WikiArt dataset, but also style-generated images from Midjourney, which cover a wide range of styles. However, IPAdapter is not actually trained for style transfer . Therefore CSGO's style control is superior to IPAdapter and its variants.

The style test sets presented in our manuscript and appendix are not all from WikiArt. e.g., the style image in Fig. 15 is from JourneyDB. the style images in Fig. 9, Fig. 10, Fig. 11, and Fig. 14 are randomly sampled from the web. Figures 4~18 in the Appendix are almost exclusively not from WikiArt; they are from Midjourney, JourneyDB, and online downloads.

Q6:human evaluation results A: Thanks to the reviewer's suggestion, we added the human evaluation results. Setting: we randomly select 100 sets of results from the test set. Of these, 20 groups of portraits and 20 groups of sketches the rest were randomized. Subsequently, a user research experiment was conducted to compare CSGO with Styleshot-lineart, instantStyle, and Stylealigned respectively. Each group contains four generated results and the user selects the best result from the transfer quality.

If our answers are more in line with your expectations, we kindly invite you to reconsider your initial rating.

Thank you for your recognition of our work and for your insightful comments.

Q1: In the second step of data creation, AdaIN with DINO features is used to filter out images with style leakage. Does this process ensure the removal of style leakage related to factors such as content pose, size, and background? It is necessary to demonstrate that the filtering process addresses various types of style leakage, including texture, color, pose, size, and background.

A: AdaIN features can be used to measure content information and style information. For the proposed dataset, we use batch generation followed by cleaning to ensure that the obtained triples are as accurate as possible. Specifically, for each content B-LoRA and style B-LoRA combination, we generate a total of more than 50 images. We then clean them using CAS metrics and use the image with the smallest CAS score as the target image. We show examples in Figure 1 of the Supplementary Material showing that CAS can filter unreasonably generated images. Meanwhile, Figure 2 of the original manuscript similarly shows examples. In Figure 3, we show the target image obtained by CAS metrics. In the experiment, the threshold is set to 1.6 and it can filter the failure cases of pose, size. In addition, for information such as color and texture, it can be filtered by calculating the style similarity through the CSD score [1]. On the other hand, the high quality style transfer results obtained from CSGO utilizing IMAGStyle will hopefully alleviate reviewers' concerns about dataset cleaning.

[1]Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping, Abhinav Shrivastava, and Tom Goldstein. Measuring style similarity in diffusion models. In ECCV, 2024

Q2: The comparison with existing methods appears to omit several relevant works. There are more recent, well-performing approaches that utilize textual inversion variants, LoRA/DreamBooth variants, and training-free methods.

A: To the best of our knowledge, we have compared the current state-of-the-art style transfer methods StyleAlign[2], Styleshot[3], InstantStyle[4], StyleID[5]. For text-driven stylized compositing tasks, in addition to the above methods we also compare with DEADiff[6]. In addition, our approach is free of fine-tuning, presenting a significant advantage over LoRA approaches that continue fine-tuning for a set of style images. As suggested by the reviewers, we compare with the current SOTA style transfer LORA scheme B-LoRA. We trained 3 sets of B-LoRA and show the results in the Supplementary Material. It can be seen that CSGO, which does not require fine-tuning, outperforms the style transfer results of B-LoRA. It should be clarified that the main task of CSGO is style transfer, so the comparison method needs to achieve preservation of both content and style, whereas Dreambooth is only able to achieve preservation of one of them.

[2]Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. In CVPR, 2024.

[3]Gao Junyao, Liu Yanchen, Sun Yanan, Tang Yinhao, Zeng Yanhong, Chen Kai, and Zhao Cairong. Styleshot: A snapshot on any style. arXiv, 2024.

[4]Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv, 2024

[5]Jiwoo Chung, Sangeek Hyun, and Jae-Pil Heo. Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer. In CVPR, 2024.

[6]Tianhao Qi, Shancheng Fang, Yanze Wu, Hongtao Xie, Jiawei Liu, Lang Chen, Qian He, and Yongdong Zhang. Deadiff: An efficient stylization diffusion model with disentangled representations. In CVPR, 2024.

Q3：hyper-parameters

A: As shown in Fig. 14 & Fig. 15 of the original paper, we show the results of the ablation experiments for the hyperparameters involved. In addition, we give the most suitable parameters for style transfer.

A:We thank the reviewers for their feedback. First, the raw data was cleaned due to storage limitations. In response to the reviewers’ questions, we retrained multiple sets of B-LoRA. Subsequently, CAS cleaning was applied, as shown in Figures 18,19, 20, 21, and 22 of the Supplementary Material. In total, 10 sets of generated images were filtered using CAS. These examples demonstrate that CAS effectively cleans illogical generated graphs in terms of pose, size, and other inconsistencies. However, we emphasize that since B-LoRA is inherently more stable for style generation, we rely on CSD for filtering these images. In our experiments, filtering with CAS alone is feasible without the need for CSD. For style images generated by arbitrary LoRA, we propose using CSA to compute style similarity and filter out images with inconsistent colors.

We would like to show that CAS is very effective and intuitive for cleaning our style transfer data (comparing pixel-level differences after DINO feature removal style). Whether it is effective for the rest of the complex scenarios is something that needs to be further verified.

Dear Reviewer,

We hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns.

Please let us know if you have any remaining questions or require additional clarification. We value your feedback and are eager to ensure our work meets the highest standards.

Thank you again for your thoughtful insights and guidance.

Best regards,

CSGO Authors

Dear Reviewer qQFs：

We appreciate your thoughtful feedback. Here, we would like to provide clarifications to address some potential misunderstandings.

Of course, the cleaned data, the IMAGStyle dataset, will certainly be published. What we are removing from storage is the raw data that has been generated directly by B-LoRA, and they are dirty data. It seems to us that these unprocessed data have no value. We will publish the IMAGStyle dataset obtained by CAS processing. Finally, if more style transfer data triples are needed, we can generate and then clean and filter them through the proposed CSGO, which is less costly and efficient compared to B-LoRA. We also hope that this framework and method can promote the development of style transfer.

Best regards,

CSGO Authors

Thank you to the authors for addressing some of the issues.

However, I still have reservations about the dataset filtering system using CAS.

In the appendix, the authors appear to have omitted the CAS filtering results for some reason. Specifically, Figures ?? are not shown in the Appendix, as follows:

we show 10 sets of CAS filtering examples in Figures ??. These cases show that CAS can clean illogical generated graphs for pose, size, and so on. However, we emphasize that since B-LoRA is actually more stable for the generation of styles, it is up to us to filter the images with CSD. In our experiments, it is possible to filter using only CAS without CSD.

I believe that the curation of a large-scale style-content dataset is a critical step in this area of research. Therefore, the dataset curation process should be presented in greater detail.

As a result, the rating remains unchanged.

Thanks to the reviewers for praising the high quality of the dataset and the methodology is easy to follow .

Q1：it is necessary to provide results using this dataset for training on other baseline methods and compare them with the proposed CSGO framework.

A: We thank the reviewers for their suggestions. We retrained the Stytr^2[1] model using IMAGStyle. In order to fully test the performance of IMAGStyle, we launched two experiments, the first is to retrain Stytr^2 using IMAGStyle only, and to fine-tuning Stytr^2 using IMAGStyle (utilizing the released model weights of 160,000 steps) .

Furthermore, since Stytr^2 employs a non-trivial training approach, instead it is implicitly constrained to generate results with content image close to the content image and style close to the style image. The significant advantage of IMAGStyle exists in the <content,style,target> triplet. Therefore, we add explicit pixel-level constraints to Stytr^2, which utilizes MSE loss to constrain the distance of the generated result from the target map in the triplet.

We retrained and fine-tuned the steps on the 8 A800-80G machines for 1W steps， batchsize is 24, and the results are shown in the table below. It can be clearly observed the effectiveness of IMAGStyle's triple data for the style transfer model.

Stytr^2 utilizes the transformer framework for style transfer and has limited generative capabilities, unlike CSGO which utilizes a diffusion modeling scheme. In addition, CSGO is designed with separate content control and style control branches, which has advantages in controling content and style. More importantly, CSGO supports text-prompt-driven stylized synthesis and text-editor-driven stylized synthesis tasks, which Stytr^2 lacks.

[1] Deng Y, Tang F, Dong W, et al. Stytr2: Image style transfer with transformers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 11326-11336.

Q2: Feature injection has not been clearly explained.

A: In the content control branch, we begin by extracting content features using the image encoder and passing them through the content mapping layer to obtain mapped content features. These features are represented as key and value matrices in the attention layers. The trans-attention layer features in the base UNet serve as the query, and the output features are fused with the original trans-attention layer input through pixel-level addition. A detailed explanation is provided in the Content Control section of the original manuscript.

A similar approach is applied to the cross-attention layer in the style control branch. For ControlNet injection, the output features of each down_block and mid_block in controlnet are injected with the base model up_block and mid_block directly through element-level addition. We adopt the methodology outlined in [2], which is elaborated in the Content Control section of the manuscript and further detailed in the appendix.

[2] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Q3: FID, Aesthetic Scores

A:

We thank the reviewers for their valuable suggestions and agree that the inclusion of additional metrics is warranted. As a result, we have provided further quantitative metrics, which are presented in the table below. Intuitively, when the content and style do not blend well, the aesthetic score tends to be lower, and the FID (Fréchet Inception Distance) will also decrease. Based on these observations, we believe that CSGO remains a highly competitive model for style transfer.

Dear Reviewer,

We hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns.

Please let us know if you have any remaining questions or require additional clarification. We value your feedback and are eager to ensure our work meets the highest standards.

Thank you again for your thoughtful insights and guidance.

Best regards, CSGO Authors

We thank the reviewers for recognizing the motivation is valid of our work.

Q1: The authors claim that the performance of image style transfer is limited because of the lack of a large-scale stylized dataset, which makes it impossible to train modes end-to-end. However, the proposed dataset is learned by training and combining different LoRAs, which means the generated stylized data is not the real ground truth for end-to-end training. In fact, the whole framework seems to try to distill the generated dataset in one adaptor.

A: We would like to clarify that a triad training approach, which uses generated target images alongside real content and style images, is closer to end-to-end training compared to a two-stage approach. For the style transfer task, it is almost impossible to collect sufficient <content, style, stylization result> data of the real scenario. Second, the proposed IMAGStyle is obtained through batch generation and high quality cleaning. Even though there are some limitations in the images generated by lora, the triples we get after filtering and manually checking by CSD and CAS metrics are in line with human intuition. This approach is also cost-effective, with a single B-LoRA achieving optimal performance within approximately 10 minutes on H800 or 15minutes on V100. Using the proposed CSGO framework, we have verified that the generated triples support high-quality style transfer. Finally, we would like to highlight that the CSGO framework effectively decouples the content and style control branches. Specifically, we use adapters to inject style features into ControlNet and the base model, respectively. Once trained, the CSGO framework unifies three types of stylistic control tasks: content-image-driven style transfer, text-driven stylized synthesis, and text-editor-driven stylized synthesis, as illustrated in Fig. 1.

Q2: Discusses relationship to face-swapping work [r1] and clarifies CSGO contributions

A: We thank the reviewer for the introduction and appreciate their observations. We agree that our approach does not conflict with the face-swapping scheme using <source, target, results>. Work [r1] utilizes real images to construct the source and target images in <Source, Target, Result>, which ensures that the target learned by the model comes from the real scene. However, the difference between the face-swapping task and the style transfer task is that it is difficult to construct content images and style images from real images. Style transfer involves both high- and low-dimensional features, such as color, texture, hue, and strokes. Although we can apply some image data-augmentation schemes to fade to other images, it limits the diversity of style transfer. For instance, it was challenging to degrade a furry doll as a target image into other styles as fake content images. Interestingly, our early approach was to generate fake content and style images by fading the style images. However, the results are significantly less effective than using the proposed IMAGStyle dataset. Therefore, it is a feasible way to construct the target image by real content image and real style image.

We believe that ControlNet and IPAdapter have become the most effective and widely accepted methods for feature injection in the era of diffusion modeling. They offer simplicity and reliability, making them ideal for scenarios requiring feature injection. Furthermore, we emphasize that the main contribution of this paper is to provide a set of style transfer dataset construction and cleaning methods while annotating a high-quality segmentation migration dataset. With the support of this dataset, we utilize the mainstream framework to build a simple but effective style transfer framework, CSGO, which enables CSGO to unify three key style control tasks: image-driven style transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis tasks through independent content control and style control.

[r1] Huang, Ziyao, et al. "Identity-Preserving Face Swapping via Dual Surrogate Generative Models." ACM Transactions on Graphics 43.5 (2024): 1-19.

Dear Reviewer,

We hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns.

Please let us know if you have any remaining questions or require additional clarification. We value your feedback and are eager to ensure our work meets the highest standards.

Thank you again for your thoughtful insights and guidance.

Best regards, CSGO Authors

Dear Reviewer zBGr:

As today is the last day to revise the manuscript, I wanted to kindly follow up regarding the concerns you raised earlier. We have already provided detailed responses to address your feedback, but we have not yet received any further comments or suggestions.

If there are any remaining points or clarifications needed, please feel free to let us know. We greatly value your insights and are eager to ensure the final manuscript meets your expectations.

Thank you for your time and thoughtful consideration.

Best regards,

CSGO Authors

We thank the reviewers for recognizing the pretty complete, solid dataset, and useful of our article.

Q1: Difference of CSGO with recent research like IP-adapter, InstanceID, InstanceStyle, and the usage of Ada-In is common too in stylization research.

A: Our main contributions include the introduction of a construction and cleaning method for constructing style migration datasets, and the labeling of the first high-quality style transfer dataset, IMAGStyle, which opens up the possibility of end-to-end style transfer. Additionally, we propose a simple and efficient CSGO unified framework following the mainstream frameworks to verify that the constructed and cleaned datasets are enhanced for style transfer. Essentially, the main role of the CSGO framework is to validate the dataset as well as unify multiple style transfer tasks. With the help of IMAGStyle, the simple CSGO framework can unify image-driven style transfer, text-driven stylized synthesis, and text-editor-driven stylized synthesis tasks. AdaIN is the metric we use to determine content similarity in data cleaning tasks, which is very effective.

Regarding the reviewer’s comment, our current approach involves injecting style features using IPAdapter, a method that has gained prominence in the era of diffusion modeling. In fact, injection methods are not the focus of our research. We study how to construct and clean the style transfer dataset and how to design a unified framework to unify the style transfer framework, similar to how CSGO can realize image-driven and text-driven based style control tasks.

Q2：Concern of dataset

A: We agree with the reviewer that style transfer research should aim to enhance controllability, such as adjusting the degree of stylization or selectively transferring attributes like colors, strokes, or tonal styles. However, achieving more finer-grained style control is a much more challenging task, especially since end-to-end style transfer was already challenging prior to the collection of IMAGStyle. To achieve finer style control, we believe that it should be similar to the proposed ability of CSGO to learn all the style elements of a style image before it can further decouple and learn different types of style elements.

The current implementation of CSGO can influence style transfer results by modulating the content injection strength scale, enabling a straightforward adjustment between more and less stylized outputs, as demonstrated in Figure 14.

Q3: Failure cases

A: Thanks to the reviewer’s suggestion, we have added the failure cases to the supplementary material. First, for real portrait stylization, as shown in the first row, there is a potential loss of facial identity. Portrait images can be difficult to collect due to the privacy issues involved, leading to some limitations in CSGO's style migration for real portraits. Second, despite incorporating styles into the ControlNet and base model, CSGO may still leak information, such as the original image's color. This phenomenon is due to the fact that the dataset still has insufficient pair data and needs to be further expanded using existing models (e.g. CSGO). In the future, we aim to enhance the CSGO framework in several ways. First, we plan to use CSGO in conjunction with LoRA to improve the portrait of the IMAGStyle dataset and enhance portrait stylization capabilities. Second, we will redesign and train the content encoder and style encoder to minimize content leakage and style leakage. However, we acknowledge that these improvements may not be achievable in the short term.

If our answers are more in line with your expectations, we kindly invite you to reconsider your initial rating, which is more confident for us to explore at a later stage.

Dear Reviewer,

We hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns.

Please let us know if you have any remaining questions or require additional clarification. We value your feedback and are eager to ensure our work meets the highest standards.

Thank you again for your thoughtful insights and guidance.

Best regards, CSGO Authors

Dear Reviewer Vjbz:

As today is the last day to revise the manuscript, I wanted to kindly follow up regarding the concerns you raised earlier. We have already provided detailed responses to address your feedback, but we have not yet received any further comments or suggestions.

If there are any remaining points or clarifications needed, please feel free to let us know. We greatly value your insights and are eager to ensure the final manuscript meets your expectations.

Thank you for your time and thoughtful consideration.

Best regards,

CSGO Authors

We thank the reviewers for recognizing the well-written, clear structure, easy to understand, and valuable dataset of our article.

Q1：Clarification of the article's main contributions and model structure

A: The primary contribution of this study is the development of a construction pipeline and a cleaning method for style transfer datasets. Using this approach, we labeled a high-quality style transfer triad to enable effective end-to-end style transfer training. Given the challenges of collecting sufficient style transfer data from real-world scenarios, synthetic data was utilized to achieve this goal. In particular, as far as we know, the proposed IMAGStyle is the first large-scale style transfer dataset. To evaluate the effectiveness of the generated dataset, we implemented a straightforward yet highly effective framework, CSGO, which adheres to the mainstream paradigm. Despite this alignment, we carefully designed a content- and style-independent control framework. Consequently, CSGO unifies image-driven style transfer, text-driven stylized synthesis, and text-editor-driven stylized synthesis tasks.

Finally, we would like to clarify that models are not assembled haphazardly. We have found that CSGO easily separates content and style branches, allowing for more precise content and style control to be implemented independently, such that CSGO can implement a wide range of text- and image-guided style control tasks. In addition, we believe that an easy-to-use CSGO framework is more likely to be extended and used by the community.

Q2: Explanation for operation mentioned in the paper —“The insight of this is to pre-adjust the style of the content image using style features making the output of the Controlnet model retain the content while containing the desired style features.

A: In CSGO, the Controlnet model is fixed-weighted to ensure that content information is as complete as possible after style transfer. Therefore, if the style features are only injected into the up block, the original content features output by Tile controlnet will be directly injected into the up block and weaken the style information. Therefore, we inject style features into the Controlnet model in advance so that the output of the controlnet contains pre-merged content and style features. In fact, the principle of injecting style features in controlnet is similar to that of the base stable diffusion model [1,2]. Fixing the weights of the base model can still adjust the style of the generated image, so it is also effective in the controlnet (i.e., the controlnet model, which structure is smliar with the base model). Finally, from the results of the supplemental quantitative experiments (see table below), the style similarity score CSD for style transfer was significantly improved after the injection of style features into controlnet.

[1]Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv, 2023.

[2] Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv, 2024.

Dear Reviewer,

We hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns.

Please let us know if you have any remaining questions or require additional clarification. We value your feedback and are eager to ensure our work meets the highest standards.

Thank you again for your thoughtful insights and guidance.

Best regards, CSGO Authors

Dear Reviewer rbd7

As today is the last day to revise the manuscript, I wanted to kindly follow up regarding the concerns you raised earlier. We have already provided detailed responses to address your feedback, but we have not yet received any further comments or suggestions.

If there are any remaining points or clarifications needed, please feel free to let us know. We greatly value your insights and are eager to ensure the final manuscript meets your expectations.

Thank you for your time and thoughtful consideration.

Best regards,

CSGO Authors