Semantix: An Energy-guided Sampler for Semantic Style Transfer

We thank for your review and helpful suggestions. Below are our clarifications of your concerns:

1. Limited Novelty:

   a. Energy Guided is widely used. While energy functions have been used to guide sampling, their design remains an area for further exploration. Unlike previous methods of energy functions for various domain, we focus on refining energy function design for zero-shot semantic style transfer, which is the first work in this field to the best of our knowledge. We have thoroughly addressed the existing energy-guided approaches in the Related Work section of the paper, e.g., Lines 181-188.

   b. Our Energy Function design is not novel. We consider that the energy function we proposed demonstrates significant innovation. We innovatively apply energy functions to semantic style transfer, incorporating DIFT, positional encodings (PE), and various feature distance. To the best of our knowledge, this is the first use of these methods in style or appearance transfer. Additionally, our use of L2 regularization with feature distance as a balancing term for style and appearance introduces a novel approach.

2. Motion Consistency:

   a. Specific adaptations for video. We would like to address and clarify some potential misunderstandings regarding our work. Our method is a sampler for diffusion models. We do not have any specific adaptations for video task. Moreover, this property is also one of the advantages of our method.

   b. Motion consistency for images. We do not consider motion aspects in the field of images. In the video domain, the content video is provided by users and we do not consider the consistency or other attributes of provided video. Regarding to the synthetic video, after style transfer through a pre-trained video model, further contains the understanding of motion consistency.

   c. Apply to pretrained video model. We wish to emphasize to the reviewer that our video experiments were conducted using a pre-trained video model. Furthermore, our proposed approach is a sampler that does not modify the pre-trained model. As a result, the motion consistency of the generated video model remains fully aligned with the motion consistency capabilities of the T2V model used, without introducing any additional damage.

3. No significant differences in Fig. 7: We observe that removing any of the guiding components leads to a noticeable decline in generation quality, as shown in the first row of Fig. 7:

   a. Without style feature guidance, the style similarity between generated image and style image slightly decreases.

   b. Without positional encoding, the image becomes blurry.

   c. Without spatial feature guidance, the image distorts and exhibits structural degradation.

   d. Without the semantic distance regularization, the generated image may overfit to style or structure, leading to visual disharmony and a reduction in image quality.

4. Selection of timesteps: During the experiment, we observed that at T = 601, there was an optimal trade-off between generation quality and sampling speed.

   a. T > 601, both previous studies [1-3] and Fig. 3 indicate that the image structure is formed early in the sampling process. Therefore, when T > 601, the spatial structure of the image has not fully developed, and the semantic correspondence is ambiguous. At this stage, feature guidance can disrupt the image structure, thereby reducing the quality of the generated image.

   b. T < 601, the correspondence tends to be more accurate. However, too few sampling steps weaken the energy function's guidance, reducing style transfer effectiveness. Fewer timesteps with more sampling steps may also lead to overfitting due to minimal feature variation. Furthermore, as mentioned in the Appendix E, our method is robust and does not require highly precise correspondence.

   c. We conducted an ablation study on the sampling timesteps, with the results shown in the Fig. 8 in the Appendix E. It also proves that the best result is achieved when T=601.

5. Runtime of Semantix: In Appendix E and Table 5 of our initial version, we had already provided a detailed comparison and analysis of the computational speed and memory consumption of our approach against those of other methods. Since our sampler is guided by energy gradients, there is a slight increase in sampling time.

[1] Voynov, Andrey, et al. "p+: Extended textual conditioning in text-to-image generation." arXiv preprint arXiv:2303.09522 (2023).

[2] Agarwal, Aishwarya, et al. "A-star: Test-time attention segregation and retention for text-to-image synthesis." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Agarwal, Aishwarya, et al. "An image is worth multiple words: Multi-attribute inversion for constrained text-to-image synthesis." arXiv preprint arXiv:2311.11919 (2023).

Dear Reviewer uDZ6,

Thank you once again for your valuable feedback and comments! We have addressed each of your concerns individually. Could you kindly confirm whether they have resolved your issues? With the deadline for the discussion approaching, we would greatly appreciate it if you could let us know whether our response meets your expectations and update our overall rating accordingly.

Sincerely,

Semantix team

4. Clarification regarding Object Region: In our revised manuscript, we have included more detailed explanations of the procedure for obtaining the object region mask, as highlighted content in Lines 366–368. Specifically,

   a. Our masks are obtained as described in [1]. In brief, during the inversion process, we apply $k$-means clustering on the self-attention maps to generate the segment masks. For further details, please refer to the original paper [1]. b. Apply: After obtaining the masks, we apply them to the features, performing calculations and guidance only within the masked regions.

5. Conflict of Different Guidance: Spatial guidance and style guidance interact with each other, and during the sampling process, the results may favor the spatial reference over the style (or vice versa). To address this issue, we introduce semantic distance as a mitigating mechanism. Ablation experiments show that the proposed semantic distance term effectively avoids overfitting and balances style and spatial feature guidance. Therefore, incorporating semantic distance helps prevent spatial feature guidance from negatively impacting style transfer (or vice versa). For the detail of how Semantic Distance works, please refer to our response to other reviewers comments, and the added sections in our revised manuscript, e.g., Line 385-388.

6. Necessary of both terms: We would like to clarify that the semantic aspect of the transfer is actually achieved through style guidance. The regularization term serves to balance the interplay between style and spatial guidance. More specifically, when the style and context images exhibit weak semantic correspondence, the semantic guidance in the regularization term can further promote the injection of style features. Furthermore, when the generated image overfits the context image, the regularization term enhances style injection through semantic strengthening, achieving a balance between style and context features. Overall, both terms are indispensable for achieving a balanced semantic style transfer.

7. Typo of explanation of sg($\cdot$): Thanks for pointing that out. "sg" here refers to stop gradient. We have revised it in manuscript.

8. Guarantee of video motion consistency: Compared to other frame-based video editing frameworks, our model, which is based on a pre-trained T2V model, ensures stronger consistency. Furthermore, our proposed approach is a sampler that does not modify the pre-trained model. As a result, the motion consistency of the generated video model remains fully aligned with the motion consistency capabilities of the T2V model used, without introducing any additional damage.

9. Clarify the demonstrated results: We are committed to transparency and accuracy in our research and do not use cherry-picking to exaggerate our model's capabilities. All qualitative experiments were conducted without selection bias, and for the quantitative experiments, we used the same benchmark datasets as studies like StyleID [2], where cherry-picking is not feasible. Our code is available in the supplementary file, and we encourage reviewers to verify the results. Furthermore, we plan to release the code publicly once the paper is published.

10. Clear presentation: For some ambiguous descriptions in the manuscript, we have rephrased them in the revised version, which can be found in the highlighted sections of the newly submitted manuscript.

11. Symbol notation: Thanks for the suggestion. We have added the definitions of mathematical notation in the revised manuscript.

12. \cite format: Thanks for raising this issue. We have made the appropriate corrections in the revised manuscript.

Finally, we sincerely thank the reviewer once again for their valuable feedback, which has greatly enhanced the clarity and presentation of our work. If you have any further questions regarding our responses, we kindly invite you to raise them during the discussion phase.

[1] Patashnik, Or, et al. "Localizing object-level shape variations with text-to-image diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Chung, Jiwoo, Sangeek Hyun, and Jae-Pil Heo. "Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We sincerely appreciate the insightful comments from Reviewer ashT, which will surely enhance the quality of our paper. In response to your concerns, we would like to provide the following clarifications:

1. Concept Novelty: We have included additional discussion in our revised version Lines 164-173. Overall, while the related work [1] mentioned by the reviewer also introduces the concept of semantic style transfer, our definitions of semantics and the task of semantic style transfer differ significantly. We would like to elaborate on our innovations from the following perspectives:

   a. In terms of semantics, the key distinction lies in the extraction and utilization of semantic features. Unlike prior works that rely on segmentation labels [2, 3] or DSI2I [1], which leverages CLIP visual branch for dense semantic feature reuse but loses alignment with the natural language dimension, our method extracts semantic features with a natural language dimension from the pretrained T2I or T2V models. These features are then employed within an energy function to effectively guide semantic style transfer.

   b. In terms of tasks, DSI2I [1] is an I2I task that only extracts dense semantics and reuse it in given image. Instead, our work first defines a more expansive zero-shot semantic style transfer task, which is focused on generation. Rather than merely reusing the low-level features of a given image, e.g., color, we leveraged a powerful pre-trained diffusion model to generate novel style images that adhere to semantic constraints derived from the features of the given image.

   c. To the best of our knowledge, we are the first to unify style and appearance transfer through semantic understanding and mapping in visual generation tasks. Our work also innovatively introduces energy-guided semantic alignment for style transfer.

2. Selection of timesteps: during the experiment, we observed that at T = 601, there was an optimal trade-off between generation quality and sampling speed.

   a. T > 601, both previous studies[4-6] and Fig. 3 indicate that the image structure is formed early in the sampling process. Therefore, when T > 601, the spatial structure of the image has not fully developed, and the semantic correspondence is ambiguous. At this stage, feature guidance can disrupt the image structure, thereby reducing the quality of the generated image.

   b. T < 601, the correspondence tends to be more accurate. However, too few sampling steps weaken the energy function's guidance, reducing style transfer effectiveness. Fewer timesteps with more sampling steps may also lead to overfitting due to minimal feature variation. Furthermore, as mentioned in the Appendix E, our method is robust and does not require highly precise correspondence.

   c. We conducted an ablation study on the sampling timesteps, with the results shown in the Fig. 8 in the Appendix E. It also proves that the best result is achieved when T=601.

3. Selection of Hyper-parameters: $\omega$ is actually set as the widely used CFG value. And hyper-parameters in Eq. 5 are set heuristically based on proportion. The energy guidance functions are also similarly to CFG. Ablation studies in Fig. 13 show that performance is not highly sensitive to these hyper-parameters($\gamma$, $\lambda$).

[1] Ozaydin, Baran, et al. "DSI2I: Dense Style for Unpaired Exemplar-based Image-to-Image Translation." Transactions on Machine Learning Research.

[2] Shen, Zhiqiang, et al. "Towards instance-level image-to-image translation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[3] Bhattacharjee, Deblina, et al. "Dunit: Detection-based unsupervised image-to-image translation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[4] Voynov, Andrey, et al. "p+: Extended textual conditioning in text-to-image generation." arXiv preprint arXiv:2303.09522 (2023).

[5] Agarwal, Aishwarya, et al. "A-star: Test-time attention segregation and retention for text-to-image synthesis." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[6] Agarwal, Aishwarya, et al. "An image is worth multiple words: Multi-attribute inversion for constrained text-to-image synthesis." arXiv preprint arXiv:2311.11919 (2023).

Dear Reviewer ashT,

Thank you once again for your valuable feedback and comments! With the deadline for the discussion approaching, we would greatly appreciate it if you could let us know whether our response meets your expectations and update our overall rating accordingly. Please don’t hesitate to let us know if there are any remaining concerns, we would be glad to address them.

Sincerely,

Semantix team

4. Demonstration problems: We have reformatted Tab.2,3,4 and addressed the issue of varied font sizes. We will further optimize the Fig.2 in the camera-ready version to enhance its clarity.

5. Selections of inversion timesteps: During the experiment, we observed that at T = 601, there was an optimal trade-off between generation quality and sampling speed.

   a. T > 601, both previous studies[1-3] and Fig.3 indicate that the image structure is formed early in the sampling process. Therefore, when T > 601, the spatial structure of the image has not fully developed, and the semantic correspondence is ambiguous. At this stage, feature guidance can disrupt the image structure, thereby reducing the quality of the generated image.

   b. T < 601, the correspondence tends to be more accurate. However, too few sampling steps weaken the energy function's guidance, reducing style transfer effectiveness. Fewer timesteps with more sampling steps may also lead to overfitting due to minimal feature variation. Furthermore, as mentioned in the Appendix E, our method is robust and does not require highly precise correspondence.

   c. We conducted an ablation study on the sampling timesteps, with the results shown in the Fig. 8 in the Appendix E. It also proves that the best result is achieved when T=601.

We sincerely appreciate your kind and encouraging feedback again. If you have any further questions about our work or responses, we kindly invite you to raise them during the discussion phase.

[1] Voynov, Andrey, et al. "p+: Extended textual conditioning in text-to-image generation." arXiv preprint arXiv:2303.09522 (2023).

[2] Agarwal, Aishwarya, et al. "A-star: Test-time attention segregation and retention for text-to-image synthesis." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Agarwal, Aishwarya, et al. "An image is worth multiple words: Multi-attribute inversion for constrained text-to-image synthesis." arXiv preprint arXiv:2311.11919 (2023).

We sincerely appreciate the strong recognition and encouragement from Reviewer 84Mq, which validates our dedication to contributing meaningfully to the community. We have carefully reviewed your comments and incorporated revisions to address them and enhance the clarity of our work.

1. Motivation of Semantic Distances: Semantic distance is introduced to balance style and context features, preventing structural distortion and style deviation. As illustrated in Figure 7, the manifestations of overfitting can be observed. Without semantic distance, the images in the first row (boy and oil painting) excessively align with the style image (the facial features and clothing of the generated boy are as blurred as those of the woman in the oil painting), while those in the second row (black suit jacket and red pants) overfit to the context image (the suit jacket color become dark instead of red). Further unbalance results can be found in appendix.

   Technically, our semantic distance term serves as an L2 regularization. During the sampling process, we aim to optimize the proposed energy function. The style guidance and spatial guidance are treated as key components of the energy function, while the semantic distance serves as a regularization term. It is worth noting that the parameters of Style guidance or Spatial guidance are actually the corresponding feature vectors, while the Swapped Cross Attention feature contains the output of style guidance and spatial guidance. By constraining and penalizing the feature vectors involved in the computation of the regular terms, e.g. Style guidance and Spatial guidance, it ensures that the style and spatial feature vectors remain within a more compact space. This leads to a smoother solution space, reduces the risk of overfitting, and ensures stability throughout the sampling process.

2. Missing Reference: Thank you for your valuable reminder. We have adopted this work [1] as a baseline and conducted comprehensive qualitative and quantitative comparisons with it on both image and video tasks. The quantitative results are provided in the Tables below:

Table 1: Quantitative Comparison of Video Style Transfer

Table 2: Quantitative Comparison of Video Style Transfer

For qualitative results and more quantitative comparisons, please refer to the revised manuscript, e.g. Figs. 4,5 & Tables 1,2. The experimental results show that our method outperforms [1] in both image and video tasks.

3. Impact of Position Encoding: In our proposed method, positional encoding plays a crucial role for the following reasons:

   a. Position encoding encodes relative spatial information, which helps maintain structural integrity. Without it, generated images may be blurry or lose the relative position of features as shown in the Fig.7.

   b. In cases where semantic correspondence is ambiguous, we can control the strength of the positional encoding to achieve accurate spatial alignment, thereby preventing a degradation in generation quality.

[1] Wu, Zijie, et al. "Ccpl: Contrastive coherence preserving loss for versatile style transfer." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

We sincerely appreciate the recognition and the valuable comments from Reviewer thdC. Your feedback has greatly contributed to improving the quality of our submission. We kindly request you to review our responses to your suggestions:

1. Lacks logical coherence: we would provide a summary of the overall motivation and purpose of our method, as well as to clarify its underlying logic.

   Given a pre-trained diffusion model and a set of noise signals from DDPM inversion of the input images, we propose Semantix, an energy-guided sampler, to achieve zero-shot training-free semantic style transfer. The energy-guided sampler is defined by three distinct guidance components, as outlined below:

   a. Style Feature Guidance: To inject style features into the context image, we introduce style feature guidance, which minimizes the feature distance between the output and style features for effective style transfer.

   b. Spatial Feature Guidance: To avoid disrupting the spatial structure of the generated image during style feature injection, we introduce spatial feature guidance, which preserves the spatial integrity of generated image.

   c. Semantic Distance: To prevent overfitting to style or context features and to maintain an effective balance between style and structure, we introduce a semantic distance regularization term, which enhances the overall image quality.

   These three components work synergistically. Style feature guidance addresses the core task of semantic style transfer, which is the alignment of semantic styles, while spatial feature guidance ensures structural consistency between the generated image and the given image. Finally, semantic distance functions as a regularization term, mitigating overfitting to either style or spatial features.

2. Insufficient theoretical foundation and clear explanations: we explain the spatial feature guidance in detail below

   Spatial feature guidance aims to preserve the consistency of the image spatial structure before and after style transfer.

   Previous studies [1, 2] have shown that features in diffusion models contain spatial information and some methods [3,4] achieve context preservation by replacing these features. However, such approaches lack a mechanism to ensure consistency when utilizing these features, which can easily cause a mismatch in distribution between the introduced and original features, resulting in artifacts or highlight distortions in the generated images.

   In contrast, our approach preserves spatial structure by minimizing the distance between context and output features. Specifically, we compute the feature distance at corresponding positions during sampling and design an energy function to guide the sampling, aligning spatial structures and maintaining spatial integrity.

   We have also updated our manuscript to clarify and provide more specific descriptions of the content related to other methods. Kindly review the red part in the updated draft, e.g., Lines 370–377. The expression will be further improved in the camera-ready version.

[1] Patashnik, Or, et al. "Localizing object-level shape variations with text-to-image diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Tumanyan, Narek, et al. "Plug-and-play diffusion features for text-driven image-to-image translation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Chung, Jiwoo, Sangeek Hyun, and Jae-Pil Heo. "Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Alaluf, Yuval, et al. "Cross-image attention for zero-shot appearance transfer." ACM SIGGRAPH 2024 Conference Papers. 2024.

3. Lack of Literatures: In the revised version, we provide a literature review on the energy function, including the following keys:

   a. Theory of energy functions. Previous research interprets diffusion models as energy-based models [2], where the energy function guides and controls the generation process for precise visual outputs.

   b. Applications of energy functions. The energy function has a wide range of applications. Some methods use the energy function to guide image editing [3-5] and translation [6]. The energy function also shows significant potential in controllable generation, guiding generation through conditions such as sketch [7], mask [8], layout [9], concept [2], and universal guidance [10,1], enabling precise control over the output.

   The corresponding content has also been updated in the draft. We kindly invite you to review the highlighted parts in the “Related Works” section, Lines 181–188.

4. Clarification regarding spatial feature guidance:

   a. Why does it work? Previous studies [11, 12] have shown that features within diffusion models contain spatial information, and some methods [13, 14] use feature replacement for visual transfer and can preserve structure. In contrast, our approach minimizes the distance between context and output features to guide the generation process, preserving the original spatial structure. Specifically, we calculate the feature distance at corresponding spatial locations during sampling and use an energy function to align the spatial structure features, ensuring effective preservation of spatial integrity.

   b. Would it impact the other guidance? The answer is yes. To prevent the impact to the other guidance after introducing spatial and style feature guidance, we proposed a semantic distance regularization term. Ablation studies demonstrate that semantic distance term effectively mitigates the negative impact and maintains a balance between style and spatial feature guidance.

[1] Yu, Jiwen, et al. "Freedom: Training-free energy-guided conditional diffusion model." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Liu, Nan, et al. "Compositional visual generation with composable diffusion models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[3] Mou, Chong, et al. "Dragondiffusion: Enabling drag-style manipulation on diffusion models." arXiv preprint arXiv:2307.02421 (2023).

[4] Mou, Chong, et al. "Diffeditor: Boosting accuracy and flexibility on diffusion-based image editing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Epstein, Dave, et al. "Diffusion self-guidance for controllable image generation." Advances in Neural Information Processing Systems 36 (2023): 16222-16239.

[6] Zhao, Min, et al. "Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations." Advances in Neural Information Processing Systems 35 (2022): 3609-3623.

[7] Voynov, Andrey, Kfir Aberman, and Daniel Cohen-Or. "Sketch-guided text-to-image diffusion models." ACM SIGGRAPH 2023 Conference Proceedings. 2023.

[8] Singh, Jaskirat, Stephen Gould, and Liang Zheng. "High-fidelity guided image synthesis with latent diffusion models." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[9] Chen, Minghao, Iro Laina, and Andrea Vedaldi. "Training-free layout control with cross-attention guidance." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

[10] Bansal, Arpit, et al. "Universal guidance for diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[11] Patashnik, Or, et al. "Localizing object-level shape variations with text-to-image diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[12] Tumanyan, Narek, et al. "Plug-and-play diffusion features for text-driven image-to-image translation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[13] Chung, Jiwoo, Sangeek Hyun, and Jae-Pil Heo. "Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[14] Alaluf, Yuval, et al. "Cross-image attention for zero-shot appearance transfer." ACM SIGGRAPH 2024 Conference Papers. 2024.

5. Clarification regarding semantic distance:

   a. We firstly discussed the mentioned literature [1] in our manuscript. While both our method and theirs employ swapping self-attention strategy, there are differences between the two approaches in both terms of tasks and methods. In terms of tasks, their work focuses on text-based style generation, whereas our approach targets general semantic style transfer based on images, with applicability to both images and videos. In terms of computation, they inject style features by directly replacing the style features in the self-attention to achieve style transfer. In contrast, our method replaces the style features and then computes the L2 distance as an energy function to guide the generation, rather than directly injecting the features.

   b. Our semantic distance term serves as an L2 regularization in this context. During the sampling process, we aim to optimize the proposed energy function. The style guidance and spatial guidance are treated as key components of the energy function, while the semantic distance serves as a regularization term. It is worth noting that the parameters of Style guidance or Spatial guidance are actually the corresponding feature vectors, while the Swapped Cross Attention feature contains the output of style guidance and spatial guidance. By constraining and penalizing the feature vectors involved in the computation of the regular terms, it ensures that the style and spatial feature vectors remain within a more compact space. This leads to a smoother solution space, reduces the risk of overfitting, and ensures stability throughout the sampling process.

6. Inference Cost: It will not significantly increase the inference time, as we provide the sampling time in the Tab. 5. All feature maps are by-products of the single inference process and do not require additional inference. We can obtain all of them in the inversion process.

Finally, we sincerely appreciate your valuable suggestions and comments, which have helped us improve the clarity and completeness of our presentation. We hope our responses have adequately addressed your concerns. Should you have any further points, we kindly invite you to share them with us during the discussion phase.

[1] Jeong, Jaeseok, et al. "Visual Style Prompting with Swapping Self-Attention." arXiv preprint arXiv:2402.12974 (2024).

Hi. Actually you can find runable experimental codes in attachment.