RB-Modulation: Training-Free Stylization using Reference-Based Modulation

Dear Reviewer YTQn,

Thank you for recognizing that our method achieves content/style disentanglement without training, with comprehensive evaluations, state-of-the-art performance, and impressive qualitative results. We are glad that you found the paper well-written and easy to follow. Below, we address your remaining questions.

(Q1) The choice of the baseline model (StableCascade) seems like a design choice. Authors might want to provide additional results on more widely adapted baseline models (e.g. SDXL), since this can show whether the inherent high fidelity results stems from the potency of StableCascade or their proposed method.

(A1) We have now included Figure 16 to demonstrate that (1) the plug-and-play nature of RB-Modulation and (2) the inherent high fidelity results indeed stem from our algorithmic innovation.

(Q2) Most of the qualitative results seem to be on generated reference style/content images. The proposed method seems to be agnostic to whether the reference image is generated or not. Results with non-generated reference images would be appreciated.

(A2) To ensure a fair comparison, we evaluated our method on the same dataset used in prior work StyleAligned. Following the reviewer’s suggestion, we have now included non-generated reference images in Figure 14.

(Q3) Authors use Consistent Style Descriptor (CSD) to extract style features. Are there alternatives for this module? What was the main reasoning behind this choice?

(A3) RB-Modulation is a generic framework that can incorporate various feature extractors in its terminal cost. For instance, neural networks could be used to extract specific features like lighting, shadow, or color. For style transfer, extractors such as ALADIN-ViT or ALADIN-NST [1] can also be used. We chose CSD due to its large-scale training on 511,921 images and 3,840 style tags, covering a wide variety of practical styles.

(Q4) For content style composition, it seems like the content is also processed in the same manner as style. Is this optimal in terms of content identity preservation? In algorithm 1 and 2, it seems like the image latent is only updated by the loss w.r.t the style descriptor loss. Would this not result in information loss about the identity of the content?

(A4) For content-style composition, the content is processed differently from the style, as discussed in L264-269.

For clarity, we summarize the key points from that section here. The pre-trained ViT in CSD includes two attention heads—one for style features and one for content features—which can be selected based on the application. We use the content attention head to extract content-specific features and the style attention head to extract style-specific features. In Algorithms 1 and 2, we simplify notation by using $\Psi(\cdot)$ as a general feature extractor; it represents a style feature extractor for stylization and a combination of content and style extractors for content-style composition.

(Q5) In figure 1, the resulting images do not seem to hold the content identity of the reference content image, but instead seems to heavily rely on the given text description. Using examples from figure 2, additional content-style composition results on including/excluding Kc,Vc would be appreciated.

(A5) The base model is a text-to-image generative model, so it naturally relies heavily on the given text description, which typically provides coarse-grained control. The AFA and SOC modules are designed to add fine-grained control beyond the base model's capabilities. To clarify this, we have now included experiments with and without $K_c, V_c$ in Figure 21 as suggested by the reviewer.

(Q6) Does RB-Modulation also work for recent flow based models? Does the standard reverse SDE for the OU process hold for flow based models, or will the optimal controller have to be redefined?

(A6) Yes, RB-Modulation can work for recent flow-based models, which is an excellent direction for future research. Here are some preliminary thoughts: The standard reverse SDE for the OU process doesn’t hold directly for flow models, as the forward process in flows is not governed by an OU process; therefore, its time-reversal differs from the standard reverse SDE of the OU process. Instead, the forward process in flows is governed by a different SDE with different drift and diffusion coefficients, leading to a distinct reverse SDE. Importantly, the theoretical tools used in Appendix A.2 and Appendix A.3 hold for general SDEs, and therefore can be used to rederive the optimal controller for this new SDE. Thank you for the insightful question.

Reference

[1] Ruta, Dan, Gemma Canet Tarres, Alexander Black, Andrew Gilbert, and John Collomosse. "Aladin-nst: Self-supervised disentangled representation learning of artistic style through neural style transfer." arXiv preprint arXiv:2304.05755 (2023).

Dear Reviewer iCFz,

Thank you for acknowledging that our optimal controller formulation is novel over the majority of prior works in this category. Below, we address your remaining questions.

(Q1) A novelty of the approach is the ‘AFA module’ proposed which separates style and content using cross-attention to mitigate content leakage from the reference image. The authors claim but do not comprehensively show this allows controlled integration of style and content elements.

(A1) We provide analysis in Figures 8 and 10 to demonstrate the controlled integration of style while effectively separating content from the reference style image. Additionally, Figures 11 and 12 offer further qualitative results that highlight content leakage. We have added a new Figure 18 to show controlled integration of style as suggested by the reviewer. We also included Figure 17 to demonstrate control of structural elements via ControlNet.

(Q2) However the practicalities of the problem seem to demand iterative reverse/feed-forward steps which negates a lot of the benefit of theoretical arguments and underlying assumptions presented.

(A2) We believe there may have been a misunderstanding regarding this key aspect of our paper. The iterative refinement only happens in the reverse process and there are no feed-forward steps. This retains the benefits because (1) it is performed in the latent space (24x24), where gradient descent adds minimal inference time, as demonstrated in Table 4; and (2) our proximal Algorithm 2 avoids the need for back-propagation through the score network, as outlined in Section 4 (Lines 212-229). Overall, RB-Modulation matches the efficiency of training-free methods and provides a 5-20X speedup over training-based approaches such as StyleDrop [11] and ZipLoRA [10]. We hope this clarifies the efficiency and practical benefits of our approach.

(Q3) Deeper comparisons could have been made to the NST literature which bear some relevance both in task and in approach particularly the recent diffusion based approaches. How the proposed approach fundamentally differs especially considering the use of cross-attention in AFA versus these works.

(A3) The proposed method is fundamentally different from these approaches [1,2,3]. We have added these related works in the revised paper (L1302-1318).

DiffusionDisentanglement [1] relies on VGG 16 for perceptual loss and ViT/B-32 for directional CLIP loss, which is prone to content leakage [4]. In contrast, our method injects features exclusively from the style attention head of the fine-tuned CSD-CLIP model, ensuring better content-style disentanglement in the AFA module. Additionally, our approach introduces an optimal controller framework to minimize a terminal cost, offering a richer design space and superior controllability compared to [1]. Lastly, our method reduces sampling bias by optimizing the controller $u$ in Algorithm 1, unlike [1], which can provably fail to sample from the correct posterior [5].

PARASOL [2] and Diff-NST [3] are training-based methods, while our approach is entirely training-free. PARASOL [2] requires supervised data via a multi-modal search (Section 3.1 in [2]) to train both the denoising U-Net and a projector network. Diff-NST [3] trains the attention processor by targeting the 'V' values within the denoising U-Net architecture. In contrast, our method uses two training-free modules: the AFA module replaces the default attention processor in the denoising U-Net to disentangle content and style, and the SOC module minimizes a terminal cost to enhance stylization and content-style composition.

Dear Reviewer iCFz,

Thank you for your feedback on the rebuttal. Below, we address your remaining two points.

(Q1) On the first point, i cited three prior works that reduce the novelty claims made over the fine grained controllability. Unfortunately these works have not been discussed in the revised paper, but simply a subsection added to the appendix repeating the rebuttal text. Please properly integrate this response into Sec.1 / 2 the paper, to contextualise the novelty claims being made in order to address my first point.

(A1) We have now integrated our response into Sec 2 (L117-120 and L129-131) with further details deferred to Appendix B.7 due to space limitations.

(Q2) On the second point, Figure 18 has been added that shows gradual increase in style transfer strength for controllability. This partially addresses my concern as the figure shows the effect of increasing the weight of the effect. However it doesn't show interpolation in the continuous style space. Showing the latter would have better evidenced the value and practicality of the continuous representation.

(A2) We are glad that the newly added results in Figure 18 partially addressed your concerns. In response to the reviewer’s suggestion, we have now included Figure 19 (L1294-1295), which demonstrates interpolation in the continuous style space. We thank the reviewer for the great suggestion. Figure 19 shows an additional capability of our method to generate novel and unseen styles by continuously interpolating between reference style images.

Concluding Remarks

We hope the above clarifications and the additional experiments in the revised draft have successfully addressed all your concerns. If satisfied, we kindly request you to consider revising your score. We are happy to address any additional points during the discussion phase.

Best,

Authors of Paper #11075

Dear Reviewer NtHN,

Thank you for noting that our approach is the first novel training-free personalization framework with theoretical justifications connecting optimal control and reverse diffusion dynamics. We also appreciate your positive feedback on the promising qualitative and quantitative results for stylization and content-style composition, supported by a user study verifying its effectiveness. Below, we address your remaining questions.

(Q1) As the title said, the method aims for personalization. Therefore, I recommend the author add some comparison with classical personalization methods, not just style transfer methods.

(A1) In Figure 15, we have now added comparison with classical personalization methods (e.g., DreamBooth, IP-Adapter and DreamBooth+LoRA) and not just style transfer methods (e.g., StyleDrop and InstantStyle).

(Q2) The section on ablation study in the paper is too brief. Considering that the AFA and SOC models are central to the article, I recommend that the authors include numerical comparisons and more qualitative comparisons.

(A2) Due to space limitations, we deferred a detailed ablation study including numerical and additional qualitative comparisons to Appendix B.3 (e.g., Figure 8) and Appendix B.5 (e.g., Figure 10). We kindly invite you to review these sections and let us know if further results would be helpful. We would be glad to provide additional results if needed.

(Q3) For the content-style composition experiment, I recommend the author to provide more complicated ref content (dog, sloth, cat are simple cases, or you can change the color of the ref to see whether the proposed method has achieved better content-style composition) to verify its effectiveness.

(A3) We would like to note that, even for simpler cases (e.g., dog, sloth, cat) as shown in Figure 4, existing training-free content-style composition methods struggle to handle complex prompts. In contrast, our method achieves results comparable to a leading training-based method, ZipLoRA, with at least a 20X speedup (Table 4). We appreciate the reviewer’s suggestion to change the color of the content image, and we have now included Figure 22 to demonstrate our method’s robustness in capturing content-specific features independently of color.

(Q4) For situations that require ControlNet to control the layout, can the method generalize well to these scenarios?

(A4) For situations requiring ControlNet to control the layout, our proposed method can easily adapt, as shown in the newly added Figure 17. Since ControlNet enhances only the denoising network, we can still minimize the terminal cost associated with the expected terminal state, ensuring that SOC remains effective in practice. Similarly, the AFA module can generalize seamlessly by substituting the default attention processor with the AFA module, even when ControlNet is present.

Dear Reviewer ufZj,

Thank you for highlighting that the paper has several merits, especially in training-free stylization. We appreciate your encouraging feedback that the idea and theoretical contributions are quite good, and this could lead to more interest in similar works. Below, we address your remaining questions.

(Q1) Title needs revision. The title RB-Modulation is pretty weird and does not suit the paper's excellent contribution. This must be revised to put the paper in the correct research context.

(A1) We would appreciate any suggestions the reviewer may have for a revised title. A key contribution of this paper is the modulation of the standard reverse SDE’s drift field using a stochastic optimal controller and disentangled attention features from a reference image. If satisfactory, we propose the following alternatives: “RB-Modulation: Stochastic Control Meets Disentangled Attention for Personalization” or “RB-Modulation: Training-Free Personalization using Reference-Based Modulation”.

(Q2) Missing references such as Deep Image Prior need to be cited.

(A2) We thank the reviewer for pointing us to the related work, Deep Image Prior [1], which we have now cited in the revised version of our paper.

(Q3) Sorry to quote from your limitation. However, I am curious to know how feature descriptors can help this work.

(A3) In our optimal control framework, a feature descriptor is used within the terminal cost function. We employ a contrastive style descriptor (CSD) that is trained on 3,840 style tags [2]. However, some unique styles may still fall outside its scope. Improving the feature descriptor could enhance our work to capture these distinctive styles.

Moreover, adopting an alternative feature descriptor could broaden downstream applications beyond personalization. For instance, a descriptor capable of capturing attributes like lighting, shadow, or haze could facilitate tasks such as relighting an image or removing shadows and haze, which are extremely important in biometric authentication platforms.

Reference

[1] Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. "Deep image prior." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446-9454. 2018.

[2] Somepalli, Gowthami, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping, Abhinav Shrivastava, and Tom Goldstein. "Measuring Style Similarity in Diffusion Models." arXiv preprint arXiv:2404.01292 (2024).

Dear Authers,

Thank you for the authors' response. I acknowledge that Alg. 2 and AFA are valuable. Considering the high similarity between Alg. 1 and FreeDom, reviewers may have overestimated the contribution of this paper. Furthermore, FreeDom and its subsequent work, "DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models," are capable of performing a wide range of editing tasks. This paper only attempted stylization, which may appear somewhat limited.

Best,

Yuchen

Dear Yuchen,

Thank you for bringing FreeDoM [1] to our attention. We have now included a citation to this missing reference, highlighting three technical differences between FreeDoM and our approach (L1311-1321):

1. There are two algorithms proposed in FreeDoM. In their first Algorithm 1, the conditional guidance term $\nabla_{x_t} \log p(c|x_t)$ is approximated by the gradient of an energy function $\nabla_{x_t} \mathcal{E}(c,x_t)$. Our Algorithm 1 differs from this approach by replacing $\nabla_{x_t} \log p(c|x_t)$ with a controller guidance parameter $u$ that is optimized to minimize this approximation error. Furthermore, recent work [2] shows that such an optimization step, as in our Algorithm 1, is provably robust for posterior sampling, whereas Algorithm 1 in FreeDoM is provably biased. We note that either algorithm (ours or the one in FreeDoM) is hard to scale up to high dimensions because evaluating the gradient requires backpropagating through the score network, which is expensive for large-scale models.

2. In FreeDoM, Algorithm 2 is proposed to reduce the error in Algorithm 1 by integrating an efficient version of a time-travel strategy, where each denoised latent is noised back and forth to alleviate the poor guidance problem. Since this resampling process is expensive, we seek an alternative approach motivated by an optimal control framework. In our approach (Algorithm 2), we optimize the expected terminal state to satisfy the terminal constraint (e.g., align the style of the generated image with that of the given input), which is significantly different from FreeDoM. Our proximal approach allows us to avoid gradient computation through the denoising score network which is expensive for large-scale foundation models, such as SDXL or StableCascade. Indeed for StableCascade, it is not straightforward to even implement Algorithm 1 because StableCascade is a hierarchical diffusion model. We have discussed this at length in Section 4 (L216-234) and Appendix B.1.

3. Finally, we propose an Attention Feature Aggregation (AFA) module that effectively disentangles the content and style by leveraging a contrastive style descriptor (Section 4(b)). Our ablation study in Section 6.1, Appendices B.5 and B.6 demonstrate the importance of the AFA module in stylization and content-style disentanglement. We would like to note that FreeDoM relies on the standard attention processor, which suffers from content leakage as discussed in prior works [3,4].

References

[1] Yu, Jiwen, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. "Freedom: Training-free energy-guided conditional diffusion model." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23174-23184. 2023.

[2] Xu, Xingyu, and Yuejie Chi. "Provably robust score-based diffusion posterior sampling for plug-and-play image reconstruction." arXiv preprint arXiv:2403.17042 (2024).

[3] Hertz, Amir, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. "Style aligned image generation via shared attention." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4775-4785. 2024.

[4] Wang, Haofan, Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. "Instantstyle: Free lunch towards style-preserving in text-to-image generation." arXiv preprint arXiv:2404.02733 (2024).

[Thank you for the authors' response. I acknowledge that Alg. 2 and AFA are valuable. I find that the proximal algorithm 2 proposed in this paper is also an existing method in the field of image processing. It's a technique that's been used many times on other missions. In particular, "Manifold preserving guided diffusion" has tried this technique on FreeDom including stylization task.]

Thank you for acknowledging (in comment 1/2) that Algorithm 2 and AFA are indeed valuable contributions. We respectfully disagree with the comment regarding overestimation of our contributions: in both theoretical development and the details of implementation, FreeDoM significantly differs from our approach. Our previous response clearly articulated three major technical differences between these two approaches. Beyond these differences in the algorithm, your comments are also ignoring the theoretical novelty, which are important as these are where the algorithmic differences originate.

More broadly, with respect to the current literature, we believe that the contributions of this paper are significant given the fact that: (a) it provides a theoretical framework for analyzing diffusion models via stochastic control theory; (b) it identifies an important source of content leakage and presents two training-free modules (SOC and AFA) to mitigate this leakage by disentangling the treatment of content and style via their respective attention heads; (c) large-scale experiments achieving a new state-of-the-art performance in stylization and content-style composition. To validate our contributions, we have compared with the current state-of-the-art InstantStyle [ArXiv’2024], other leading methods including StyleAligned [CVPR 2024 Oral], DreamBooth [CVPR 2023], and ZipLoRA [ECCV 2025].

Novelty of SOC: We are not aware of prior works that view stylization and content preservation as the solution to a stochastic optimal control with terminal constraints. Our formulation provides a novel theoretical perspective and this is valuable because it allows the derivation of new algorithms from a principled perspective: the resulting algorithm differs in that the guidance function is an optimization problem with respect to the style constraints. This constrained stochastic optimal control view is the key contribution – once we have developed this view, the proximal algorithm (Algorithm 2) emerges naturally. The proximal method is a well known optimization technique and has been used in many fields including image processing. Indeed as reviewer ufZj commented, this also has connections to “training-free stylization problems similar to Deep Image Prior”, which effectively leads to a proximal method for inverse problems. Our use of the proximal method is unique in the sense that it uses disentangled representation derived from the style and content attention heads of a pre-trained ViT (see also detailed discussion toward the end of this response). Until now, all the prior works relied on the standard CLIP which suffers from content leakage.

Novelty of AFA: (In response to the comment we received through email from the Openreview portal, which seems to have been deleted on the webpage: “[I also find that AFA has been proposed in DragonDiffusion and used to control global appearance similarity.]”) We believe that there may have been a misunderstanding regarding the attention mechanism proposed in AFA and the above reference DragonDiffusion. DragonDiffusion uses DDIM inversion to find the initial latent and construct a memory bank, which cannot be applied to hierarchical diffusion models, such as StableCascade. Further, at the time of writing our paper, the state-of-the-art method for stylization called InstantStyle already highlighted the issues of content leakage in such methods. We have compared our approach with this state-of-the-art method for stylization and other leading methods including StyleAligned [CVPR 2024 Oral], DreamBooth [CVPR 2023], and ZipLoRA [ECCV 2025].

[Furthermore, FreeDom and its subsequent work, "DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models," are capable of performing a wide range of editing tasks. This paper only attempted style transfer, which may appear somewhat limited if viewed as an incremental work.]

We have compared our approach with the current state-of-the-art methods in style transfer as discussed above. Beyond stylization experiments considered in these prior works, we also demonstrate superior performance in content-style composition. We provided empirical evidence showcasing capabilities of our method in the same experimental setting as these promising methods in the personalization community.

I agree.

I would like to keep the rating provided the title is revised something like below.

RB-Modulation: Training-Free Stylization using Reference-Based Modulation

Further, I request authors not to make tall claims like personalization in the text of the paper. Over punching things which are not justifiable does not augur well for the advancement of research.

Personalization needs to perform transformations to source images (e.g., rotating the view, changing clothes, scenes, postures, etc.). Emphasizes facial and feature consistency. This is far beyond just stylization. If the article emphasizes personalization, it should have the function of other personalization works.

We thank the reviewers and Yuchen Liang for engaging with us in the discussion. Following the suggestion, we will change the title to: “RB-Modulation: Training-Free Stylization using Reference-Based Modulation”, and update the text to avoid emphasizing personalization in the revised version.