Minimal Impact ControlNet: Advancing Multi-ControlNet Integration

Q3: Balance between the first stage and the second stage.

We apologize for not including the related experiment initially. To address this, we have added the $Asym$ component to the baselines in Figure 4.(b) as part of the second experiment in Common Concern 1.

In our experiments, training Stage-2 alongside Stage-1 proved challenging, likely due to the higher variance of the Stage-2 loss as estimated by Hutchinson’s trace estimator [4]. To address this issue, we adopted a post-training approach for Stage-2, restricting it to 2000 steps. Experimental results show that this shorter training duration is sufficient for the loss to effectively achieve its objective and consistently enhance FID performance across different tasks.

Q4: Extend our methods by first separating foreground and background for guided generation and then synthesizing the layers.

This is a highly promising idea. With the rapid advancements in base models, recent work [5] has shown the capability to directly generate images layer by layer. In such scenarios, challenges similar to those addressed in our problem settings are likely to emerge. We believe that applying our method to these new approaches offers a compelling direction for future research.

[1] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Sener, Ozan, and Vladlen Koltun. "Multi-task learning as multi-objective optimization." Advances in neural information processing systems 31 (2018).

[3]Désidéri, Jean-Antoine. "Multiple-gradient descent algorithm (MGDA) for multiobjective optimization." Comptes Rendus Mathematique 350.5-6 (2012): 313-318.

[4] Hutchinson, Michael F. "A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines." Communications in Statistics-Simulation and Computation_18.3 (1989): 1059-1076.

[5]Zhang, Lvmin, and Maneesh Agrawala. "Transparent image layer diffusion using latent transparency." arXiv preprint arXiv:2402.17113 (2024).

We extend our heartfelt gratitude to reviewer TnUC for recognizing the depth of our problem analysis and the theoretical contributions of our work. Your detailed suggestions helped us include additional details in the preliminaries and provide a clearer explanation of the conservativity loss, enabling our work to reach and resonate with a broader audience.

W1-3: About writing.

• For ControlNet[1], we will reference the work earlier in the text and provide a brief introduction to its definition.
• We apologize for the lack of clarity. Because the signal has a shape like $C \times H \times W$, and the silent control signal lies in the $H \times W$ portion, the term “related areas” refers to the region within $H \times W$ where the silent control signal is located.
• Sorry for the lack of citations about Pareto optimal solution, We will add the related work[2-3] in the final version.

W4: More analysis on the balancing of coefficients among multiple control signals.

For our methods, the combination of different features varies across timesteps and layers, making it challenging to analyze systematically. However, we compare our method with different fixed coefficients. We introduce fixed coefficients for the first control signals, labeled ControlNet$^{0.5}$ and ControlNet$^{1.5}$, maintaining a total scaling factor of 2.0 to match the original ControlNet and our feature combination method. The FID performance is shown in the following table.

We observe that our dynamic balanced coefficients generally outperform fixed coefficients, demonstrating the effectiveness of balancing of coefficients among multiple control signals.

W5-6: A more detailed ablation.

We apologize for not providing a clear ablation analysis of our three contributions earlier. To address this, we first added the $Asym$ component for the baselines in Figure 4.(b) as part of the second experiment in Common Concern 1. Furthermore, we have included a comprehensive ablation study, as detailed in Common Concern 2.

Q1: Symbols in Equation (8) and (9).

We apologize for the lack of clarity. Here, $v_1$ and $v_2$ are simply vectors of the same shape. What we aim to introduce is the operator symbol $\lambda_i$. For each layer, $\lambda_i(\cdot, \cdot)$ takes the features of the corresponding layer as input. Consequently, the properties at different layers naturally differ due to the layer-specific inputs and operations.

Q2: Further clarification about conservativity.

We apologize for not making this clear.

TL;DR: If a vector field is the gradient of a scalar function, it exhibits the property of conservativity. For example, in real space, the gravitational field is the gradient of the gravitational potential energy, which makes the gravitational field conservative.

In the context of Diffusion Models, the score function that the neural network aims to fit is expressed as $\nabla_{\mathbf{x}_t}\log p_t(\mathbf{x}_t)$. This gradient-of-a-scalar-function formulation inherently guarantees that the score function exhibits the conservativity property.

Dear Reviewer TnUC,

Thank you for your thoughtful and insightful feedback. We have carefully addressed your concerns, including adding more details about ControlNet and conservativity loss, refining the manuscript, and conducting more comprehensive ablation studies to support the three key contributions.

As the discussion phase draws to a close, we welcome any additional questions or suggestions you might have. We are happy to engage further and address any remaining points of concern.

Dear Reviewer TnUC,

As the discussion stage approaches its conclusion, we would like to inquire if you have any additional questions or suggestions. We would be delighted to engage in further discussion with you.

We sincerely thank reviewer DjNC for acknowledging the relevance of our problem settings and the soundness of our work, particularly the theoretical aspects. Your suggestion to provide clearer ablation studies has greatly improved the clarity, comprehensibility, and overall completeness of this work.

W1: The second stage is computationally expensive.

Yes, the conservativity loss requires constructing a second-order computational graph, which is computationally expensive. However, we employ the following strategies to improve efficiency in our experiments:

• FP16 Precision: We use FP16 precision, which, although not natively supported for second-order computational graphs on accelerate, effectively reduces memory usage and computation time.
• Simplified Loss Function: We develop a simplified loss function, $\mathcal{L}_{QC}^{simple}$, which avoids constructing a second-order computational graph for the encoder part of the U-Net.
• Selective Flash Attention: Since Flash Attention does not currently support backpropagation through second-order computational graphs, we apply Flash Attention only to components that do not require gradient backpropagation for a second-order graph.
• Efficient Training Steps: The training consists of only 2000 steps. As a result, the second-stage training requires approximately 7 hours, which is entirely reasonable compared to the 48+ hours needed for the first-stage training.

W2: Need a clear version of ablation.

We apologize for not providing a clear ablation analysis of our three contributions earlier. To address this, we first added the $Asym$ component for the baselines in Figure 4.(b) as part of the second experiment in Common Concern 1. Furthermore, we have included a comprehensive ablation study, as detailed in Common Concern 2.

W3: Some other details.

• We will refine the notations and symbols to enhance clarity and understanding. Details related to layers will be hidden in the main text, with comprehensive explanations provided in the appendix for those seeking further information.
• We apologize for any inconvenience caused. In Figure 6, the shape represents the method, and the color represents the condition. We will revise and repaint the figure to make it more readable.
• In line 121, the value under $f$ indeed represents the layer number.

We are deeply grateful to reviewer ZUxW for recognizing the significance of the issues we aim to address and acknowledging the strong performance of our methods. Your valuable assistance in conducting experiments across detailed, brief, and null conditions has greatly enhanced the persuasiveness and robustness of our proposed approach.

W1: Why is it not the canny signal that suppresses the openpose signal?

This is an important question, and we apologize for not explaining it more clearly. First, this phenomenon is supported by our experimental results. We believe the underlying cause lies in the feature injection and combination process. Specifically, in regions that should be influenced by the canny control signal, the openpose “silent” control signal may take precedence, effectively “speaking louder” than the Canny signal. Consequently, these regions generate content that aligns more closely with the OpenPose “silent” control signal.

W2-1: Lack of the corresponding prompts.

We apologize for not including detailed information about the prompts. For single-condition scenarios, the prompts are taken directly from the dataset, while for multi-condition scenarios, we concatenate the prompts for each condition. In the final version of our work, we will provide all the prompts used for the conditions.

Specifically, regarding Figure 5, we have extended it into Figures 7 and 8 and included the related prompts for better clarity and completeness.

W2-2: Add comparison with detailed/brief/null text conditions. on TV and visual results.

This is an excellent suggestion for evaluating our work. Regarding the TV and FID comparison, we have provided detailed experimental results and analysis in the first additional experiments addressed in Common Concern 1.

For the visual results, we present a more detailed comparison in Figures 7 and 8 on page 18 of the updated PDF. These figures illustrate examples under detailed, brief, and null text conditions, highlighting the corresponding visual outputs. From this comparison, we have observed the following key findings:

• To implement the conservativity loss, we use SD 1.5 as the base model. However, under null text conditions, CFG is not applicable, leading to less visually appealing results. Despite this limitation, our model clearly demonstrates the ability to generate textures effectively under the silent control signal, whereas the original ControlNet fails to achieve this.
• More detailed text conditions lead to better visual results overall.
• Our conservativity loss improves the model’s alignment with prompts. For instance, in the Figure 7, our 2-stage model generates a snow-filled background, and in the Figure 8, the bread appears more complete.

We extend our heartfelt thanks to reviewer ESQy for recognizing the clarity and systematic nature of our solution, as well as the robustness of our comprehensive experiments. Your valuable input on refining the experiments for single conditions has significantly enhanced the completeness of this manuscript. Additionally, your insightful suggestions have provided us with valuable direction for exploring future work.

W1: Evaluate the performance in more than two conditions.

This is an excellent suggestion for future work. However, addressing more than two conditions poses a significant challenge: unlike the two-condition case, a straightforward analytical solution is not readily available. Instead, iterative methods are typically employed to approximate or achieve the final combination. We are currently investigating the use of the Frank-Wolfe Algorithm[1] to tackle this issue. Upon completing the experiments, we plan to include the corresponding results and analysis in the final version of our work.

W2: Conservativity loss on single condition.

In response to Common Concern 1, we present detailed experimental results and analysis from the first experiment, comparing MIControlNet (1-stage) with MIControlNet (2-stage) to perform an ablation study on the conservativity loss.

[1] Sener, Ozan, and Vladlen Koltun. "Multi-task learning as multi-objective optimization." Advances in neural information processing systems 31 (2018).