FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation

We sincerely thank the reviewer for acknowledging the novelty and performance of our paper. We hope the following answers reflect your questions.

Quantitative comparison and ablation on ControlNeXt..

We appreciate the reviewer’s concern regarding the need for additional comparisons with efficient control models. In response, we have conducted further experiments on ControlNext and Omini-Control[1], as mentioned in our reply to Reviewer eHec. It is important to highlight that while optimizing control block efficiency is valuable, our focus is different: we propose a dynamic routing strategy that adaptively determines the most efficient control strategy across different time steps and samples. This approach complements existing efficient control methods rather than solely aiming to reduce the cost of control blocks. Our experiments confirm that integrating our method with these efficient control models further enhances their performance while improving efficiency.

-[1] Tan, Z., Liu, S., Yang, X., Xue, Q. and Wang, X., 2024. OminiControl: Minimal and Universal Control for Diffusion Transformer. arXiv e-prints, pp.arXiv-2411.

The paper lacks explanation or ablation study on how to determine the hyperparameter $\lambda_C$

We appreciate the reviewer’s request for further clarification on determining the hyperparameter $\lambda_C$​ in Equation (18). $\lambda_C$ serves as a scaling factor to balance different objectives: the diffusion loss optimizes image quality, while $L_C$ regulates and enforces control block sparsity. To ensure the trained gating mechanism achieves the desired sparsity while maintaining generation quality, $\lambda_C$​ needs to be tuned empirically. While the optimal value may vary slightly across models, our experiments indicate that setting $\lambda_C$ = 0.5 provides the best trade-off in practice. To compute $L_C$​, we utilize a precomputed block-wise FLOPs lookup table, following methodologies from prior work ([a], [b], [c]). This approach ensures an efficient and structured way to regulate computational cost while preserving performance.

-[a] Meng, L., Li, H., Chen, B. C., Lan, S., Wu, Z., Jiang, Y. G., & Lim, S. N. (2022). Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12309-12318).

-[b] Rao, Y., Liu, Z., Zhao, W., Zhou, J., & Lu, J. (2023). Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 10883-10897.

-[c] Han, Y., Liu, Z., Yuan, Z., Pu, Y., Wang, C., Song, S., & Huang, G. (2024). Latency-aware unified dynamic networks for efficient image recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence.

The desired sparsity γ needs to be specified before training. During inference, is it possible to manually adjust the number of activated blocks to achieve an efficiency-performance trade-off?

We appreciate the reviewer’s suggestion regarding an inference-time scaling mechanism. To explore this, we conducted additional experiments where a ControlNet-Large model was trained with all blocks activated on a segmentation mask control task. During inference, we dynamically adjusted the gating threshold to control the number of activated blocks. Our results confirm that scaling activation blocks at inference is feasible, leading to better performance than the ControlNet baseline while maintaining comparable FLOPs and inference speed. However, the performance does not fully match that of the $\gamma$-aware trained version as we proposed in the paper, indicating that explicit training with sparsity constraints remains crucial for achieving the optimal efficiency-performance trade-off. Detailed results are presented below.

Issue of parameters

We acknowledge the reviewer’s concern about parameter count. Our goal is not just to reduce control block cost but to develop a dynamic routing strategy that optimally adapts efficiency across time steps and samples. As shown in our experiments and discussed in our response to Reviewer eHec, integrating our method with efficient control models further improves both performance and efficiency, reinforcing its broad applicability of our methods.

We sincerely thank the reviewer for the detailed and constructive feedback.

ControlNext.. LoRA-based… We appreciate the reviewer’s feedback regarding the comparison with other methods. While a direct comparison is not applicable (as our work focuses on control block integration rather than parameter fine-tuning), we have instead integrated our methods with two representative approaches: ControlNeXt and Omini-Control [1] (a recent popular LoRA-based control method). Specifically, for Omni-Control, instead of following Flux’s approach of concatenating control tokens into new tokens, it appends condition image tokens with noisy image tokens as a longer sequence and leverages LoRA to jointly process them.

Our results show that, unlike methods that control all blocks by default, our approach achieves superior performance with fewer activated blocks, demonstrating its adaptability and broader applicability. Detailed comparisons follow below.

[1] Tan, Z., et,al. OminiControl: Minimal and Universal Control for Diffusion Transformer. arXiv e-prints, pp.arXiv-2411.

On segmentation mask

On Canny

Table 4.. Gamma=0.3…Further exploration..gamma value…

We appreciate the reviewer’s observations on FlexControl’s performance across $\gamma$ values. Even at $\gamma$ = 0.3 our method surpasses standard ControlNet while being significantly more efficient. Though slightly behind the more computationally expensive ControlNet-Large, it achieves over three times the efficiency, highlighting its effectiveness. To provide further insights, we conducted additional ablation studies on segmentation and Canny tasks, analyzing $\gamma$ values from 0.2 to 0.8. The results, detailed below, illustrate the trade-offs between efficiency and performance:

On segmentation mask

On Canny

We apologize for any confusion caused by our text and table presentation that may have led to misunderstandings. We will refine our wording to ensure greater clarity in the camera-ready version.

SD3 adaptation

In SD3 tasks, we select all transformer blocks as candidates and use our dynamic routing strategy to flexibly decide which transformer block to add control to. We copy all parameters in a block for both modalities.

We hope the answer below solves all the clarity issues.

The proposed method requires more parameters than the typical controller.

Yes, our method requires more parameters than standard ControlNet. However, compared to ControlNet-Large, it achieves better generation quality and controllability while halving inference FLOPs and improving inference speed. As discussed in Remark (page 4), the additional parameters have a negligible impact on GPU memory and inference performance. Importantly, our focus is not on parameter efficiency within control blocks but on dynamic routing for adaptive efficiency. Our approach is also compatible with recent parameter-efficient control methods, such as ControlNext and OmniControl, as noted in our response to Reviewer eHec.

not compare with ControlNet++ and controlnext on flops and parameters

We would like to clarify that our research does not focus on designing more efficient control block structures but rather on investigating how to integrate control blocks effectively and efficiently into pre-trained diffusion models. In this sense, our work is orthogonal to ControlNeXt methods, which is why we initially excluded it from Table 1 and Table 2. However, our approach can potentially be adopted by ControlNeXt to further enhance its performance.

We appreciate the reviewer's suggestion for additional experiments. To evaluate our method in the context of ControlNeXt, we refer the reviewer to our response to Reviewer eHec. Briefly, since ControlNeXt applies control to all blocks by default, our method achieves comparable performance to ControlNeXt at $\gamma$ = 0.3 and outperforms it at $\gamma$ = 0.5.

Regarding ControlNet++, it primarily focuses on improving training strategies while maintaining the same inference cost as the standard ControlNet. In contrast, our work explicitly targets inference efficiency. Thus, a direct comparison is not applicable. Nevertheless, we acknowledge the relevance of these methods and will consider discussing them in future versions of our paper.

In Tab.5, what is the control signal? Why do not provide …

Table 5 presents an ablation study on different control strategies under the "Canny edge" control signals. We agree with the reviewer's suggestion and will reorganise the table in a camera-ready version to improve the clarity.

How to optimize the cost loss (Eq. 17).

The FLOPs loss is computed by referencing pre-computed FLOPs values from a lookup table, which are then combined with the diffusion loss using a scaling factor. This approach aligns with prior research[a,b,c] (as referred to in response to reviewer eKoB), on dynamic and efficient neural network architectures that optimize computational cost while maintaining performance. We will add those parts in the related works.

There are many previous works for dynamic routing strategy…

We appreciate the reviewer’s suggestion to discuss prior work on dynamic routing strategies. While our method shares some conceptual similarities with existing approaches, its goal and implementation are fundamentally different. Below, we clarify these distinctions concisely:

• [1] (Dynamic Routing Networks): Introduces a model with multiple branches, where a learned router selects the best path for each input to improve efficiency. Their method is trained from scratch for classification and focuses on reducing FLOPs. In contrast, our approach dynamically adjusts the influence of a fine-tuned ControlNet within a pre-trained diffusion model, aiming for controlled generation rather than computational savings.

• [2] (SkipNet): Uses a gating mechanism to decide whether to skip certain convolutional layers, reducing computation for easier inputs. Unlike SkipNet, which skips layers within a single network, our method balances contributions between a fixed diffusion backbone and a fine-tuned control module. We modulate control strength from ControlNet to a pre-trained diffusion model for better adaptation.

• [3] (DiT: Dynamic Token Routing): Dynamically routes image tokens within a Vision Transformer, deciding which tokens to process at each layer for efficiency. While DiT optimizes computation by selectively processing tokens, our method adjusts how much control blocks the ControlNet influences the final output, without altering token flow within the transformer.

• [4] (RAPHAEL): A large-scale diffusion model using a mixture-of-experts (MoE) to assign different paths for different styles or concepts. RAPHAEL is trained from scratch on massive datasets, while our approach efficiently adapts a pre-trained diffusion model using dynamic routing, making it suitable for low-data settings.

These prior works focus on optimizing efficiency or designing new architectures, while our method adapts an existing pre-trained model for more flexible and controlled generation. We appreciate the reviewer’s suggestion and will incorporate this discussion into the final version of our paper.

We thank the reviewer for the positive feedback and support of our work. We hope to have answered all of your questions satisfactorily below. Please let us know if you see any further issues in the paper that must be clarified or addressed.

The ablation study … with simpler alternatives, such as random selection of control blocks…

We sincerely thank the reviewer for their suggestion to conduct ablation studies on simpler alternatives to the random selection of control blocks. In response, we have considered the following alternative sampling strategies.

As suggested, we first evaluate the simplest strategy—uniform sampling—where 50% of the control blocks are randomly selected, denoted as Uniform.

The experimental results corresponding to these sampling strategies are presented below.

Notably, compared to all random sampling strategies, our approach with $\gamma$ = 0.3 achieved superior performance and higher inference speed. Moreover, the FID score improves from 17.21 to 14.80 when $\gamma$ is increased from 0.3 to 0.5. This ablation study further demonstrates the effectiveness of our method, and we will incorporate these findings into the camera-ready version of our paper.

At the optimal performance setting ($\lambda$=0.5)... regarding efficiency…

We would like to clarify that our method with $\gamma$ = 0.3 has already outperformed the ControlNet baseline in both SD1.5 and SD3.0 experiments, as demonstrated in Table 4 and Table 5 of the original paper, while also achieving significantly lower FLOPs, as shown in Table 3.

Our choice of $\gamma$ = 0.5 for comparison in Table 1 and Table 2 is not because it represents the optimal value, but rather because it provides a more direct comparison with the ControlNet baseline in terms of computational cost (280G FLOPs vs. 233G FLOPs). In contrast, when $\gamma$ = 0.3, the FLOPs are substantially lower at just 168G. We decide to add all $\gamma$ value experiment results in Table 1 and Table 2 to increasing the clarity in camera ready version.

code links

We provided anonymous links for comparison and reproduction:【https://github.com/Anonym916/Anonymity】

In Table 3. why does the speed decrease from 0.3 to 0.5

Since $\gamma$ represents the expected overall number of activated blocks, increasing $\gamma$ results in a higher number of active blocks, leading to increased computational cost (FLOPs) and consequently lower inference speed. This explains the observed decrease in speed from $\gamma$ = 0.3 to $\gamma$ = 0.5 in Table 3. We will refine our text in the camera-ready version to improve clarity.

How should ($\lambda_C$) be chosen, and does its value affect performance?

$\lambda_C$​ is a scaling factor that balances the diffusion objective and the FLOPs constraint objective. Since these two loss functions operate on different scales, $\lambda_C$​ is necessary to ensure proper weighting between them. We tune this value to achieve the target activation percentage while maintaining overall performance. Proper selection of $\lambda_C$ ensures that the model activates the desired number of blocks without significantly degrading generation quality. We proposed to use $\lambda_C$ = 0.5 in all our experiments as an experience value.