CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation

Thank you very much for your constructive suggestions for the experiments.

 

Q1: Consider specifying 1-2 new image conditions and key metrics (e.g., adaptation speed, data efficiency, performance) for comparing UniControl [1] fine-tuning to CtrLoRA. This would provide a clear, focused comparison.

Although UniControl [1] trains a unified model on multiple conditions, its delicate design makes it not straightforward to be quickly extended to new conditions. On the contrary, our fine-tuning stage keeps consistent with the pre-training strategy of our Base ControlNet, and therefore the adaptation to new conditions is natural and efficient. As can be seen from the above comparison, our CtrLoRA significantly outperforms UniControl, demonstrating the superior adaptability of our method to new conditions. It can also be observed that our CtrLoRA trained on 1000 images even surpass UniControl trained on 100,000 images, demonstrating the data efficiency of our method.

Q5: I have noted that CtrloRA can perform low-level image enhancement tasks, such as low-light image enhancement. Could the authors demonstrate how CtrloRA performs in comparison to other diffusion models for low-light image enhancement? This could highlight potential advantages or unique features of CtrloRA in practical applications.

Since we have no special design for these low-level tasks, it can be almost sure that we cannot perform better than methods specialized in these tasks. In this period, we train and compare a state-of-the-art method RetinexFormer [2] for low-light image enhancement. As can be seen, our CtrLoRA lags behind the state-of-the-art performance.

However, note that the main purpose of training CtrLoRA for low-level image enhancement tasks is to prove the generalizability of our method to various novel conditions. Although far from state-of-the-art, the results are visually satisfactory and significantly better than related competitors (see the dehazing performance in Table 3 of the paper).

---

 

[1] Qin, Can, et al. "UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild." Advances in Neural Information Processing Systems 36 (2024).

[2] Cai, Yuanhao, et al. "Retinexformer: One-stage retinex-based transformer for low-light image enhancement." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Q2: Additional baselines are required for each base image condition. Comparisons should be made with a fully trained ControlNet, which has been trained exclusively under a single image condition, to establish a more comprehensive benchmark.

Q3: Similarly, for the new condition, it is essential to compare the performance of CtrLora against ControlNet when ControlNet has been fully trained on a single modality. This will provide a clearer understanding of their relative efficiencies.

Thanks for this valuable suggestion. Below, we compare our method with multiple community models, including fully trained ControlNet from the community. It can be seen that our CtrLoRA outperforms fully trained ControlNet for both base and new conditions. (The ControlNet for Densepose, Inpainting and Dehazing are trained by ourselves with 100k images.)

 

Base conditions:

 

New conditions:

---

 

Q4: It would be beneficial to explore how the number of image conditions used during the training of the base ControlNet affects its ability to learn new conditions. Insights into the scalability and adaptability of the base network could prove crucial for future applications.

We are currently running experiments on the number of base conditions and will present the results as soon as possible.

We highly appreciate your constructive feedback.

We have carefully responded to each of your questions and revised our paper accordingly. The revision details are listed in "Paper Updates" at the top of this page. Besides, we've clarified our motivation and contribution in the "General Response" at the top of this page.

We look forward to your reply and welcome any discussion on unclear points regarding our paper and our response.

We highly appreciate your constructive comments.

 

W1: The paper primarily aims to improve the training efficiency of all kinds of conditional models, hence it employs a series of LoRAs to train the newly introduced conditions based on the "Base ControlNet". However, there is relatively little comparison and discussion of existing methods that efficiently train ControlNet, such as T2I-Adapter, ControlLoRA, and SCEdit.

Existing efficient methods such as T2I-Adapter [1] and SCEdit [3] mainly focus on decreasing the model sizes, but the data and GPU resources needed to train these models are still beyond the reach of ordinary users. For example, T2I-Adapter is trained on 164k~600k images with 4 V100 GPUs for around 3 days, and SCEdit is trained on 600k images with 16 A100 GPUs. On the contrary, our method can achieve satisfactory performance by training on ~1000 images with a single RTX 4090 GPU within 1 hour, while keeping the model sizes comparable to or even smaller than T2I-Adapter and SCEdit, thereby greatly lowering the cost for ordinary users to create their customized ControlNets.

As for ControlLoRA [2], which also employs LoRA for conditional generation, it suffers from poor performance with a small amount of training data. Below we show the results of training ControlLoRA on 1000 images. It is clear that our method significantly outperforms ControlLoRA. We will further discuss the core difference between our method and ControlLoRA in the response to W2.2 below.

We will add these discussions in the revised paper.

---

 

W2.1: There currently exists a viable controlnet-union model, which can handle different conditions using a single model. This may be a higher-level representation of the training of the "Base ControlNet" model discussed in the paper.

Thanks for pointing out controlnet-union [4]. We found that it was released on July 2, thus it should be considered as a concurrent work. This method is similar to UniControl [5] and Uni-ControlNet [6] which manage multiple conditions within a unified model. However, while these methods only emphasize union, our work also emphasizes adaptation to new conditions at a substantially low cost. Thus we are solving distinct problems with different techniques.

---

 

W2.2: the use of LoRA for fine-tuning is relatively straightforward and has been implemented in previous community works, such as ControlLoRA. In comparison, the overall innovativeness of the paper is limited.

Using LoRA for new conditions "seems to be straightforward", which has been already tried by the community model ControlLoRA [2] as you mentioned. However, LoRA itself is not the challenge; the real challenge is how to make a new LoRA perform well with limited data (1000 images in our paper), as it is difficult for an ordinary user to collect a large customized dataset. This challenge is not straightforward to solve and existing methods cannot handle it well including ControlLoRA. To this end, we propose to train a Base ControlNet with shifting strategy to capture the general knowledge of I2I generation. With our well trained Base ControlNet, 1000 data samples is sufficient to learn a LoRA for a new condition with satisfactory results.

Compared to ControlLoRA, which directly uses LoRA to fine-tune Stable Diffusion, our method emphasizes the importance of general I2I knowledge (the Base ControlNet). The results in the response to W1 demonstrate that our method is significantly superior to ControlLoRA. We believe the emphasis on the necessity of a Base ControlNet is one of our main contributions and significantly distinguishes our method from ControlLoRA.

---

 

W3: The paper does not discuss how many conditions to use or how to select conditions for training the "Base ControlNet" to achieve optimal knowledge transfer effects.

We are currently running experiments on the number of base conditions and will present the results as soon as possible.

Q1: Regarding the discussion of "Adaptation to new conditions," while training a comparison method from scratch with a small amount of data may indeed result in slow convergence, what would be the results if we used a pre-trained conditional model (analogous to possessing a Base ControlNet) for fine-tuning?

A straightforward manner is to fine-tune a pretrained ControlNet or UniControl [5]. However, both are less effective than our method. As discussed at line 173 of our paper, a pre-trained ControlNet is extensively trained to fit a particular condition, and therefore not general enough to efficiently adapt to different conditions. For UniControl, as discussed at line 161 of our paper, although it is trained on multiple conditions, its delicate design makes it not straightforward to be quickly extended to new conditions. On the contrary, our fine-tuning stage keeps consistent with the pre-training strategy of our Base ControlNet, and therefore the adaptation to new conditions is natural and efficient.

Below we add the comparison to directly fine-tune a pre-trained ControlNet and UniControl on 1000 images. As can be seen, our CtrLoRA significantly outperforms these methods when adapting to new conditions, demonstrating the effectiveness of our Base ControlNet and the potential of our idea to learn the general knowledge of I2I generation.

Q2: I'm curious about the performance between a pre-trained controlnet model available in the community and a model trained using proposed "Base + LoRA" with same conditions.

Thanks for this valuable suggestion. Below, we compare our method with multiple community models. As for base conditions, our CtrLoRA achieves comparable performance to the state-of-the-art method UniControl [5] and outperforms the rest of the competitors. As for novel conditions, our CtrLoRA performs better than the competitors in most cases. (The ControlNet for Densepose, Inpainting and Dehazing are trained by ourselves with 100k images.)

 

Base conditions:

 

New conditions:

---

 

[1] Mou, Chong, et al. "T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.

[2] Wu, Hecong. "ControlLoRA: A Lightweight Neural Network To Control Stable Diffusion Spatial Information." GitHub.

[3] Jiang, Zeyinzi, et al. "Scedit: Efficient and controllable image diffusion generation via skip connection editing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] xinsir, et al. "controlnet-union-sdxl-1.0." Hugging Face.

[5] Qin, Can, et al. "UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild." Advances in Neural Information Processing Systems 36 (2024).

We highly appreciate your constructive feedback.

We have carefully responded to each of your questions and revised our paper accordingly. The revision details are listed in "Paper Updates" at the top of this page. Besides, we've clarified our motivation and contribution in the "General Response" at the top of this page.

We look forward to your reply and welcome any discussion on unclear points regarding our paper and our response.

Q1: The results in Figure11b demonstrate that the different conditions are effectively disentangled, with a direct summation module according to Figure 3c. Could you clarify why this module is effective, such as presenting the results of two elements both separately and after sum-up.

As explained in the LoRA paper [1], the LoRA layers work by amplifying existing knowledge in the base network. In our scenario, we have a speculation that the LoRAs trained on different conditions amplify different knowledge in the Base ControlNet by orthogonal semantic directions, therefore direct summation is natural for the model to handle both conditions simultaneously. It's a good suggestion to present the results of two elements both separately and after sum-up. Since we cannot include figures in the comment, we will present these results in the revised paper.

---

 

Q2: A detail, why not presenting all 9 base-condition results comparison to UniControl in Table 2?

We didn't include all results only for typesetting consideration. The result for "Bbox" condition is presented below, which does not change our conclusion in line 336 of our paper: "for base conditions, our base ControlNet performs on par with the state-of-the-art UniControl, demonstrating its robust fundamental capabilities". We will include it in the revised paper.

---

 

[1] Hu, Edward J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." International Conference on Learning Representations.

We sincerely thank Reviewer GM4E for carefully reading our paper and giving a valuable review.

 

W1: In line 70, the authors state that ControlNet with Canny edges requires 3 million images over 600 GPU hours for one condition. In contrast, line 244 indicates that Base ControlNet necessitates millions of images for 6000 GPU hours for 9 conditions. Although it is not fair enough, but it implies that the proposed method does not significantly reduce the computational burden.

We would like to clarify that our work aims at reducing the computational burden for any new conditions, but not for the base conditions (Base ControlNet). Given our pre-trained Base ControlNet, the community users just need to collect about 1000 data pairs of a customized condition, and then fine-tune the Base ControlNet with less than one hour on a single GPU. That is to say, we make efforts to train and release a Base ControlNet while letting the ordinary user create their customized ControlNet at a significantly low cost.

---

 

W2: In line 239, the mechanism of training with 9 conditions is not clear enough. As different conditions have different levels of sparse information of input images, why they have equal training iterations? And continuous shifting between different conditions may make the training hard.

We agree that setting different weights (iterations) to different conditions may further enhance the model performance. However, ablating the weight choices is tough since training the Base ControlNet for once consumes a large amount of resources, while our devices cannot support such an amount of ablation. Therefore, without any prior knowledge of information density of different conditions, we just choose an equal weight for them and find it works well enough. Nonetheless, this is a very interesting perspective, and we will keep investigating this problem in future works.

---

 

W3: the motivation why the new conditions are not trained as the Base ControlNet by a shifting mechanism is not clear enough.

When learning new conditions, we freeze the parameters of the Base ControlNet and only train the newly added LoRA layers. In this case, the learning of each condition is independent and only affects its own corresponding LoRA. Thus, there is no need to train the new conditions by the shifting mechanism.

---

 

W4: Most results are from "Base CN + CtrlLoRA'', and results from "Community Model + CtrlLoRA" in Figure 11a are rare, not enough to convince that CtrlLoRA is effective when transferring to other community models.

In Figure 16 in the appendix, our CtrLoRA is applied to 7 community models of various styles for style transfer, demonstrating its adaptability to other community models.

---

 

W5: pretrained-VAE seems to be only an interesting trick.

Using pretrained VAE is vital to alleviate the sudden convergence phenomenon and achieve fast convergence. We have made extensive analyses on the use of pretrained VAE as the condition embedding network, in Section 3.4, lines 412-415 (Table 4, Figure 7), and Appendix Section A (Figure 12). For example, as shown in Figure 7, without pretrained VAE, a ControlNet needs more than 40,000 steps to converge, while applying pretrained VAE shortens the convergence to 4,000 steps. This choice is built upon our deep understanding of the convergence problem of ControlNet, and we believe our analysis is useful and can benefit the ControlNet community.

---

 

W6: putting all the prompts in the appendix makes reading inconvenient.

Since our method mainly focuses on image-to-image generation, the text prompts are of less importance. Therefore, we put the prompts in the appendix in order to better organize the figures and keep the presentation cleaner.

We have uploaded a revised paper with details in "Paper Updates" at the top of this page, including the response for Q1 in Appendix Figure 17.

Q1: The results in Figure11b demonstrate that the different conditions are effectively disentangled, with a direct summation module according to Figure 3c. Could you clarify why this module is effective, such as presenting the results of two elements both separately and after sum-up.

We sincerely thank Reviewer UthF for the precise comments and positive feedback.

 

W1: The authors train a base ControlNet for the subsequent LoRA fine-tuning. However, why not directly fine-tune a pre-trained ControlNet or Uni-ControlNet?

While directly fine-tuning a pre-trained ControlNet or Uni-ControlNet is feasible, they do not perform as well as fine-tuning our Base ControlNet. As discussed in line 173 of our paper, a pre-trained ControlNet is extensively trained to fit a particular condition, and therefore not general enough to efficiently adapt to different conditions. For Uni-ControlNet [2] and UniControl [3], as discussed in line 161 of our paper, although they are trained on multiple conditions, their delicate design makes them not straightforward to be quickly extended to new conditions. On the contrary, our fine-tuning stage keeps consistent with the pre-training strategy of our Base ControlNet, and therefore the adaptation to new conditions is natural and efficient.

Below we add the comparison to directly fine-tune a pre-trained ControlNet and UniControl on 1000 images. As can be seen, our CtrLoRA significantly outperforms these methods when adapting to new conditions, demonstrating the effectiveness of our Base ControlNet and the potential of our idea to learn the general knowledge of I2I generation.

---

 

W2: Lack of comparison to: ControlNet++ [1].

We agree that including a comparison to ControlNet++ will make the analysis more comprehensive, and the corresponding results are presented below. It is reasonable that our method lags behind ControlNet++ because the latter is explicitly optimized by the metric functions. But note that this technique is orthogonal to our method and less relevant to our focus.

---

 

W3: The paper does not explore whether this method can be generalized to other diffusion models such as SDXL and Pixart.

We agree that it is important and useful to apply our method to more powerful backbones such as SDXL and Pixart. However, the development and extensive analysis of our Base ControlNet on SD 1.5 have exhausted our available devices (only 8~12 RTX 4090 GPUs with 24GB VRAM); we lack sufficient resources for larger backbones.

Nonetheless, the whole design philosophy of our CtrLoRA, especially the training strategy, is not restricted to the current SD 1.5 backbone. Therefore, we believe our method and its advantages have the potential to be generalized to larger backbones (just like ControlNet, originally built upon SD 1.5, it is well generalized to various backbones). Of course, we would like to develop our method upon more powerful backbones for future works when we have more devices.

---

 

[1] Li, Ming, et al. "ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback." European Conference on Computer Vision. Springer, Cham, 2025.

[2] Zhao, Shihao, et al. "Uni-controlnet: All-in-one control to text-to-image diffusion models." Advances in Neural Information Processing Systems 36 (2024).

[3] Qin, Can, et al. "UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild." Advances in Neural Information Processing Systems 36 (2024).