Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance

Thank you for your time and feedback on our work. We address your suggestions and concerns below. You can find the referred figures in the PDF attached with the “global” response. Tables can similarly be found in the main body of the “global” response.

Painting style generation results. Our proposed method works for any domain. Many realistic and non-painting samples generated are included in Figure 1, 4, 5 and Supplementary Figures 10, 11 in the paper. We will update the paper to make everything clear. Moreover, many samples in the attached rebuttal PDF also have realistic generation results—if the base model (e.g., SDXL) can generate realistic results for a prompt (which SDXL almost always can), then our method can, too.

Comparing with training-based baselines for individual structure or appearance control. Thanks for the suggestion! For appearance-only control, we compare Ctrl-X (by dropping structure control) against IP-Adapter in Figure 4, where Ctrl-X displays better appearance alignment than IP-Adapter for both the subjects and backgrounds. For structure-only control, we compare Ctrl-X against both training-based baselines (ControlNet and T2I-Adapter) and training-free baselines (FreeControl, Plug-and-Play, etc.) in our Appendix, specifically Figure 10 and Table 3 in the paper Appendix.

Additionally, we present additional experimental results with Uni-ControlNet [1], a ControlNet-based model trained to control structure and appearance at the same time. We report the user study and quantitative results in Tables 1 and 2. Ctrl-X is significantly ahead of Uni-ControlNet in terms of human preference for overall fidelity, indicating our method’s ability to balance structure and appearance control. Moreover, though Uni-ControlNet has better DINO self-sim, it struggles to balance structure preservation with appearance transfer with worse DINO-I scores, which is echoed by the user study.

Trade-off between training-time and quality. Thanks for the thought-provoking discussion starter. Ctrl-X provides a training-free and optimization-free method that achieves a delicate balance between structure control and appearance transfer. Though the argument that adapters have made it less necessary to train control modules from scratch is totally valid, the training cost (or backpropagation cost for guidance-based methods) cannot be ignored as we need to backpropagate through new models that will only get bigger. For example, Ctrl-Adapter [2] is trained on 200K images—paired data for any condition that Ctrl-X supports is difficult to gather at this scale. Also, FreeControl (being guidance-based) requires ~44GB of VRAM on SDXL to run, compared to Ctrl-X’s 11.5GB VRAM usage (Table 3).

The flexibility of training-free methods is also important, as Ctrl-X works with a large range of structure control signals (including higher-level conditions like bounding box and pose skeleton as shown in Figure 3), which training-based methods are limited by, as training requires paired data difficult to gather for in-the-wild conditions like 3D mesh, point cloud, etc. Thus, Ctrl-X’s training-free property has many upsides that may not necessarily make it a “trade-off.”

Of course, training-based methods still excel at the specific tasks they are trained for—a canny ControlNet, for example, is great at canny-image-conditioned generation. However, their limited flexibility makes the “trade-off” of Ctrl-X’s training-free nature a lot more appealing, as Ctrl-X works for a much wider range of applications, condition types or models. This may be closer to what we ultimately want for “controllable generation,” that is, humans can use any (visual) medium to influence generative models’ outputs.

[1] Shihao et al. “Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models.” NeurIPS 2023.

[2] Lin, et al. “Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model.” arXiv:2404.09967.

Thank you for your time and feedback on our work. We address your suggestions and concerns below. You can find the referred figures in the PDF attached with the “global” response. Tables can similarly be found in the main body of the “global” response.

Performance of Ctrl-X, qualitative evaluation. Thanks for your suggestion of conducting a user study to prove the superior performance of Ctrl-X. We report the user evaluation in Table 1. We randomly selected 15 sample pairs from our dataset and then assign each sample pair to 7 methods: Splicing ViT Feature, Uni-ControlNet, ControlNet + IP-Adapter, T2I-Adapter + IP-Adapter, Cross-Image Attention, FreeControl, and Ctrl-X. We invited 10 users to evaluate pairs of results, each consisting of our method, Ctrl-X, and a baseline method. For each comparison, users assessed 15 pairs between Ctrl-X and each baseline, based on four criteria: “the quality of displayed images,” “the fidelity to the structure reference,” “the fidelity to the appearance reference,” and “overall fidelity to both structure and appearance reference.” We collected 150 comparison results for between Ctrl-X and each individual baseline method. We reported the human preference rate, which indicates the percentage of times participants preferred our results over the baselines. The user study demonstrates that Ctrl-X outperforms training-free baselines and has a competitive performance compared to training-based baselines.

Moreover, we have included more qualitative Ctrl-X experiments on more challenging conditions (Figure 1), higher-level conditions (Figure 3), appearance- and structure-only control (Figures 4 and 5) in the PDF attached in the “global” response.

Performance of Ctrl-X, quantitative evaluation. We respectfully disagree that DINO self-sim score is “not widely used.” The DINO self-sim score has been employed by previous works (InstructPix2Pix [1], Pix2Pix-Zero [2], Plug-and-Play [3], FreeControl [4]) to evaluate the similarity between the global structures of two images.

Additionally, we do not think that “Ctrl-X has a bad performance.” Ctrl-X consistently achieves high fidelity to both structural and appearance references, and it performs better in terms of DINO self-sim and DINO-I scores compared to ControlNet + IP-Adapter and Cross-Image Attention. There is, in fact, a trade-off between structure consistency (DINO self-sim) and appearance similarity (DINO-I), as these are competing metrics—increasing structure preservation corresponds to worse appearance similarity, as shown in Figure 2 where we ablate Ctrl-X structure and appearance schedules. Single metrics are not representative of overall method performance, which is why we survey overall fidelity in our user study (Table 1), where Ctrl-X achieved the best overall fidelity. We will add examples to illustrate this trade-off in the camera-ready version.

For the evaluation of appearance similarity, we re-evaluated all generated samples using the DINO-I score, which has been employed by appearance customization works such as DreamBooth [5], P+ [6], and Break-A-Scene [7]. The results are reported in Table 2. The DINO-I score computes the cosine similarity between the DINO [CLS] embeddings of two images, while our original DINO-CLS directly computes the mean square error of these embeddings. This new evaluation metric further demonstrates our promising performance compared to both training and training-free baselines.

[1] Brooks et al. “InstructPix2Pix: Learning to Follow Image Editing Instructions.” CVPR 2023.

[2] Parmar et al. “Zero-shot Image-to-Image Translation.” SIGGRAPH 2023.

[3] Tumanyan et al. “Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation.” CVPR 2023.

[4] Sicheng et al. “FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Models with Any Condition.” CVPR 2024.

[5] Ruiz et al. “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.” CVPR 2023.

[6] Voynov et al. “P+: Extended Textual Conditioning in Text-to-Image Generation.” arXiv:2303.09522.

[7] Avrahami et al. “Break-A-Scene: Extracting Multiple Concepts from a Single Image.” SIGGRAPH Asia 2023.

Thank you very much for the in-depth reading of our paper and the helpful comments. Our responses are listed below. You can find the referred figures in the PDF attached with the “global” response. Tables can similarly be found in the main body of the “global” response.

Model extension to multiple object generation. Great suggestion! We conduct an additional experiment that involves multiple subjects in both the structure and appearance images, as seen in Figure 1. We tested Ctrl-X and ControlNet + IP-Adapter with two objects (house and tree) and three objects (dog, plant, and chair). Ctrl-X captures strong semantic correspondence between different objects and achieves balanced structure and appearance alignment. On the contrary, the training-based baseline often fails to maintain the structure and/or transfer the subjects’ appearance.

Adjust the influence of structure images. Thanks for the suggestion! We present a new experiment that ablates different combinations of appearance and structure control schedules in Figure 2. Doing so changes the influence of the structure and appearance images on the output, making cross-class structure-appearance pairs (e.g., horse normal map with puppy appearance) look more realistic.

Limited to lower-level control types. We present our experiments with new higher-level condition types: bounding boxes and human pose skeletons in Figure 3. Ctrl-X can handle these sparser and higher-level conditions by decreasing the structure schedule, making our method applicable to other higher-level control types, too.

Limited methodological novelty. Indeed, Ctrl-X is inspired by previous literature on training-free structure control and appearance customization. However, we suggest the reviewer consider how our work advances guidance-free controllable text-to-image generation.

Existing guidance-based methods have limited structure condition types and/or only handles either structure control or appearance control. Also, they require backpropagation through T2I/T2V which are only getting increasingly larger. In comparison, Ctrl-X enables fully disentangled control of structure and appearance while being fast and cheap in terms of inference time and GPU memory usage (Table 3). Moreover, whereas guidance-based methods are sensitive to guidance weights for each score function, Ctrl-X is more robust to its control schedules—in fact, the schedules can be varied to change the influence of the structure and appearance images (Figure 2).

Compared to the training-based [34], where the query and key for attention come from trained embeddings, Ctrl-X directly uses the query and key within Stable Diffusion’s self-attention layers by exploiting the spatial correspondence property of self-attention observed by us and also prior training-free works. Thus, our method re-frames appearance control as a local style transfer task achieved by spatially-aware normalization of intermediate features. These techniques allow Ctrl-X to achieve multi-subject generation (Figure 1) and higher-level conditions like bounding box and pose (Figure 3) in an all-in-one method which none of our baselines can achieve.

We believe our training-free and guidance-free method's disentanglement of structure control and appearance control, along with its flexibility and robustness, is novel, and we hope our method's utilization of attention layers is useful to the community—especially as visual generation moves towards transformer-based architectures.

Lack of flexibility with regards to using appearance OR structure control. Thanks for the observation. Ctrl-X is specifically designed to solve the combined controllable generation by disentangling control from given structure and appearance images. Our baselines show that achieving a good balance between structure alignment and appearance transfer is often difficult, training-free or not. Thus, Ctrl-X aims to provide a solution for that.

However, Ctrl-X can achieve appearance-only control by simply dropping structure control (and thus not needing to generate a structure image), as shown in Figure 4, displaying better appearance alignment for both the subject and background than the training-based IP-Adapter.

Indeed, for structure-only control Ctrl-X needs to generate an appearance image, but Table 3 shows that the additional inference latency cost is not high and the peak GPU memory usage is in fact lower than ControlNet + IP-Adapter and T2I-Adapter + IP-Adapter as there are no additional modules. Plus, with multi-subject generation, we believe control over the content of the image can be achieved by simply providing an appearance image.

Impact of generated appearance image quality on structure-only generation. We present structure-only generation samples alongside their jointly generated appearance images (equivalent to vanilla SDXL generation) in Figure 5. There is minimal quality difference between the generated appearance images and appearance-transferred output images, indicating that Ctrl-X’s appearance transfer does not greatly impact image quality. Thus, the output quality of Ctrl-X only grows as its base model’s quality improves.

Thank you for your positive feedback! We address your questions/concerns below. You can find the referred figures in the PDF attached with the “global” response. Tables can similarly be found in the main body of the “global” response.

Model extension to multiple-subject generation. Great suggestion! We conduct an additional experiment that involves multiple subjects in both the structure and appearance images, as seen in Figure 1. We tested Ctrl-X and ControlNet + IP-Adapter with two objects (house and tree) and three objects (dog, plant, and chair). Ctrl-X captures strong semantic correspondence between different objects and achieves balanced structure and appearance alignment. On the contrary, the training-based baseline often fails to maintain the structure and/or transfer the subjects’ appearance.

Report peak GPU memory and inference latency. Thanks for your suggestion. We report the inference latency and peak GPU memory usage in Table 3, re-tested on a single NVIDIA H100 GPU for a fair comparison. Ctrl-X (SDXL) is slightly slower than training-based baselines yet is significantly faster than training-free baselines and Splicing ViT Features. Moreover, Ctrl-X has lower peak GPU memory usage than SDXL training-based methods and significantly lower memory than SDXL training-free methods. (We note that Uni-ControlNet and Cross-Image attention uses the base model SD v1.5, which is ~4–5x faster and uses ~3x more memory compared to SDXL. Splicing ViT Features also trains its own much smaller custom model.)