Learning Vision and Language Concepts for Controllable Image Generation

Thank you for the valuable feedback. Please see our responses below and our uploaded results at https://anonymous.4open.science/r/ICML2025-F636/rebuttal.pdf.

1. Human evaluations.

Thank you for the nice suggestion. We have added the human evaluation results to the uploaded Figure 7. The human scores favor our method across all benchmarks.

2. Typos in Equation (1).

We have corrected the typo – thank you!

3. Capturing statistical dependence and spurious correlation.

Thank you for this thoughtful question. The statistical dependence among textual concepts $z ^{T}$ actually strengthens our framework rather than contradicts our causal goals. To clarify:

Regardless of these dependencies, Theorem 4.4 guarantees we can disentangle individual textual concepts. If $z ^{T} _1$ represents "beach" and $z ^{T} _{2}$ represents "palm tree," we can identify them and intervene on "beach" without automatically changing "palm tree".

At the same time, concepts may naturally exhibit statistical dependencies (like "beach" and "palm tree"). Our framework accommodates these dependencies, making it more general than models that impose independence constraints.

4. Sufficient variability in real-world datasets.

Great question! We have included experiments on training our model on short, coarse captions in uploaded Table 4. The negligible performance variation indicates our method’s robustness to the caption quality.

Further, we have included the following discussion in our revision. ``In the paper, we give precise sufficient conditions – given adequately diverse data, we can achieve desirable component-wise identifiability. In general, we would expect this condition to be satisfied – standard text-image datasets (e.g., LAION) contain millions of captions, far exceeding the number of possible visual concepts. Additionally, we may follow existing methods (e.g., [1]) to employ vision-language models to generate higher-quality captions.’’

5. Implementation details.

Thank you for the thoughtful feedback. We have included implementation details in our revision.

Training data: “We follow baseline SANA’s protocol to first generate 2 million images with Flux. 1 Schnell, and then apply QWEN2.0-VL for re-captioning. In total, we use 2M text-to-image data for finetuning.”

Evaluation: “We use PIE-BENCH (Ju et al. 2024) for evaluation, which contains 700 editing instructions and corresponding input and output captions. We utilize the input and output captions to generate paired images and measure the similarity of each pair.”

Original prompts: please see examples in uploaded Figure 2.

6. Comparative experiments & Is “controllable” consistent with prior work?

Prior work on “controllable T2I” often employs side information (e.g., edges) to control generation [2], whereas we use only text prompts to control atomic concepts. Thus, we didn’t compare with them. Thanks to your feedback, we have made this distinction explicit in our revision. We have included comparative experiments with recent image-editing methods in the uploaded Table 3. Our approach outperforms baseline approaches, including those trained on expensive paired editing data.

7. Images with multiple concepts modified & scalability of the learned concepts.

Thank you for the insightful question. We’ve included results on simultaneously modifying multiple concepts in the uploaded Figure 8. Our method manages to simultaneously edit up to four concepts, while the baseline quickly loses the subject identity.

8. Training and inference cost analysis.

We finetune our model (SANA backbone with LoRA) for 10000 steps, which takes around 12 hours on 8H100 GPUs. We provide the inference speed comparison in the uploaded Table 7. Thanks to our compact representation (64 vs. 300 tokens in SANA), our method is the fastest inference method (0.48s/image).

9. “What if the input images come directly from real-world sources?”

Given real-world images, we apply diffusion inversion to obtain the initial noise value, as in standard inversion-based models. We then feed the noise and the target prompt into our model for editing. The uploaded Table 3 and Figure 3 demonstrate that on real-world sources, our method is superior to or comparable with the baselines across all metrics.

10. Online updates for new concepts.

Great question – in that case, our model is flexible to incorporate a continuous learning strategy (e.g., dynamic memory expansion [3]) to absorb new concepts while retaining learned ones – thank you for suggesting this interesting direction.

Please let us know whether your concerns are addressed. Thank you in advance!

References

[1] ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. Chen et al. ECCV 2024.
[2] Adding Conditional Control to Text-to-Image Diffusion Models. Zhang et al. ICCV 2023.
[3] Online Task-Free Continual Generative and Discriminative Learning via Dynamic Cluster Memory. Ye et al. CVPR 2024.

We are grateful for your thorough assessment. Please find the responses below and our uploaded results at https://anonymous.4open.science/r/ICML2025-F636/rebuttal.pdf.

1. Detailed ablation analyses.

Thank you for your constructive feedback. We have added evaluations of the loss terms in the uploaded Table 4 and Figure 1. The sparsity regularization improves our model across almost all metrics. Without KL regularization, the exogenous variable $\epsilon$ contains excessive information about the input image, and the model ignores the text information as shown in Figure 1. The diffusion loss is the primary objective for the diffusion model, without which the model experiences negligible updates (Figure 1).

2. DINO similarities between original and edited images.

Great suggestion! We have included the DINO similarity in Table1, 2,3,4,6, and our revision. Our method consistently obtains the highest DINO similarity score across benchmarks.

3. Additional justifications for conditions.

Thank you for the thoughtful feedback. We have included the following discussion in our revision.

Condition 4.2-i: ``The invertibility ensures the observed variables preserve all latent variables’ information. Otherwise, it would be impossible to recover these latent variables from observed variables. Practically, the high dimensionality of images often offers sufficient capacity to hold all information of $z ^{I}$, and human language is often a verbose articulation for concise, abstract textual concepts $z ^{T}$, which makes this condition feasible. The smoothness allows us to use partial derivatives to establish identifiability, following prior work (Khemakhem et al., 2020a;b).

Condition 4.3-4: “Condition 4.3-4 calls for sparse connections from the textual to the visual concepts. Consider concepts like "fur" and "ears" when describing a cat. These concepts should affect partially distinct visual features. If every visual feature triggered by "ears" was also triggered by "fur," these concepts aren't genuinely atomic and should be restructured. Theoretically, Condition 4.3-4 has been adopted in recent work (Kivva et al. 2021) and offers greater flexibility compared to alternatives in prior literature. For instance, prior work [1,2] assumes each $z ^{T} _{i}$ has at least one unique child $z ^{I} _{j}$, which is strictly stronger.

We acknowledge that we only provide sufficient conditions that show the possibility of learning concepts. We completely agree that some conditions can be weakened. Nevertheless, developing necessary conditions for general problems is much more challenging, and we hope our work provides a foundation that can be iteratively refined by the research community.

4. Image-editing baselines.

Thank you for the helpful feedback. We have included image-editing baselines and datasets in the uploaded Table 3 and Figure 3. Our approach outperforms baseline approaches, including those trained on expensive paired editing data.

5. Computation efficiency analysis.

We finetune our model (SANA backbone with LoRA) for 10000 steps, which takes around 12 hours on 8H100 GPUs. We provide the inference speed comparison in the uploaded Table 7. Thanks to our compact textual representation (64 vs. 300 tokens in SANA), our method is the fastest inference method (0.48s/image).

6. Key related works.

Thank you for these valuable references. We have added the following discussion.

“Xu et al. focus on designing attention maps where they replace the target attention map with the source map for a particular word. In contrast, we concentrate on developing superior conditioning representation. These two approaches are complementary and we leave investigating this synergy as future work.’’

``Rajendran et al. formulate concepts as affine subspaces of latent variables and provide identifiability guarantees for these subspaces. In contrast, we directly identify each latent variable, which enables us to directly control atomic aspects.’’

7. Figure optimization.

Thank you for the helpful feedback. We have re-designed the framework diagram (Figure 2) and annotated Figure 4 and Figure 5 – see uploaded Figure 4, 5, 6.

8. Additional downstream task comparisons and parameter analyses.

We have included additional ablation results as indicated in responses 1, 2, and 4. These results further highlight the advantages of our framework and validate our theoretical insights – thank you for the constructive suggestions!

9. Dataset Selection rationale.

We select PIE-BENCH (Ju et al. 2024) because it covers ten editing types and evenly distributed styles over four categories. This broad coverage can provide a thorough evaluation.

Please let us know if you have any questions. We would be happy to discuss further!

References

[1] A Practical Algorithm for Topic Modeling with Provable Guarantees. Arora et al. ICML 2013.
[2] Identifiable Variational Autoencoders via Sparse Decoding. Moran et al. TMLR.

Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below and the uploaded results at https://anonymous.4open.science/r/ICML2025-F636/rebuttal.pdf.

1. Experiments on various domains.

Thank you for your constructive comment. In light of your comment, we have included experiments on the EMU-edit testset [1] in our manuscript and uploaded Table 2. The EMU-edit testset contains 3589 paired prompts and covers 7 common editing types, including background alteration (background), comprehensive image changes (global), style alteration (style), object removal (remove), object addition (add), localized modifications (local), and color/texture alterations (texture). We also provide generated samples in the uploaded Figure 2. In addition to the controllable text-to-image generation task, we also evaluate our method on real-world image editing tasks against image editing baselines in uploaded Table 3 and Figure 3. Across all metrics, our method is superior or comparable to the baselines, showcasing the effectiveness of our framework.

2. “Real-world text-image datasets might violate the `sufficient variability’ condition. Add experiments to show how robust the method is.”

Thanks to your suggestion, we have included in our manuscript and the uploaded Table 4 experiments on short, coarse captions to demonstrate the robustness of our approach. In our main experiments in the submission, we follow our baseline SANA’s [2] protocol to generate the training data: we first randomly sample 2 million short prompts from DiffusionDB [3] to generate 2 million images with Flux. 1 Schnell, and then apply QWEN2.0-VL to generate detailed captions. Here, to assess our model’s robustness to the text quality, we replace the detailed captions with original short prompts for model training. We can observe that across all metrics, the text-quality degradation makes negligible impacts on our method, demonstrating its robustness to text quality changes.

To further address your concern, we have included the following discussion in our revision. `` In the paper, we give precise sufficient conditions – given adequately diverse data, we can achieve desirable component-wise identifiability. In general, we would expect this condition to be satisfied – standard text-image datasets (e.g., LAION) contain millions of captions, far exceeding the number of possible visual concepts. Additionally, we may follow existing methods (e.g., [3,4,5]) to employ vision-language models to generate higher-quality captions. Even if this condition is not met, oftentimes other natural properties can be leveraged to greatly weaken this condition. For instance, if the generating function $g ^{I}$ is simple or sparse, less variability would be needed to guarantee the identifiability (e.g., [6]). Such sparsity is often encouraged implicitly or explicitly in generative models (e.g., sparse attention patterns). A thorough theoretical investigation into combining these properties is an interesting problem, which we leave as future work.''

Please let us know if there are any further concerns, and we are more than happy to address them in the following stage.

References

[1] Emu Edit: Precise Image Editing via Recognition and Generation Tasks. Sheynin et al. CVPR 2024.
[2] SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers. Xie et al. ICLR 2025.
[3] DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-image Generative Models." Wang et al. arXiv.
[4] ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. Chen et al. ECCV 2024.
[5] Improving Image Generation with Better Captions. Betker et al. Technical report.
[6] Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning. Li et al. ICLR 2025.

We appreciate your valuable time and efforts dedicated to reviewing our work. Please find our responses below and the uploaded results at https://anonymous.4open.science/r/ICML2025-F636/rebuttal.pdf.

1. “It is recommended to incorporate additional metrics that specifically assess disentanglement to provide a more thorough evaluation.”

Thank you for this valuable suggestion. Unfortunately, most existing disentanglement metrics require access to ground truth latent variables (e.g., Mutual Information Gap [1]), which makes them not applicable to the text-to-image task where ground-truth latent variables are unavailable. Based on the definition of disentanglement, we design the following evaluation protocol to assess whether interventions on one latent factor would lead to unintended changes in other factors: We prompt ChatGPT to randomly name 10 animals, 10 actions, and 10 backgrounds. For each animal, we fix the animal's identity and edit its action and background, resulting in 200 original & edited pairs. We repeat over 10 random seeds and end up with 2000 pairs in total. For evaluation, we employ QWEN2.5-VL-Instruct7B to examine whether the editing retains the animal identity (subject consistency) and whether the targeted modification is achieved (prompt consistency). As you can see from uploaded Table 5 and Figure 9, our method achieves the highest subject consistency and prompt consistency scores simultaneously against SANA and finetuned SANA, demonstrating its superior disentanglement capability.

2. Details on the training dataset, dataset sizes, domains, and whether SANA is tuned with other components.

Thank you for pointing this out. As you suggest, we’ve included a dedicated section in the appendix to cover these details. “SANA employs around 30 million text-to-image paired data to train the model. Unfortunately, the data is not publicly available. Therefore, we follow SANA’s protocol to first generate 2 million images with Flux. 1 Schnell, and then apply QWEN2.0-VL for re-captioning. In total, we use 2M text-to-image data for finetuning.”

In our model, the SANA backbone is fine-tuned via LoRAs. Thanks to your suggestion, we have included results on finetuning SANA via LoRAs on our data. As shown in Table 4, our model maintains the leading margin against SANA-Finetune, showcasing our approach’s effectiveness.

3. ``The token number for $z ^{T}$ is set to 64. Is this sufficient to faithfully represent text information, especially for long text?’’

Thank you for the insightful question. In light of your question, we have added comparative experiments on long captions. Specifically, we expand the short prompts in PIE-BENCH [2] to longer captions with QWEN2.5-Instruct-32B. We can see from Table 6 that our method outperforms SANA and SANA-Finetune (which uses a token number of 300) while enjoying faster training and inference, thanks to our compact representation (a token number of 64).

4. ``Details on the evaluation dataset.’’

Thank you for raising this point. We have included the following details in the appendix.

“We use the benchmark PIE-BENCH [2], which contains 700 editing instructions and corresponding input and output captions. We utilize the input and output captions to generate paired images and measure the similarity of each pair.”

5. ``More results.’’

Thanks to your feedback, we have included two new sets of evaluation results: 1) In Table 2 and Figure 2, we present additional experiments on the EMU-edit test set [3], which consists of 3,589 paired prompts and encompasses seven common editing types: background alteration (background), comprehensive image changes (global), style modification (style), object removal (remove), object addition (add), localized modifications (local), and color/texture changes (texture). We can observe that our approach either outperforms or is comparable with baseline methods across all metrics, demonstrating its effectiveness. 2) We also added real-image editing results in Table 3 and Figure 3. Our method shows superior performance against image editing baselines.

6. Typos.

We have corrected all these typos in our manuscript – thank you so much for the helpful feedback!

We were wondering if we have addressed all your concerns. Please let us know if there is anything we could further discuss – your further feedback would be greatly appreciated.

References

[1] Isolating sources of disentanglement in variational autoencoders. Chen et al. NeurIPS 2018.
[2] PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code. Ju et al. ICLR 2024.
[3] Emu Edit: Precise Image Editing via Recognition and Generation Tasks. Sheynin et al. CVPR 2024.