Scaling Concept With Text-Guided Diffusion Models

Clarification on Motivation

We appreciate the reviewer’s comments and would like to clarify that our motivation centers on scaling inherent concepts within real input effectively. Existing editing methods, such as InstructPix2Pix and AUDIT [a], focus on providing editing effects through explicit instructions or attention control (e.g., Prompt-to-Prompt).

In contrast, our work emphasizes continuous scaling of inherent concepts directly from the input (as shown in Fig. 1). The key challenge lies in obtaining scalable concept representations, inspired by our observations in Sec. 3.2. We have included additional discussion on related works [1,2] as suggested by the reviewer and updated the paper accordingly (L181–184).

[a] AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models

---

Experiment on Cross-Attention Mechanisms

To further investigate our hypothesis, we follow the approach outlined in [a] to visualize the cross-attention map of an example image (Fig. 2). We then assess the zero-shot probing accuracy using the CLIP model. The results are shown below:

In the removal branch, the cross-attention map is concentrated on the sky region rather than the church. This focused attention contributes to the successful removal of the church concept and the subsequent inpainting with sky elements.

[a] Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing

---

Connection Between Observations and Proposed Method

As evidenced by common inversion techniques [a,b,c], inversion can reconstruct the concept. In our empirical study, we investigate the ability to remove a concept. Since concept removal and reconstruction begin from the same starting point (see Fig. 2), we define the divergence between these two processes as the trend of concept decomposition.

While this does not directly equal concept decomposition, it is pivotal for our method. We model the difference between the removal and reconstruction branches to extract concept-related representations and enable scaling.

[a] Null-text Inversion for Editing Real Images using Guided Diffusion Models

[b] ReNoise: Real Image Inversion Through Iterative Noising

[c] Direct Inversion: Boosting Diffusion-Based Editing with 3 Lines of Code

---

Is the Method Limited to Concept Scaling?

We clarify that our method is tailored for concept scaling but not limited to this application. The design of "Step 2: Concept Scaling" (L311–367) focuses on enhancing or suppressing concepts. Disabling this step converts our method into a variant supporting concept addition or modification, similar to ReNoise. Moreover, we want to highlight that beyond concept scaling, our method addresses tasks such as Object Stitching and Audio Removal. These applications are beyond the scope of current image enhancement methods.

---

Experiment on Concept Suppression

In the revised version, we include additional experiments on concept suppression using the TEdBench dataset (see Table 1, L412–419). The dataset includes diverse images spanning multiple categories and complex concepts. We compare our method’s performance with existing techniques (L442–462). Our method demonstrates generalization to diverse datasets and surpasses existing methods on TEdBench, showcasing its superior performance.

---

Prompt [class] in Empirical Study

For the empirical study, the prompt [class] is diverse and depends on the dataset:

1. We use 95 samples from 10 common classes in the COCO dataset. The [class] prompt corresponds to the class label for each image.
2. We use the AVE dataset which contains 28 sound classes. 5 audio clips are sampled per class, with the [class] prompt being the class label.

Additionally, Fig. 1 demonstrates that the choice of concept for a single input image can be arbitrary, supporting the versatility of our approach.

---

Connections to Image Enhancement

Existing image enhancement methods [1,2] leverage generative priors to address common image corruptions (e.g., low resolution, blur, poor lighting). In contrast, our method focuses on scaling-based editing.

• Connections: Our method can treat certain corruptions as concepts, enabling tasks such as Deraining/Dehazing, Creative Enhancement (Fig. 11), Anime Sketch Enhancement (Fig. 12, 13)
• Differences: Unlike image enhancement methods, our approach extends to tasks such as Canonical Pose Generation (Fig. 9), Object Stitching (Fig. 10), Face Attribute Scaling (Fig. 14), and Audio Removal and Highlighting (Figs. 1, 7, 8)

We will include this discussion in the related work section to clarify the differences and connections between our method and image enhancement techniques.

Comparison to Existing Methods or Techniques

In the experimental section, we provide both quantitative and qualitative comparisons with existing methods, including InstructPix2Pix and LEDITS++:

1. Quantitative Comparison: Table 1 showcases comparisons on Concept Scaling Up using our proposed WeakConcept-10 dataset and Concept Scaling Down using the public TEdBench dataset. Our method surpasses both InstructPix2Pix and LEDITS++, demonstrating its effectiveness quantitatively.

2. Qualitative Comparison: Figures 5 and 6 include visualizations comparing scaling results with other methods. Our method achieves better scaling results, as evidenced by higher visual fidelity and better alignment with the scaling target.

In all, Section 4.2 provides an in-depth discussion of these comparisons.

---

Complexity of the ScalingConcept Method

Our ScalingConcept method is simple in implementation and highly adaptable to most text-guided diffusion models:

1. Inversion Process: Most inversion techniques can be used, including the standard DDIM or the more advanced ReNoise, which is versatile across almost all diffusion models.

2. Sampling Process: Our method only interacts with the input ($x_t$) and output noise predictions ($\epsilon$) of the diffusion model. It does not modify the internal network architecture, ensuring compatibility with a variety of text-guided diffusion models across different domains (e.g., audio).

---

Scalability and Effectiveness Across Diverse Datasets

We thank the reviewer for their insightful question regarding our method's generalization. In the revised version, we include additional experiments using the TEdBench dataset (see Table 1, L412–419), which features diverse images spanning multiple categories and complex concepts.

Evaluation on TEdBench: We evaluate concept scaling down on this real-world dataset and compare the performance to existing methods (L442–462). Our method demonstrates strong generalization to diverse inputs and complex concepts, validating its overall effectiveness. Our approach surpasses existing methods on TEdBench, highlighting its robustness and superior performance.

---

More Cases in the Application Zoo

While the quantitative results in Table 1 validate the effectiveness of our method, the Application Zoo illustrates the versatility of ScalingConcept across diverse downstream tasks. In the revised paper, we include two new tasks to demonstrate the universal applicability of our method (see L805–833, Figs. 13 and 14):

• Face Attribute Editing: Adjusting attributes like age and smile.

• Anime Sketch Enhancement: Improving and refining anime sketches.

These cases emphasize the practicality and adaptability of our method to both common and novel tasks, further demonstrating its utility in real-world scenarios.

We hope these additions and clarifications address the reviewer’s concerns and demonstrate the broad applicability and simplicity of our method. Thank you for your valuable feedback!

Scientific Question in the Paper

We clarify that our scientific question focuses on how to scale the inherent concepts within real input effectively. Existing editing methods in both image and audio domains often emphasize providing editing effects through explicit instructions (e.g., InstructPix2Pix [1] and AUDIT [2]). These methods typically require specialized datasets and fine-tuning of diffusion models or attention control (e.g., [3]).

In contrast, our work emphasizes continuous scaling of inherent concepts in the input (as shown in Fig. 1). The key challenge lies in obtaining scalable concept representations directly from the real input, inspired by our observations in Sec. 3.2.

References:

[1] InstructPix2Pix: Learning to Follow Image Editing Instructions. (CVPR 2023)

[2] AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models. (NeurIPS 2023)

[3] Prompt-to-Prompt Image Editing with Cross Attention Control. (ICLR 2023)

---

Usage of Null-Prompt in Empirical Study and Analysis

We use a null prompt in the empirical study because of its versatility across all input types. Our method consists of two branches:

1. Reconstruction Branch: The concept used here is specified by the user during the inversion process.

2. Removal Branch: A null-prompt is employed to avoid additional user effort in selecting another concept.

To illustrate, Fig. 17 compares results using a null-prompt versus a replacement prompt. While the null-prompt achieves the desired removal effect, adding a replacement prompt can enhance results further.

We also propose an automatic and optional step (as shown in Fig.4) involving visual language models (for images) or audio-language models (for audio) to parse the concept and suggest a replacement prompt. This addition enriches the editing space and provides better controllability for users.

---

Experimental Comparison

We appreciate the reviewer’s feedback regarding the evaluation of our method. In the revised version, we include additional experiments using the widely-used TEdBench dataset (see Table 1, L412–419). TEdBench features diverse images with complex concepts, providing a real-world test for our method.

On TEdBench, we evaluate concept scaling down and compare against existing methods (L442–462). The results demonstrate that our method generalizes effectively to diverse datasets and complex concepts, outperforming existing methods.

Additionally, the quality of our proposed WeakConcept-10 dataset is visualized in Fig. 18. These images exhibit weak and incomplete representations of target concepts, making them ideal for evaluating concept scaling methods.

---

Analysis of Method Shortcomings

We acknowledge the reviewer’s concern that amplifying a certain concept in an image may weaken other concepts (as shown in the reviewer’s example). This issue could be mitigated by incorporating attention masks to constrain the scaling area, thus reducing interference with other concepts. Attention control is a well-explored area in editing methods ([1, 2, 3]), and we believe such techniques can be integrated into our framework to further enhance its effectiveness in future work (see L535–539).

References:

[1] Prompt-to-Prompt Image Editing with Cross Attention Control (ICLR 2023)

[2] LEDITS++: Limitless Image Editing using Text-to-Image Models (CVPR 2024)

[3] Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation (CVPR 2024)

---

LPIPS Performance Degradation with Noise Regularization

Better LPIPS performance indicates higher perceptual similarity between the scaled result and the original input. Using Noise Regularization without early exit introduces latent variables from inversion at every step, effectively providing the original input during sampling. This unsurprisingly improves LPIPS performance but significantly degrades generation quality (FID increases from 272.2 to 282.6).

To balance image quality and perceptual similarity, we introduce an early exit strategy, which maintains reasonable LPIPS while avoiding a substantial drop in FID.

---

Rationality of the Proposed Method

Our method is grounded in empirical observations (e.g., concept reconstruction and removal in Fig. 2) and validated across domains (e.g., audio in Sec. 3.2). The experiments on various datasets (Table1, Fig. 5/6) for both concept scaling up and scaling down demonstrate the method’s effectiveness compared to existing editing approaches.

Additionally, the Application Zoo highlights diverse application scenarios (see Fig. 7/8 and Appendix A.1), further showcasing the generalizability and practicality of our method.

While we believe the empirical evidence supports the method’s validity, we acknowledge that theoretical analysis could strengthen our work. This is an area we plan to explore in future research.

---

Formatting Issue

We have fixed them in the revised PDF.

More Diverse and Complex Datasets

We thank the reviewer for their insightful question regarding the evaluation of our method on diverse and complex datasets. In the revised version of our paper, we have included additional experiments using the public and widely-used image editing dataset TEdBench (see Table 1, L412–419 in the paper). TEdBench contains diverse images across various categories and incorporates complex concepts.

On this real-world dataset, we evaluate the performance of our method in concept scaling down, comparing it against existing methods (L442–462). The results demonstrate that our method generalizes effectively to other datasets and handles more complex concepts, validating the overall effectiveness of our approach. Moreover, our method surpasses existing methods in performance on this benchmark.

---

Application Scenarios

Our method provides two key functionalities (illustrated in Fig. 1):

1. Concept Scaling in Input: Our approach achieves automatic editing suggestions, which are particularly useful when users want to explore potential editing effects related to the concepts in the input.

2. Continuous Scaling Property: This property is valuable in scenarios where editing instructions are difficult to quantify using text alone. For instance, it allows adjustments in the presence (e.g., loudness) of an audio sound source within a mixture.

---

Scalability

We demonstrate the scalability of our method through experiments on both concept scaling up and scaling down across two datasets. Additionally, our Application Zoo showcases results in diverse domains, including but not restricted to:

• Natural Images
• Face Images
• Anime Images
• Audio Spectrograms

Our method supports high resolutions, with image outputs of 1024×1024 pixels and audio outputs up to 10 seconds in length, reflecting its ability to handle complex data.

---

Complexity and Sensitivity

Our method does not require fine-grained hyperparameter tuning. Specifically:

• For scaling up experiment in Table.1, we use $\omega_{\text{base}} = 5$.
• For scaling down experiment in Table.1, we use $\omega_{\text{base}} = -5$.
• Other hyperparameters, such as $\gamma = 3$ and $t_{\text{exit}} = 35$, are kept constant across experiments.

These results show that a single set of hyperparameters is generalizable to different image types.

Regarding inversion techniques, we use ReNoise as our default method. While not perfect, it delivers strong results, as evidenced by our experiments. Future advances in inversion techniques can be seamlessly integrated into our framework to further enhance performance.

---

Dependence on Text-to-X Association Quality

We acknowledge the reviewer’s concern about the dependence of text-guided diffusion models on the quality of text-to-X associations. The quality of text prompts is indeed critical, as suboptimal prompts may yield less-aligned results.

This limitation is inherent to most editing methods relying on frozen, pretrained text-guided diffusion models, whose understanding is bounded by their training. Two potential avenues for improvement are:

1. Automated Prompt Refinement: Leveraging tools such as LLMs to generate better prompts as a post-refinement step.

2. Enhanced Text-Guided Models: Improving the language understanding capabilities of text-guided diffusion models, which aligns with the broader mission of advancing generative AI. Encouragingly, we are witnessing consistent performance improvements in this area each year.

---

We hope this addresses the reviewer’s concerns. Thank you for your valuable feedback, which has helped strengthen our paper.