EditInfinity: Image Editing with Binary-Quantized Generative Models

Thank you for your insightful comments. We respond to your concerns point by point as follows.

Main Comment

Q1: Please add more comparisons with top models like gpt-4o or Gemini.

A1: We sincerely thank the reviewer for pointing out this valuable concern. As shown in Table 1 below, we provide a comparison with the closed-source Gemini, which serves as a strong commercial baseline. Due to time constraints, we are not able to include results for GPT-4o. Nonetheless, we believe the comparison with Gemini provides a representative and meaningful reference point for assessing our method. As shown in Table 1, the thoughtful design of our method enables it to outperform even the closed-source Gemini.

Table 1: Quantitative results of multi-scale autoregressive editing.

Q2: Better ablation study for multi-scale autoregressive editing instead of putting one visualization example.

A2: We thank the reviewer for this insightful question. Due to space limitations in the main paper, this has been discussed in Table 4 of the supplementary material.

Table 4: Quantitative results of multi-scale autoregressive editing.

By blending source tokens at each scale in an autoregressive (AR) manner, our method provides continuous guidance for editing region generation at subsequent scales. In contrast, the non-autoregressive (NAR) approach blends source tokens only at the end of each scale, without influencing the token generation process at the next scale. This leads to incoherent transitions and visually inconsistent edits, as illustrated in Figure 9 of main paper and Figure 2 of supplementary material. Table 4 here further supports this observation: AR consistently outperforms NAR in both background preservation and text alignment. These results highlight the necessity of autoregressive guidance for achieving harmonious and realistic edits.

Q3: Better write-up for method section 4, including a/ better definition of symbols, b/ training strategy (stage 1, stage 2, ....)

A3: We sincerely thank the reviewer for pointing out the need for clearer symbol definitions in Figures 2 and 3, as well as a more structured presentation of the training strategy in Section 4.

(a) We have added dedicated legend areas to Figures 2 and 3 to explain the meaning of all relevant symbols, such as $t_{sou}, t_{ins}, t_{l}, R_{1...K}^{sou}, R_{1...K}^{inv}, R_{1...K}^{tar}, E_{1...K}^{tar}, \mathcal{L}_{inv}, \mathcal{G}$, etc. As the rebuttal does not support figures, we are unable to show the updated figures here.

(b) We have clearly annotated Stage 1 and Stage 2 in Figure 2 and provided a more precise and structured definition of both stages in Section 4, as outlined below.

• Image inversion stage 1: textual prompting rectification.
• Image inversion stage 2: image style preservation.

We greatly appreciate your suggestions, which will help improve the clarity and rigor of our method section.

We sincerely appreciate your thoughtful and constructive feedback. Below, we respond to each of your comments in detail and have incorporated the corresponding revisions into the final version of the paper.

Main Comment

Q1: How well does the method handle vague or high-level prompts that lack explicit object references or structure?

A1: We sincerely thank you for highlighting this concern. We provide a detailed response from the following two perspectives: ’Prompt Understanding‘ and ’Open-ended Editing Tasks‘.

(a) Prompt Understanding. Our method is designed to operate with known editing regions and does not focus on prompt understanding. In scenarios involving vague or high-level prompts without clear object references, external multimodal large language models (MLLMs) may be integrated to enhance prompt understanding.

(b) Open-ended Editing Tasks. Given that PIE-Bench contains nearly 50% natural images, it already serves as a comprehensive benchmark for assessing our method. However, to specifically evaluate complex scenes, we select 20 MagicBrush images (due to time constraints) filtered by GPT-4o, comprising five samples each with 2, 3, 4, and 5 primary objects. Table 1 demonstrates the superiority of our method in handling complex scenes.

Table 1: Quantitative results on the complex scene images from MagicBrush .

Note: Due to time constraints, we select representative methods with strong performance from both diffusion U-Net-based models and diffusion transformer-based models for comparison.

Q2: The inversion and editing stages seem computationally demanding. What is the latency for a single edit?

A2: We conduct a runtime comparison of our method and other methods on a single NVIDIA L20 GPU, measuring both inversion and editing time, as shown in Table 2 below. A key advantage lies in its efficiency during multiple edits on one image—a common real-world scenario. Once the inversion for a given image is completed, subsequent edits can be performed within 3.64 seconds—over 7× faster than other methods on average (initial inversion time only 4× longer than other methods on average). This design effectively front-loads the computational cost.

Table 2: Runtime comparison of inversion and editing.

Q3: How is the edited region determined in practical scenarios? Can the method infer edit regions automatically based on differences between source and target prompts, or does it rely entirely on manual mask annotation?

A3: We appreciate this insightful question. Our method assumes the user provides masks. Indeed, this is a well‑established task setting in image editing [1] [2], especially when text alone is insufficient for the precise localization of the user-desired editing region. This challenge of accurately conveying user intent has long been recognized in controllable image generation. To enhance controllability, ControlNet [3] leverages visual priors such as edge maps, while DreamBooth [4] utilizes user-provided images to capture detailed features not easily conveyed by text.

While our method assumes user-provided masks by default, it can also leverage Infinity’s cross-attention maps [5] for automatic mask generation without modifying the framework. Specifically, we automatically align differing words $x$ between the source and target prompts. After completing inversion, we input the source or target prompt containing $x$ into Infinity and extract the cross-attention map corresponding to $x$. A threshold is then applied: values above the threshold are set to 0 (mask foreground), and others to 1 (background). Table 3 shows that our method is not highly sensitive to the source of the mask—strong performance is achieved in both cases. We will include further discussion on this aspect in the final version.

Table 3: Quantitative results on the random class of PIE-Bench. Ours-u denotes user-provided masks; Ours-c denotes cross-attention masks. Best and second-best results are shown in bold and italics.

Note: Due to time constraints, experiments are conducted on the “random class” of PIE-Bench, which includes all editing types and supports efficient and unbiased evaluation of model components.

[1] Nitzan Y, et al. Lazy diffusion transformer for interactive image editing[C]//ECCV. 2024.

[2] Zhuang J, et al. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting[C]//ECCV. 2024.

[3] Zhang L, et al. Adding conditional control to text-to-image diffusion models[C]//ICCV. 2023.

[4] Ruiz N, Li Y, et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation[C]//CVPR. 2023.

[5] Amir Hertz, et al. Prompt-to-Prompt Image Editing with Cross-Attention Control[C]//ICLR. 2023.

Thanks for your careful and valuable comments. We will explain your concerns point by point.

Main Comment

Q1: How are the masks used in the editing process obtained in the evaluations? Does the method assume that these masks are provided by the user or is there an automated pipeline to achieve this?

A1: We appreciate this insightful question. Our method assumes the user provides masks. Indeed, this is a well‑established task setting in image editing [1] [2], especially when text alone is insufficient for the precise localization of the user-desired editing region. This challenge of accurately conveying user intent has long been recognized in controllable image generation. To enhance controllability, ControlNet [3] leverages visual priors such as edge maps, while DreamBooth [4] utilizes user-provided images to capture detailed features not easily conveyed by text.

While our method assumes user-provided masks by default, it can also leverage Infinity’s cross-attention maps [5] for automatic mask generation without modifying the framework. Specifically, we automatically align differing words $x$ between the source and target prompts. After completing inversion, we input the source or target prompt containing $x$ into Infinity and extract the cross-attention map corresponding to $x$. A threshold is then applied: values above the threshold are set to 0 (mask foreground), and others to 1 (background). Table 1 shows that our method is not highly sensitive to the source of the mask—strong performance is achieved in both cases. We will include further discussion on this aspect in the final version.

Table 1: Quantitative results on the random class of PIE-Bench. Ours-u denotes user-provided masks; Ours-c denotes cross-attention masks. Best and second-best results are shown in bold and italics.

Note: Due to time constraints, experiments are conducted on the “random class” of PIE-Bench, which includes all editing types and supports efficient and unbiased evaluation of model components.

[1] Nitzan Y, et al. Lazy diffusion transformer for interactive image editing[C]//ECCV. 2024.

[2] Zhuang J, et al. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting[C]//ECCV. 2024.

[3] Zhang L, et al. Adding conditional control to text-to-image diffusion models[C]//ICCV. 2023.

[4] Ruiz N, Li Y, et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation[C]//CVPR. 2023.

[5] Amir Hertz, et al. Prompt-to-Prompt Image Editing with Cross-Attention Control[C]//ICLR. 2023.

Q2: Is the method able to achieve disentanglement for un-maskable edits such as facial attribute change?

A2: We thank the reviewer for this insightful question.

(a) Facial attribute change. Yes, our method can realize facial attribute change. PIE-Bench, used for evaluation in main paper, is a comprehensive image editing benchmark covering natural human-centric scenarios. We use GPT-4o to filter editing instructions, resulting in 16 representative examples such as 'change the expression from laughing to angry' and 'add the age of the woman'. Our method performs favorably on these cases. Please refer to Table 1 in the main paper for qualitative results, given image upload constraints.

|-- random
|-- change_object
     |-- natural*
          |-- animal
          |-- human*
          |-- indoor
          |-- outdoor
     |-- artificial
          |-- ...
|-- add_object
     |-- ...
|-- delete_object
     |-- ...
|-- change_attribute_content
     |-- ...
|-- change_attribute_pose
     |-- ...
|-- change_attribute_color
     |-- ...
|-- change_attribute_material
     |-- ...
|-- change_background
     |-- ...
|-- change_style
     |-- ...

Furthermore, we randomly select 20 face images from FFHQ (due to time constraints) to perform unmasked facial attribute changes, including age, expression, skin tone, and more. Since the setting is unmasked, there is no background to preserve, and thus standard metrics that rely on background consistency are not applicable. In addition to retaining the Whole metric (CLIP score between the entire edited image and the target prompt), we introduce ArcFace [6] for evaluating identity preservation, and CLIP-I for measuring similarity between the source and edited images. The results are reported in Table 2 below, which shows that our method achieves the best performance compared to others. Thanks to our proposed image inversion algorithm, which effectively learns the generative trajectory of the source image, our method can accomplish the intended edits.

Table 2: Quantitative results on facial images from FFHQ.

Note: Due to time constraints, we select representative methods with strong performance from both diffusion U-Net-based models and diffusion transformer-based models for comparison.

[6] Jiankang Deng, et al. ArcFace: Additive Angular Margin Loss for Deep Face Recognition[C]//CVPR. 2019.

(b) On-the-wild images. Given that PIE-Bench contains nearly 50% natural images, it serves as a comprehensive benchmark for assessing our method. However, to specifically evaluate complex scenes, we select 20 MagicBrush images (due to time constraints) filtered by GPT-4o, comprising five samples each with 2, 3, 4, and 5 primary objects. Table 3 demonstrates the superiority of our method in handling complex scenes.

Table 3: Quantitative results on the complex scene images from MagicBrush .

Q3: What are the failure cases of the method in detail?

A3: We appreciate your valuable suggestion. In the final version, we will provide additional failure cases and elaborate on the limitation as follows.

Limitation. While our method demonstrates strong performance across diverse editing tasks, it shows limitations in extreme cases such as style change, where no background needs to be preserved, and the image contains detailed structural patterns. In such cases, the blending between source and target tokens is constrained, which may lead to suboptimal preservation of structural fidelity from the original image. Nonetheless, thanks to our image inversion strategy, which effectively learns the generative trajectory of source image, our method can still accomplish intended edits, despite slight structural degradation. In contrast, other methods often fail in such challenging scenarios. For example, as shown in Figure 6, row 9 of the main paper, while other methods are unable to convert the painted bird into a realistic one, our method successfully achieves the style change, with only minor deviation in the bird's head pose.

Q4: Is there are timing constraint on the method, as a two stage optimization is required for editing?

A4: We conduct a runtime comparison of our method and other methods on a single NVIDIA L20 GPU, measuring both inversion and editing time, as shown in Table 4 below. A key advantage lies in its efficiency during multiple edits on one image—a common real-world scenario. Once the inversion for a given image is completed, subsequent edits can be performed within 3.64 seconds—over 7× faster than other methods on average (initial inversion time only 4× longer than other methods on average). This design effectively front-loads the computational cost.

Table 4: Runtime comparison of inversion and editing.

Thanks for your comments. We will explain your concerns point by point.

Main Comment

Q1: I would like to encourage the authors provide more detailed explanation on the optimization of t_l and the blending of quantized tokens.

A1: We appreciate the reviewer’s helpful suggestion. We now explain in detail (i) how our method optimizes $t_l$, and (ii) how the blending of quantized tokens is performed. We have also refined the narrative in the main paper to make these components clearer and better integrated.

(a) The optimization of $t_l$:

First, we introduce learnable prompt embeddings $t_l$, whose size is fixed to 20 tokens. Second, we pass both the source prompt $t_{sou}$ and the instruction prompt $t_{inv}$ (i.e., "the language style of this prompt is") through text encoder of Infinity $\Psi(\cdot)$ to obtain text embeddings $\Psi(t\_{sou}, t\_{ins})$. We then concatenate those embeddings with $t_l$ to form the textual conditioning input for Infinity. Finally, while keeping Infinity's weights frozen, we optimize only $t_l$ using cross-entropy loss, where the supervision signal comes from tokens $R_{1...K}^{sou}$ derived from the source image.

(b) The blending of quantized tokens:

(i) The blending operation is performed on the quantized features $(\\{R_{k}\\}_{k=1}^K, R_k \in \mathbb{R}^{h_k \times w_k \times d})$ rather than on indices, which is formulated as $E_k^{tar} = \mathrm{Upsample}(R^{tar}_k, (h_K,w_K)) \odot (1-\mathcal{G}) + \mathrm{Upsample}(R^{sou}_k, (h_K,w_K)) \odot \mathcal{G}$.

At each step $k$ of the autoregressive generation, the target token $R_k^{tar}$ is generated conditioned on the concatenation of the target prompt embedding and instruction embedding $\Psi(t\_{tar},t\_{ins})$, and optimized $t_l$. It is then blended with source tokens $R_k^{sou}$ of the source image under the guidance of the piecewise linear smoothing kernel $\mathcal{G}$. Although this formulation is already provided in Algorithm 1 of the main paper, we appreciate the reviewer’s suggestion and will enhance the textual explanation in the final version to improve clarity and emphasis.

(ii) The feasibility of performing linear blending at the quantized feature level is grounded in the fact that the Infinity decoder takes quantized features as input to generate images. As such, performing interpolation in the feature space is both reasonable and commonly adopted in generative modeling [1-3], as it supports smooth and semantically meaningful transitions.

[1] Alec Radford, et al. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[C]//ICLR. 2016.

[2] Kingma D P, Welling M. Auto-encoding variational bayes[C]//ICLR. 2014.

[3] Zhu J Y, et al. Toward multimodal image-to-image translation[C]//NeurIPS. 2017.

Q2: Also, I would like to see clarification on the experimental results. Are other baseline methods also using Inifinity as the base model? If not could the worse results for these methods be due to their worse base models?

A2: This is an insightful question. Our method is a lightweight framework tailored to Infinity and indeed relies on the generative capacity of the base model. However, this dependency is not unique to our approach. We explicitly present base models used by all methods in Table 1 below and evaluate their generative performance on the GenEval benchmark [4], one of the most widely used benchmarks for assessing text-to-image generation quality. As shown in Table 2, Infinity achieves comparable overall generative performance to the recently popular FLUX [5], and even underperforms in certain tasks, such as two-object and counting. Nevertheless, our Infinity-based image editing method significantly outperforms FLUX-based approaches like StableFlow and RF-Edit, demonstrating the effectiveness of our method spite the base model not having a clear advantage.

Table 1: Quantitative results on PIE-Bench.

Table 2: Evaluation on the GenEval benchmark.

Note: When evaluating Infinity, we adopt the same evaluation protocol as used for Stable Diffusion v1.4, v1.5, and FLUX.1-dev, i.e., without prompt rewriting.

[4] Dhruba Ghosh, et al. Geneval: An object-focused framework for evaluating text-to-image alignment[C]//NeurIPS. 2024.

[5] Black Forest Labs. Flux. 2024.