ParallelEdits: Efficient Multi-Aspect Text-Driven Image Editing with Attention Grouping

Q: Clarification of the inversion and editing method.

A: We agree with the reviewer. Each baseline in Table 1 and throughout the paper includes a submethod for inversion and another for editing. The table below details these submethods for each baseline. We will update line 36 of the introduction, Table 1/Figure 5 captions, and lines 269-272 to specify the submethods used for inversion and editing for each baseline.

Q: P2P is an editing method for generated images and needs inversion method to edit real image.

A: For P2P, we adopt the default settings in the original P2P paper [1] (introduced in the Real Image Editing section) to conduct DDIM inversion to edit real images.

Q: Is ParalleEdits able to add multiple object in the image?

A: ParallelEdits can add multiple objects to an image, as shown in Fig 4 of the main paper (adding harness and flowers) and Fig 10 in the Appendix (d,e,f). PIE-Bench++ includes test samples for evaluating multiple object addition. We tested the example provided by the reviewer—a cat wearing a necktie and sunglasses—as detailed in the document attached in the general response (Figure 1), and we welcome any additional challenging examples the reviewer wishes to suggest.

Q: Need to explain how to get the attention map for a real image.

A: An attention map is generated by running a few diffusion steps (up to 5 steps) as in line 472 of the main paper and it is created by averaging cross-attention matrices of UNet layers, as outlined in P2P (see section 3.2 of [1]). The main paper (line 175) mentions that the aspect grouping process uses a dual-branch diffusion model conditioned on source and target prompt to generate and group attention map. Attention maps from the two branches are associated with source prompt tokens and target prompt tokens, and the attention maps of source branch correspond to the aspects in real/source images. We will include these details in the appendix of the revised manuscript.

Q: Figure 2, Why does the attention map from the original image combine with the target prompt.

A: The source and target attention maps are associated with source and target prompt, while both source and target attention maps have been to used to identify the editing type, as shown in Figure 3. We visualize the target attention map overlaid in the original image to illustrate which part of the original image need to be edited based on the attention map.

Q: What if cross-attention is inaccurate? For example, is the duck attention map clear to show?

A: Yes, adding "ducks" to an image creates a cross-attention map as Figure 3 of the main paper shown. Current image editing works [1,2,3,4] have also used attention map to indict object layouts. The iterative diffusion process is able to generate an accurate object layouts in the attention maps ( as described in section 3.2 of P2P paper [1]). Moreover, attention maps have also been utilized for even more intricate tasks such as image segmentation [5]. Although failure situations resulting from faulty attention maps do occur, they are rare.

Q: Need qualitative and quantitative experiments to support the claim that order in which aspects are modified when applying single-aspect text-driven image editing methods can affect the quality.

A:. Thanks for pointing this out. We have conducted additional qualitative and quantitative experiments to support the claim, please check Table 2 and Figure 4 attached in the general response. We found that the AspAcc-CLIP is significantly different when using varying orders for each single-aspect editing method, and each method achieves the best performance with different orders. Hence, it is hard to choose a proper editing order for sequential editing, not to mention that sequential editing is extremely slow.

Other minor issues

Q: z should be $z_0$ in Line 119.

A: Yes, we will reviese the manuscript.

Q: The definition of tokens is not clear.

A: The tokens are words obtained from the tokenizer which includes nouns, adjectives and other words splitted by spaces. For example, in the figure 3 the prompt has been broken down into tokens, all the words are token but only the underlining ones are asepcts.

Q: Misused of reference.

A: Thanks for pointing this out, StyleDiffusion refers "Prompt-Embedding Inversion for Text-Based Editing", and Xu et al. refers the [3] Inversion-Free Image Editing with Natural Language. We will update the references in the revision.

[1] Prompt-to-Prompt Image Editing with Cross Attention Control, Hertz et al, ICLR23'

[2] DiffEdit: Diffusion-based semantic image editing with mask guidance, Couairon et al, ICLR 23'

[3] Inversion-Free Image Editing with Natural Language, Xu et al, CVPR24'

[4] MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, Cao et al, ICCV23'

[5] Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models, Marcos et al, CVPR24'

Q: The DDCM process in [1] comes to a simple conclusion that the output z equals $z_0$ after dozens of iterations.

A: Indeed, ParallelEdits' source branch uses DDCM, as described in [1], which uses the consistency sampling step popular in consistency model literature [6,7]. DDCM process for source branch was chosen based on [1]'s observations that the method does not suffer cumulative errors over sampling, making the overall method more efficient (fewer samples to generate target image), which is desirable for image-editing applications. Although $z=z_0$ in every iteration of Algorithm 1, the difference between the reconstructed output $z_\tau$ and $z_0$ as captured by $\epsilon_\text{cons}$ continues to evolve (see Figure 3b of [1]). The target branch is calibrated using $\epsilon_\text{cons}$ thereby introducing the desired edit.

Q: Results about intermediate editing results for sequential editing

A: We have included further results in Figures 3 and 4 of the attached document in the general response. As previously stated in line 48 of the main paper, sequential editing is an ineffective approach due to the potential for undoing previous edits and accumulating errors over time.

Q: Additional benchmark needs to be included for comprehensive evaluation.

A: We thank the reviewer for highlighting this interesting benchmark. Although it predominantly contains single-aspect editing samples, unlike the more complex multi-aspect samples in PIE-Bench++, we evaluated our method on it and reported the results in Table 1 of the global rebuttal. Our method achieves state-of-the-art performance in terms of the overall LMM score proposed in [2]. Inspired by this, we plan to release a benchmark focused on multi-aspect editing, using PIE-Bench++ and the Aspect-accuracy/preservation metrics from our paper. We are confident it will gain strong recognition from the community.

Q: Explanation of man has changed after editing in Fig. 2.

A: We appreciate the reviewer’s subtle observation. As shown in Figure 2, the “boat” object nearby has been transformed to “white rubber boat” and leaks into the “man” attention map, and the man has been edited to fit the context of changing a traditional boat to a modern rubber boat. Similar with other attention-map-based approaches [1,3,4,5], the attention maps could leak, even though such minor changes do not affect the overall context of this figure. Finally, such leakage can be easily solved by applying a higher threshold to the attention map.

[1] Inversion-Free Image Editing with Natural Language, Xu et al, CVPR24'

[2] Diffusion-Model Based Image Editing: A Survey, Huang et al, ArXiv24'

[3] Prompt-to-Prompt Image Editing with Cross Attention Control, Hertz et al, ICLR23'

[4] DiffEdit: Diffusion-based semantic image editing with mask guidance, Couairon et al, ICLR 23'

[5] MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, Cao et al, ICCV23'

[6] Latent consistency models: Synthesizing high-resolution images with few-step inference, Luo et al, ArXiv23'

[7] Consistency Models, Song et al, ICLR23'

Q: The pairing process imposes an additional burden on users when performing multi-object editing.

A: The algorithm receives the pairing process (editing action) to determine which aspect is added, removed, swapped, or left unaltered. Meta-data of this form is not a burden but a necessity because it precisely reflects the user's intent when editing without errors. A user-friendly UI can conveniently capture such meta-data. Moreover, prior works also benefit from meta-data e.g. [1] needs rich text input and [2,3] need editing pairs.

Q: How the hyperparameter $\lambda$ to be chosen and how it affects the editing.

A: The hyperparameter $\lambda$ is the threshold for determining whether an edit is classified as a rigid edit or a non-rigid edit (see equation 3 of the paper). The hyperparameter $\lambda$ was selected through cross-validation by identifying the optimal $\lambda$ that gave the highest aspect accuracy. Experiments showed that both quantitative results and qualitative results do not vary for different choices of $\lambda$. Please see the table below and the Figure 2 of the PDF appended to this rebuttal. We will gladly share additional results if the reviewer desires.

Q: Lacking discussions on controlling the editing strength of multiple editing.

A: We concur with the reviewer that edit strength control is important. As shown in [4,5], single attribute editing tasks provide clear editing strength and flexibility. As for multi-aspect editing, the reviewer would agree that it is unclear if edit for multiple aspect is itself possible. Moreover, evaluation metrics and benchmark datasets for multi-aspect editing are not yet community-standardized. Thus, instead of handling editing flexibility, contemporary efforts [6,7] focus on feasibility aspects of simultaneous multiple edits.

[1] Expressive text-to-image generation with rich text. Ge et al, ICCV23.

[2] Prompt-to-prompt image editing with cross attention control. Hertz, et al, ICLR23.

[3] Null-text inversion for editing real images using guided diffusion models. Mokady, et al, CVPR23.

[4] An edit friendly ddpm noise space: Inversion and manipulations. Huberman-Spiegelglas et al, CVPR24.

[5] Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. Wang, et al, CVPR23.

[6] Ground-A-Score: Scaling Up the Score Distillation for Multi-Attribute Editing. Chang, et al, Arxiv24.

[7] Iterative Multi-granular Image Editing using Diffusion Models. Joseph, et al, WACV24.