Vision-guided and Mask-enhanced Adaptive Denoising for Prompt-based Image Editing

1. Lack the comprehensive comparison with a wide range of image editing methods. i.e., [1][2][3]

[1] InstructPix2Pix: Learning to Follow Image Editing Instructions

[2] MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing

[3] Guiding Instruction-based Image Editing via Multimodal Large Language Models

1. Lack of Clarity on Prompt Limitations: The authors claim in Line 66 that "the target prompt can only highlight the core editing intention... and its capacity to depict finer visual details... is inadequate." However, this statement appears unsupported, as well-designed prompts can indeed capture nuanced visual details, such as color or expression changes in the edited subject. This limitation is not sufficiently justified or empirically verified.

2. Justification for Method Choice: While image embeddings are presented as providing more accurate guidance, it is unclear what specific challenges led to the development of this approach over existing methods. An explicit discussion on the limitations of prior approaches and the specific advantages offered by ViMAEdit’s embeddings would provide valuable context.

3. Contradictory Assertions on Attention-Based Methods: The claim in Line 74 that "existing methods mostly only rely on cross-attention maps...to locate the editing area as patches relevant to the blend words" seems inaccurate, as methods such as MasaCtrl allow both auto-aggregated masks and user-specified masks, while others like PnP do not require a mask, and LEDITS++ employs averaged masks for all editing tokens. This generalization appears to overlook these alternatives.

4. Potential for Misidentifying Editing Areas: The self-attention-guided iterative grounding module aims to refine word-to-patch relationships to produce a mask targeting the salient object. However, this approach may struggle when the target area is not the most prominent object, such as when editing a minor object in a detailed scene. It remains unclear how the proposed method manages cases where the target object is not the most visually dominant element.

5. Sampling Variance and Editing Strength: The assertion in Line 87 that "sampling variance controls the editing strength applied to each pixel..." seems overstated. While increased sampling variance introduces greater variation, this does not necessarily translate to stronger or more controlled editing. This approach risks introducing inconsistencies without a clear mechanism for managing variance across different parts of the image.

6. Inconsistencies with Dissimilar Prompts: The paper proposes an image embedding-enhanced denoising process, estimating the target image embedding based on the source image using Equation (7). If there is a significant semantic gap between the source and target prompts, however, the resulting embedding may lack coherence or relevance. How the method adapts to or mitigates inconsistencies arising from such disparities is not addressed.

7. Excessive Alterations in Results: Figure 5 shows cases where the method alters background details or adjusts object pose and view significantly. This behavior may indicate a lack of control over the degree and specificity of the edits, suggesting further refinement is necessary for tasks requiring subtler changes.

8. Absence of User Study for Evaluation: A user study evaluating the method's efficacy in real-world applications, such as its performance on varied and user-specified edits, would add significant value. User studies are instrumental in assessing how well the method aligns with user intent and the overall editing experience.

9. Complexity in Practical Applications: Real-world image editing tasks often involve scenes with multiple objects, where edits must be made based on flexible prompts. The need to specify a source prompt, blend word, and various conditions can complicate the process and may restrict the practical usability of the system for typical users seeking simpler image adjustments.

• The paper introduces $\triangle e$ to represent the difference between text embeddings in the CLIP embedding space. However, it is unclear how $\triangle e$ and the image embeddings are processed together. Specifically, for models like Stable Diffusion 1.5, the text embedding dimension is $77 \times 768$. How does the model align this dimension with the image embeddings, which typically have a different structure? More clarification on the dimensional operations and how the embeddings are effectively integrated is required.

• The proposed approach utilizes an IP-Adapter module, which is trained separately and introduces additional knowledge. This creates an unfair advantage compared to other baseline methods that do not use such a pre-trained module. It would be more appropriate to conduct separate ablation studies to highlight the impact of IP-Adapter or to compare methods without the additional pre-trained components to ensure fair comparisons.

• Typo error: On line 309, "$P_2 \times P_2$" should be "$P_2$ ."

1. Comparison with State-of-the-Art Methods: It would be more validating if the proposed method were compared with state-of-the-art methods, such as Guide-and-Rescale [1] and MAG-Edit [2]. Especially, MAG-Edit, which has a very similar setting.

2. Ablation Study Lacks Qualitative Results: The ablation study lacks qualitative results, which are essential to confirm the role of the image embedding mentioned in lines 65-72. Including qualitative visualizations would help clarify the importance of each component in the proposed approach.

3. Insufficient Explanation of Why Proposed Key Aspects matter: The qualitative experimental results show the advantage of the proposed method in the alignment of adjectives such as color editing, but it is not clear which specific techniques contribute to achieving good alignment. From our experience, aligning adjectives like in Figure 4 (the example of the flower) is challenging if P2P can not address it well. The paper could benefit from explaining why the adjective alignment works effectively and which methods facilitate this, such as in MAG-Edit[2], which proposes cross-attention alignment. Since a better grounding technique could potentially preserve the background more effectively without negatively impacting the editing results, it could not contribute to the alignment of editing regions. Authors are suggested to explain why their method can do well in such a complicated setting like figure 4 a blue flower with red in the center.

4. Single Object Limitation: The results presented in the paper only contain a single object. It is unclear if the self-attention guided grounding can effectively locate a single object in images with multiple objects (for instance, multiple dogs, only edit one dog). This limitation should be addressed to demonstrate the method's applicability in more complex scenarios. Although the self-attention-guided iterative region grounding is a promising approach, its robustness in challenging or diverse image scenarios is not sufficiently addressed. It would be beneficial for the authors to include additional experiments to verify the method's consistency across varied and complex datasets.

5. Time-Consumption of Iterative Grounding: There is no discussion on whether the iterative grounding process is time-consuming. The authors should provide an analysis of the computational cost of the iterative approach.

6. User Study: Including a user study would enhance the credibility of the proposed method.

References: [1] Titov V, Khalmatova M, Ivanova A, et al. Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing[C]//European Conference on Computer Vision. 2024. [2] Mao Q, Chen L, Gu Y C, Fang Z, Shou M Z. MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidance[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 6842-6850.