CAMILA: Context-Aware Masking for Image Editing with Language Alignment

We sincerely thank the reviewer for the thoughtful and encouraging feedback. We are especially grateful for the recognition of the importance of instruction executability, the practical relevance of our problem setting, the effectiveness of our proposed Token Broadcaster module, and benchmark dataset design. Below, we provide our responses to the weaknesses and questions.

---

[Weakness 1] Marginal Improvement of Surrogate Module Training

We acknowledge the reviewer’s observation that the improvement from the surrogate module is marginal. However, even without the surrogate module training, our framework outperforms other baselines on most metrics, so we introduced the surrogate module as an additional training for potential refinement. We had expected further gains in text alignment score, assuming that improving CLIP-T score would more capture user-intended modifications. While we did observe some improvement, further gains were inherently limited by the already strong baseline performance. We plan to investigate more advanced strategies to secure larger improvements in future work.

---

[Question 1] Detailed Analysis of Synthetic Data Generation Process

We appreciate the request for a more detailed analysis of our synthetic data generation process. We used the “gpt-4o-2024-05-13” model to generate non-applicable instructions. These were based on a controlled but flexible prompt covering various editing types (add, remove, replace, transform, etc.), requiring that the target object not be present in the image and that the instruction remains realistic and grounded. We include the full generation prompt below, which was paired with the input image to generate five non-applicable instructions for each image.

“I want to create image editing instructions for an image. Instructions like 'Put something', 'Remove something', 'Replace something', 'let there be', 'have there be', 'add something', 'make something something', 'Change something', 'What if something is', 'transform something to something' and similar. \n\n Generate five non-applicable image editing instructions for the given image. \n1. The instructions must target objects or elements that are not present in the image.\n2. Instructions can involve adding, removing, or modifying objects, but:\nIf adding, it must involve placing something next to, inside, or interacting with an object that does not exist in the image.\nIf removing or altering, the instruction should involve an object that is completely absent from the scene.\n3. Ensure the instructions are grounded in reality and fit the theme of the image (e.g., no fantastical elements).”

This ensures that the generated non-applicable instructions are not only structurally diverse but also semantically grounded in the actual content of the image, avoiding generic or gameable patterns. Additionally, to ensure the instructions were not trivially gameable, we randomly sampled one instruction out of the five and used the GPT model to explicitly verify its non-applicability to the image. This two-stage filtering guarantees semantic correctness and variety. Furthermore, Appendix C.2 shows that non-applicable instructions are distributed across multiple editing types, supporting the conclusion that the dataset is not skewed toward a single pattern.

We thank the reviewer for providing insightful feedback. We are encouraged by the reviewer’s acknowledgement that our paper tackles an important challenge in complex image editing and that our method demonstrates superior performance over existing baselines. Below, we respond to the reviewer’s points.

---

[Weakness 1] Technical Novelty

We appreciate the reviewer’s concern regarding the novelty of our proposed method. While CAMILA leverages established components such as pretrained multimodal large language models (MLLMs) and pretrained diffusion models, its core novelty lies not in introducing a new architecture per se, but in defining and addressing a previously overlooked problem in text-guided image editing: the executability of instructions in complex, real-world scenarios involving multiple instructions.

Existing text-guided image editing models typically assume that all instructions are fully executable and attempt to apply them uniformly. This often leads to semantic failures when instructions are non-applicable or refer to nonexistent objects. CAMILA departs from this paradigm by introducing a context-aware mechanism that explicitly distinguishes between executable and non-executable instructions before any editing takes place.

Specifically, we introduce specialized tokens, [MASK] and [NEG] tokens, to gate editability. To the best of our knowledge, this is novel in the context of image editing, particularly in alignment with the visual semantics of the image. Furthermore, we designed the Token Broadcaster to ensure the coherence between the instructions and the image, enabling relevant edits to be applied only to appropriate regions while ignoring non-executable instructions.

In this way, CAMILA introduces a new modeling objective, context-awareness and executability filtering, along with new modules to implement it, and provides empirical results that demonstrate its effectiveness, particularly in failure cases that existing methods are unable to handle. We respectfully argue that this constitutes a meaningful and original contribution to the field of text-guided image editing.

---

[Weakness 2] Lack of Recent Baselines

We appreciate the reviewer’s suggestion to include additional comparisons. However, we would like to clarify the following practical considerations.

First, SEED-Edit [1] is not available as an open-source model, which makes direct experimentation and fair comparison infeasible at this time. Also, our model and all the baselines included in the paper are built on Stable Diffusion v1.5 [2], which operates on 512×512 resolution and has been widely used in text-guided image editing. In contrast, SDXL [3], used by SEED-Edit and other recent models, is pretrained on 1024×1024 resolution and differs significantly in both architecture and capacity. Direct comparison across models that rely on different diffusion backbones is not straightforward and may result in unfair evaluation, as the underlying architectures differ in model capacity and resolution.

Similarly, Step1X-Edit [4] employs a different diffusion backbone, DiT (Diffusion in Transformer), which again makes direct comparisons challenging. However, as the model is publicly available, we conduct experiments on Context-Aware Instruction dataset and include the comparison results in Table 1.

Table 1. Additional baseline comparison on Context-Aware Instruction dataset

CAMILA outperforms the state-of-the-art Step1X-Edit model across all metrics. This indicates that, like other models, the diffusion-based Step1X-Edit lacks the capability for true context-awareness. Furthermore, while Step1X-Edit takes 55.4 seconds to run on an NVIDIA A100 80GB GPU, our model only requires about 9.2 seconds. Despite being a significantly smaller diffusion model than Step1X-Edit, our model achieves better performance due to its context-awareness capabilities.

We hope this clarifies the rationale behind our baseline selection and the scope of our comparative analysis.

---

[Weakness 3] Scope of Edits

We thank the reviewer’s observation regarding the scope of edits. Our Context-Aware Instruction Image Editing dataset includes global editing operations, such as background changes. Our model generates [MASK] tokens that are aligned with the semantics of the given instruction. Consequently, the size and coverage of the edited region are entirely determined by editing instruction. When an instruction implies a global transformation, such as changing the background, the model produces correspondingly large masks, enabling global edits.

This is demonstrated in the examples provided in the Appendix. Specifically, Figure 5 (c), (d) and Figure 9 (e) illustrate edits that affect the background of the image, showcasing the model’s ability to perform global modifications. We believe these examples support the model’s capacity to handle both local and global editing instructions, depending on the context provided.

---

References

[1] Shi, Yichun, Peng Wang, and Weilin Huang. "Seededit: Align image re-generation to image editing." arXiv preprint arXiv:2411.06686 (2024).

[2] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Podell, Dustin, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis." arXiv preprint arXiv:2307.01952 (2023).

[4] Liu, Shiyu, et al. "Step1x-edit: A practical framework for general image editing." arXiv preprint arXiv:2504.17761 (2025).

Dear Reviewer QraL,

Thank you for your previous comments and for the discussion so far. We hope our recent responses have addressed your questions and clarified the points you raised.

We would be happy to engage in any further discussion and look forward to hearing your thoughts.

Thank you again for your time and valuable feedback.

Best regards,

The Authors

Thank you for your review. However, we respectfully believe that there may have been some misunderstandings or discrepancies between your comments and the content of our submission.

• The summary mentions "Context-Aware Aggregation (CAA)" and "masked token reconstruction," but our paper does not contain these terms or such methods. Similarly, benchmarks such as ImageNet-1K, ADE20K, and COCO are not used in our work.
• Our method, CAMILA, is not an encoder-decoder architecture and does not involve any reconstruction-based tasks or visualizations, which are mentioned in your listed strengths.
• Additionally, the citation at the end of the review ("PnP Inversion: Boosting Diffusion-based Editing…") is not related to our paper and does not appear in our references.

Given these points, we are concerned that the review may not reflect a full reading or an accurate understanding of our submission. We would greatly appreciate it if you could revisit the paper to reassess its contributions based on its actual content.

Thank you again for your time and effort. Below, we respond to the reviewer’s points.

---

[Weakness 1] Limited Theoretical Justification

While we agree that theoretical analysis is always valuable, CAMILA is primarily proposed as a practical solution for context-aware image editing. Regarding the mention of “receptive fields or token mixing patterns,” we are not sure about the specific context in which this was raised. We would be happy to provide further explanation during the discussion period if the reviewer could clarify this point.

---

[Weakness 2] Missing Token Broadcaster Ablation

We acknowledge the reviewer’s concern regarding the lack of formal proof of Token Broadcaster optimality. To address this point quantitatively, we measure the accuracy of token alignment through our Token Broadcaster module. Our Token Broadcaster achieves a token alignment accuracy of 94.46% on our Context-Aware Image Editing dataset, which demonstrates the precision of our alignment strategy.

---

[Weakness 3] Uncertainty in Mask Accuracy

When the attention map is extracted from a UNet cross-attention layer of a diffusion model, as FoI did, it does not effectively capture contextual relationships and is insufficient for understanding non-executable content. The UNet cross-attention layer is based solely on the entire prompt input, so it lacks the ability to understand the overall context.

Dear Reviewer T2mz,

We're writing to gently follow up on our response to your comments. We hope our clarifications helped address your concerns and provided a clearer perspective on our work.

We would be happy to engage in any further discussion and look forward to hearing your thoughts.

Thank you again for your time and valuable feedback.

Best regards,

The Authors

We thank the reviewer for the thoughtful feedback. Also, we are encouraged by the recognition of our core contribution, modeling instruction executability with our specialized tokens, as a promising direction for text-guided image editing. Below are our responses to each comment.

---

[Weakness 1] Training and Inference Efficiency

The concern about the training and inference overhead of CAMILA raises an important point. While CAMILA integrates multiple components, including an MLLM, Token Decoder, Token Broadcaster, and diffusion model, we would like to clarify the implications of this design in terms of inference runtime and training cost.

As shown in Appendix D.2 and Table 9, CAMILA exhibits a higher total inference time than MGIE and SmartEdit. However, it achieves the lowest MLLM inference time among all models, reflecting the lightweight design of our Token Decoder and Token Broadcaster modules compared to the more complex components used in MGIE and SmartEdit. While CAMILA and FoI show increased diffusion time due to attention modulation on the IP2P backbone, CAMILA provides the unique advantage of supporting single-shot, multi-instruction editing without preprocessing (e.g., keyword extraction or sequential handling). This leads to a more integrated and streamlined editing pipeline, despite slightly higher latency.

From a training time perspective, CAMILA is significantly more efficient than other MLLM-based methods. Unlike MGIE and SmartEdit, which finetune both the MLLM and the diffusion model jointly, our approach updates only the MLLM, Token Broadcaster, and Token Decoder. This design substantially reduces training cost and memory overhead. Although FoI is a training-free method and thus not directly comparable, among MLLM-based editing frameworks, CAMILA offers a more compute-efficient training strategy while achieving competitive or superior editing performance.

---

[Weakness 2] Design Choice of Token Decoder

We thank the reviewer for raising the question regarding our choice to use a two-layer Transformer in the Token Decoder. While we do not currently provide an ablation study on the number of layers, our design decision was guided by the architecture of the Segment Anything Model (SAM) [1]. SAM’s mask decoder is composed of two attention blocks, each consisting of self-attention, cross-attention, MLP, and cross-attention layers. To preserve comparable model capacity, we adopted a comparable two-block configuration in our Token Decoder.

Unlike SAM, which does not take edit instructions as input, our decoder explicitly conditions on text instructions through a cross-attention mechanism. This design enables our model to produce spatially grounded editing masks that are semantically aligned with the given instructions. As shown in Appendix D.1, our model outperforms SAM in instruction-based editing tasks, demonstrating the effectiveness of this architecture.

We agree that further investigation into architectural choices, such as the number of decoder layers, would be a valuable direction for future work.

---

[Weakness 3] Main Novelty

We acknowledge the reviewer’s concern regarding the novelty of our approach. While CAMILA builds on existing components such as pretrained MLLMs and diffusion models, its primary contribution lies in identifying and tackling a previously underexplored challenge in text-guided image editing, determining whether instructions are actually executable in complex, multi-instruction scenarios. To this end, we introduce [MASK]/[NEG] tokens along with the Token Decoder and Token Broadcaster modules to explicitly represent instruction executability and align text with image regions, enabling context-aware editing.

---

[Question 1] Mask Precision

We thank the reviewer for pointing out the failure cases in Appendix E.3, particularly regarding under- or over-segmentation of small and thin objects. We agree this is an important challenge and believe the issue arises from two key factors:

1. Limitations of the ground-truth masks in the MagicBrush dataset: We utilized the MagicBrush dataset because, to the best of our knowledge, it is currently the only publicly available dataset that supports multi-instruction editing with editing masks. While we are unable to directly share examples of the masks due to the review policy, we encourage the reviewer to inspect the dataset directly, as it is publicly accessible, and observe that many of the masks are relatively coarse. Since our model is trained on these masks, it naturally inherits their limitations, particularly in handling very small or thin objects, where accurate mask supervision is lacking.

2. Resolution bottleneck introduced by Stable Diffusion’s attention modulation: Although the model inputs and predicted masks are in 512×512 resolution, the Stable Diffusion backbone internally downsamples masks to 16×16 resolution during attention modulation. This reduces the model’s capacity to precisely localize or attend to structures smaller than approximately 32×32 pixels in the original image. Consequently, extremely small or thin regions are often not sufficiently represented, which can lead to segmentation errors.

These two factors jointly contribute to the observed failure cases. We believe that future improvements could be achieved by incorporating higher-quality, fine-grained masks and adopting diffusion models with higher spatial resolution conditioning. We consider this a promising direction for future work.

---

[Question 2] Generalization beyond the Synthetic Benchmarks

We appreciate the reviewer’s concern regarding the generalization beyond the synthetic benchmarks and the suggestion to evaluate CAMILA on a user-collected set. In response, we conducted an additional experiment using human-generated non-executable instructions.

Specifically, we randomly sampled 8 images from the MagicBrush test set, and we asked our peers to generate 5 non-executable instructions per image, resulting in a total of 40 human-generated instances. We mixed these non-applicable instructions with the original applicable ones from MagicBrush in random order. Human-generated non-executable instructions were intentionally designed to resemble realistic user mistakes or confusion. For example, requesting to “remove the zebra” when the image contains a black horse, or asking to “change the red green tree to yellow,” reflecting syntactically or semantically ambiguous phrasing.

Using this user-collected dataset, we evaluated CAMILA’s token classification performance. The results show that CAMILA effectively filters out non-applicable instructions, achieving a token classification accuracy of 70% (28/40) on this human-generated dataset. While CAMILA achieved a token classification accuracy of 90.21% on the original synthetic benchmark, its performance on the user-collected dataset was lower. However, this is expected, as the model was not trained on any human-generated non-applicable instructions, and the user-collected evaluation dataset was relatively small. Importantly, many of the user-generated non-executable instructions were more nuanced and semantically ambiguous than those in the synthetic benchmark. Despite this, the results still indicate a reasonable level of robustness in CAMILA’s ability to identify and filter real-world ambiguous instructions. We believe that incorporating such human-generated data into the training dataset could further improve performance, and we sincerely thank the reviewer for this valuable and constructive suggestion.

---

[Question 3] The Ablation of the MLLM Choice

We appreciate the reviewer’s insightful suggestion regarding the choice of MLLM. We selected LLaVA-7B as the backbone for our experiments primarily to ensure fair comparison with MLLM-based image editing models such as MGIE, which also solely rely on LLaVA-7B. Currently, due to limited computational resources, we have not yet conducted experiments using larger MLLM models (e.g., LLaVA-13B). If feasible within the discussion period, we will report preliminary results. Otherwise, we plan to include such experiments as part of future ablation studies.

That said, we would like to clarify the role of the MLLM within our framework. Its primary function is to generate [MASK] and [NEG] tokens, which guide the editing process for the diffusion model. Since our pipeline does not fine-tune the diffusion model and relies on the accuracy of these token predictions, the effectiveness of CAMILA largely depends on how well the MLLM generates [MASK] /[NEG] tokens. In our current experiments, CAMILA achieves a token classification accuracy of 90.21%, indicating that the model already performs robustly in identifying infeasible instructions. While it remains an open question whether scaling the MLLM would yield further gains, our results suggest that the current configuration provides a strong foundation.

---

References

[1] Kirillov, Alexander, et al. "Segment anything." Proceedings of the IEEE/CVF international conference on computer vision. 2023.