M2Edit: Locate and Edit Multi-Granularity Knowledge in Multimodal Large Language Model

Thank you for providing valuable feedback on our paper. Below, we have addressed your comments in detail and hope our responses sufficiently clarify the points you raised. Should you have any further questions or concerns, please do not hesitate to let us know. (please note that we break our response into two parts due to space constraints)

Q1: Exploring broader multimodal knowledge editing tasks.

Knowledge itself is modality-agnostic[6], with text and images serving as different representations of the same information. Currently, the construction of multimodal large language models (MLLMs)[5] predominantly centers on large language models (LLMs), with additional encoders (e.g., visual encoders) integrated through representation alignment. As a result, most MLLMs produce textual outputs, and existing definitions of knowledge editing tasks in these models adhere to this paradigm. In contrast, image editing[7] constitutes a separate research direction, focusing on editing visual concepts in generated images, which targets different types of models. Expanding to broader knowledge editing tasks, as you suggest, would require further development of specific application scenarios and datasets to advance this line of research.

---

Q2: Definitions of entity-level, relation-level, and event-level knowledge.

In constructing knowledge graphs, knowledge is typically categorized into three types[1] :

• Entity knowledge: Refers to information about specific objects in the real or conceptual world. These objects can be tangible, like "Apple Inc." or "Mount Huangshan," or abstract, like "love" or "economics." Entities can be represented in textual or visual form. For instance, as illustrated in the left panel of Figure 2, the concept of "capybara" can be represented either as the word Capybara or as its corresponding image. This panel essentially represents the triplet (Image of capybara, is a, capybara).
• Relational knowledge: Represents the semantic relationships between entities, such as interactions and associations. For example, the middle panel of Figure 2 depicts a locational relationship, representing the triplet (Image of Arcadia, State, California).
• Event knowledge: Concerns activities involving one or more participants (agents) in a specific spatiotemporal context, centered around a particular theme. For instance, the right panel of Figure 2 demonstrates the concept of the action "Running," representing its structural relationships with the associated "agent" and "place".

---

Q3: Addressing the integration of all three types of knowledge.

Figure 1 illustrates a real-world scenario encompassing all three types of knowledge. In our experiments, we observed that different types of knowledge are stored in different regions of the model. While unified editing approaches achieve some success, they lack sufficient performance in multimodal generalization and overall efficacy. Therefore, we believe targeted editing is essential. Developing datasets that closely resemble real-world scenarios remains challenging. Ensuring that a single query encompasses all three types of knowledge, that none of these types are pre-existing in the MLLM, and that test cases consistently evaluate the model's edited capabilities is a complex task. Nevertheless, our proposed task addresses gaps in prior work by exploring knowledge across multiple granularities.

---

Q4: Clarifications on the data annotation process and ChatGPT's involvement.

As stated in Section 2.2, the M2Edit dataset primarily builds on existing datasets such as Oven[2], FB15k-237-IMG[3], and ImSitu[4], which have been validated for data quality in prior studies. To suit our needs, we constructed the dataset through automated methods combined with manual filtering. To enhance question diversity, we used ChatGPT to generate questions, but all generated questions were manually reviewed and filtered to ensure data quality. In future versions of the paper, we will include a diagram of the annotation process in the appendix for greater clarity.

---

Referrence

[1] Entity, Relation, and Event Extraction with Contextualized Span Representations. EMNLP 2019.

[2] Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. ICCV 2023.

[3] MMKG: multi-modal knowledge graphs. ESWC 2019.

[4] Situation recognition: Visual semantic role labeling for image understanding. CVPR 2016.

[5] A Survey on Multimodal Large Language Models for Autonomous Driving. WACVW 2024.

[6] Multi-Modal Knowledge Graph Construction and Application: A Survey. IEEE Trans. Knowl. Data Eng.

[7] Diffusion Model-Based Image Editing: A Survey. Arxiv 2024.

Q5: Difficulties in understanding Figure 2.

We apologize for any confusion regarding Figure 2. To address this, we have provided more detailed explanations in General Response Q1, which we invite you to review for further clarification.

---

Q6: Explanation of Figure 3 and the meaning of s, r, and t.

In Figure 3 (lines 216-235), the top-left section illustrates the architecture of an MLLM[8,9,10,11], which consists of three main components: the Visual Encoder, Multimodal Interface, and Large Language Model. The bottom-left parallelogram sequences represent the layers of these components. Variables $s$, $r$, and $t$ denote the Key Knowledge Layers, as referenced in Equation 6 (lines 247-249), which are the layers most relevant to the outputs for the corresponding knowledge types. These layers are the focus of our editing process, as illustrated in the bottom-right editing diagram. The top-right section outlines four evaluation dimensions for MLLM knowledge editing, as defined in Equations (1)-(4) in our paper.

---

Q7: The absence of causality handling in multi-granularity knowledge editing and the necessity of a locate-then-edit approach for multimodal large language models (MLLMs).

As discussed in the Introduction of our paper, existing knowledge editing methods[12,13] do not simultaneously address editing across different components of multimodal large language models. This limitation reduces the models' generalization ability after editing (see Table 2, lines 324–347). Our approach, in contrast, edits three distinct components simultaneously. Furthermore, during the localization process, we identified that different types of knowledge are stored in different regions of the model (see Figure 4, lines 378–389). For this reason, we believe that localization is essential for effective knowledge editing, which experimental results have also verified.

---

Q8: Validation limited to older MLLMs (e.g., BLIP2-OPT and MiniGPT4), which may not represent state-of-the-art models.

Thank you for this valuable suggestion. In response, we conducted additional experiments with LLaVa-7B [11]. The results are presented below:

$$$$

\hline Method &Entity & Relation & Action \\ \hline FT &33.5& 27.0& 30.2\\ MEND & 82.3 &70.6&77.8 \\ ROME &71.2& 52.5 & 78.3 \\ \hline MLE & 86.4 & 75.9 & 86.0
\end{array}$$

This table summarizes the comprehensive editing results of different methods applied to LLaVa-7B [11] on our dataset. It demonstrates that our method continues to exhibit superior generalization performance.

---

Q9: Suggestions to enhance experimental analysis.

Thank you for your insightful suggestion. In our current paper, we have already shown that our method outperforms baseline models in both comprehensive performance and multimodal generalization across various types of knowledge. Additionally, we have analyzed the distribution of different knowledge types within the model, which supports the superiority of our approach. To further strengthen our work, we have added the results of batch editing experiments (see General Response Q1) and plan to include a case study analysis in the appendix of the updated version.

---

Q10: Missing anonymous links to access model code and data examples.

We appreciate this suggestion. During the initial submission, we provided data examples. We have now updated the supplementary material to include the MLE code in the attachments.

---

Q11: The meaning of in-scope editing.

In-scope editing refers to the scope of content that should be modified during a single editing operation. This concept was introduced in [reference]. For in-scope content, all relevant elements must be modified, while out-of-scope content should remain unaffected. Detailed examples of in-scope and out-of-scope editing are provided in General Response Q1.

---

Q12: Grammatical and punctuation errors.

Thank you for pointing out the grammatical and punctuation issues. We sincerely appreciate your careful review and constructive feedback. We will address these issues thoroughly to meet ICLR standards.

---

Referrence

[8] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. NeurIPS 2023.

[9] Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2023.

[10] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. ICLR 2024.

[11] Improved Baselines with Visual Instruction Tuning. CVPR 2024.

[12] Can We Edit Multimodal Large Language Models? EMNLP 2023.

[13] MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing. ACL 2024.

Q4: Provide more details about the annotation process.

Details regarding the annotation process were already present in the original manuscript. To improve clarity, we have added further explanations in the appendix (Lines 748–791) and introduced a new Figure 7 (Line 756). We hope these additions address your concerns.

---

Q5: Update image content.

Thank you for pointing this out. We have updated all images in the manuscript to clearer vector graphics, adhering to your suggestion.

---

Q6: The necessity of locate-then-edit for multi-granularity knowledge editing.

We would like to clarify the rationale behind our proposed approach. While our locate-then-edit strategy is a means to achieve multi-granularity knowledge editing, it is not the only possible method. The key contributions of our work, which distinguish it from prior studies, are as follows:

1. Editing across multiple components: Unlike prior methods that edit a single component of an MLLM, our approach edits the visual encoder, multimodal interface, and large language model collaboratively, resulting in superior generalization capabilities (See Figure 4).

2. Feasibility through localization: Our approach is grounded in parameter adjustment for specific layers (following the methodology of [3]). Without precise localization, global fine-tuning would result in excessive interference with unrelated knowledge.

3. Knowledge distribution analysis: Our analysis identifies the uneven distribution of knowledge across components, substantiating the necessity of targeted edits (See Figure 5 and Lines 378–403).

---

Q7 & Q8: Experimental limitations.

Despite limited resources, we have supplemented our experiments with:

1. Batch editing results. (paper line 432-446 and line 459-466)
2. Performance on different sizes of LLaVa models.

Previous studies ([1,2,3]), regardless of modality, typically evaluated medium-scale models. To ensure rigor, we extended our evaluations to various LLaVa configurations (Edit Score), with results summarized below:

$$$$

\hline Model & Method &Entity & Relation & Action \\ \hline **LLaVa 1.5 7B ** & \text{FT} &33.5 & 27.0 & 30.2\\ & \text{MEND} & 82.3 & 70.6 & 77.8 \\ & \text{ROME} &71.2 & 52.5 & 78.3 \\ & MLE & 86.4 & 75.9 & 86.0 \\ \hline **LLaVa 1.6 7B ** & \text{FT} &18.7 & 15.2 & 24.3\\ & \text{MEND} & 79.5 & 63.3 & 81.0 \\ & \text{ROME} &75.1 & 61.8 & 76.4 \\ & MLE & 81.2 & 70.2 & 83.6 \\ \hline **LLaVa 1.6 13B ** & \text{FT} &42.5 & 36.2 & 43.6\\ & \text{MEND} & 85.2 & 78.9 & 84.7 \\ & \text{ROME} &81.5 & 72.4 & 84.6 \\ & MLE & 89.2 & 79.9 & 86.8 \\ \hline **LLaVa 1.6 34B ** & \text{FT} &53.2 & 37.7 & 44.5\\ & \text{MEND} & 87.0 & 79.2 & 85.3 \\ & \text{ROME} &82.2 & 74.4 & 85.0 \\ & MLE & 88.4 & 79.5 & 85.9 \\ \hline \end{array}$$

The Edit Score was computed as:

$$$$

Edit Score = \frac{4}{\frac{1}{\mathbf{O}^{rel}} + \frac{1}{\mathbf{O}^{gen} _ v} + \frac{1}{\mathbf{O}^{gen} _ t} + \frac{1}{\mathbf{O}^{loc}}}.

$$> These results demonstrate the robustness of our method across diverse multimodal models. --- In conclusion, we believe our current experiments and settings adequately demonstrate the effectiveness of our method. We will strive to incorporate further analyses and insights based on your feedback in future revisions. Thank you for your invaluable comments! ---- **References** [1] Can we edit multimodal large language models? EMNLP 2023 [2] MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing. ACL 2024. [3] MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency. Arxiv 2024.$$

We appreciate very much your constructive comments on our paper. Please kindly find our response to your comments below. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.

Q1: The proposed method is evaluated on a limited range of multimodal models.

• In line with the methodologies employed in previous studies [1], which focused on Mini-GPT4 and BLIP-2, we have maintained their experimental framework to uphold the integrity of our research. To substantiate the efficacy of our approach, we have extended our investigation with further experiments:

• Evaluation of the Edit Score for LLaVA-1.5 7B [2,3]:

$$$$

\hline Method &Entity & Relation & Action \\ \hline FT &33.5 & 27.0 & 30.2\\ MEND & 82.3 & 70.6 & 77.8 \\ ROME &71.2 & 52.5 & 78.3 \\ \hline MLE & 86.4 & 75.9 & 86.0
\end{array} $$

The results clearly demonstrate the superior performance of our method when applied to LLaVA-1.5 [2,3].

---

References

[1] Exploring Edits in Multimodal Large Language Models. EMNLP 2023.

[2] Enhancing Visual Instruction Tuning. NeurIPS 2023.

[3]Improved Baselines with Visual Instruction Tuning. CVPR 2024.

Thank you for your thoughtful feedback. Below, we provide detailed responses to each of your questions and concerns. (please note that we break our response into two parts due to space constraints)

---

Q1: Lack of discussion on the complexity of the method.

In prior work on knowledge editing, the complexity of such methods has seldom been emphasized. This is likely because in real-world scenarios, frequent and large-scale editing is uncommon. Additionally, editing model parameters through knowledge editing involves significantly fewer parameters than fine-tuning, making it much more efficient—our method is over 5x faster than fine-tuning. Specifically, editing 500 samples from our dataset takes only 1/50th the time required for fine-tuning.

Although our method involves clustering and similarity calculations, clustering is performed only once beforehand. During testing, the time spent on similarity calculations is negligible compared to the time required to adjust model parameters. Below is a table showing the time required for editing 500 M2Edit samples on BLIP2-OPT 7B [1] using NVIDIA GeForce RTX 3090 GPUs:

$$$$

\begin{array}{lc}\hlineMethod &Time(s)\\ \hline FT & 1617.5 \\ROME & 13.1 \\\hlineMLE & 29.2 \end{array}

------ ## Q2: ```Lack of discussion on failure cases.``` > Due to the limitations of displaying image content on OpenReview, we will include updated case studies in the next version of the paper. In general, traditional methods such as MEND [2] and ROME [3] struggle to generalize well to changes in images after editing a specific component. > For example, when editing entity-level knowledge about "Taco," traditional methods can generalize text queries from "What is the name of this dish?" to "What is this food called?" (T-G) and still output "Taco." However, they fail when dealing with synonymous but visually distinct images of "Taco" (V-G). In contrast, our method can correctly answer in such scenarios, demonstrating superior visual generality. ------ ## Q3: ```Only dataset was uploaded; no code provided.``` > Thank you for pointing this out. While we submitted dataset samples during the initial submission, we have now updated the supplementary material to include the MLE code in the attachments. ------ ## Q4: ```Lack of batch editing results.``` > We appreciate your suggestion. Batch editing results and analyses have been added and can be found in **General Response Q1**. ------ ## Q5: ```Lack of results on real-world datasets.``` > Thank you for the suggestion. While the two valuable references you provided are focused on text-only modalities, our method specifically targets multimodal large language models. This difference makes validation on these datasets challenging. The key challenge in multimodal knowledge editing lies in the lack of real-world, high-quality datasets for evaluation [4,5]. Should such datasets become available, we would be delighted to validate our method on them. ------ **References** [1] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML 2023. [2] The Tenth International Conference on Learning Representations. ICLR 2022. [3] Locating and Editing Factual Associations in GPT. NeurIPS 2022. [4] Can we edit multimodal large language models? EMNLP 2023. [5] MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing. ACL 2024.

Q6: Issues with grammar and clarity in figure presentation.

We apologize for the lack of clarity in Figure 2. We have provided a detailed explanation of its content in General Response Q2 and encourage you to review it.

---

Q7: Metric calculation issues.

The calculation methods for Reliability (R), Visual Generality (V-G), Text Generality (T-G), and Locality (L) are detailed in Equations (1)–(4) of our paper. Below, we briefly summarize their computation:

• Reliability (R): Measures accuracy on edited samples $(x_i, v_i, y_i)$.

$\mathbf{O}^{rel}(\hat{\Theta}) = \mathbb{E}_{(x_i,v_i,y_i) \in \mathcal{D}} [\mathbf{I}( \hat{\Theta}(x_i,v_i) = y_i) ].$

• Text Generality (T-G): Tests the edited sample with synonymous text $x_j$ paired with the original image $v_i$.

$\mathbf{O}^{gen}(\hat{\Theta}) = \mathbb{E}_{(x_i,v_i,y_i) \in \mathcal{D}} [\mathbf{I}( \hat{\Theta}(x_j,v_i) = y_i) ], s.t. ~ xj\sim xi.$

• Visual Generality (V-G): Tests the edited sample with synonymous images $v_j$ paired with the original text $x_i$.

$\mathbf{O}^{gen}(\hat{\Theta}) = \mathbb{E}_{(x_i,v_i,y_i) \in \mathcal{D}} [\mathbf{I}( \hat{\Theta}(x_i,v_j) = y_i) ], s.t. ~ vj \sim vi.$

• Locality (L): Measures the impact of editing on unrelated samples $(x_k, v_k)$.

$$$$

s.t.~(x_k,v_k) \perp (x_i,v_i). $$

We hope our responses address your concerns adequately. Please feel free to share any additional feedback or questions you may have. Thank you again for your thoughtful and constructive comments!