MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

---

W1: The training dataset size seems to be small, which might not be enough for mend/ serac that needs training. This could potentially lead to lower performance of such methods.

On the one hand, we aim to demonstrate that our benchmark is comparable in magnitude to existing benchmarks, such as VLKEB and MIKE. On the other hand, to illustrate the impact of training data, we have plotted the training loss of SERAC and MEND in the appendix (Figures 13–14, Lines 1355–1399) of the revised manuscript. As shown, the loss consistently decreases during the training process and eventually converges to a small value, indicating that the training data is sufficient for effectively training the model.

---

W2: There is potential to enhance the T-Loc test.

As for T-LOC, some methods, such as FT-Alignment, achieve nearly 100% accuracy, which can be attributed to the nature of their editing approach. Specifically, FT-Alignment updates only the projection model between the vision and language models. Consequently, for a given text-only question, the LLM's output remains almost identical before and after editing, resulting in a T-LOC close to 100%. Furthermore, to better understand the impact of question complexity on T-LOC, we generated more challenging T-LOC questions using the LLM on visual semantic editing. We provided the original knowledge (e.g., "This is a fist in a life gesture. The thumb and pinky finger are extended while the other fingers are clenched into a fist. It signifies strength or readiness to strike.") and item type (e.g., "life gesture") to the LLM, prompting it to generate type-related questions (e.g., "What does it mean when someone waves their hand up and down enthusiastically?"). We conducted experiments using visual semantic editing data, and the results are summarized in this table. As shown, the performance of most editing methods decreases with harder T-LOC questions, whereas FT-Alignment and SERAC remain nearly unaffected. This further demonstrates that both methods can consistently achieve high T-LOC across different question types.

---

W3: The division of the three tasks lacks clarity.

We have included a figure in the appendix (Figure 12, Line 1322) of the revised manuscript to illustrate the differences between visual entity editing and semantic editing. In summary, visual entity editing focuses on entity recognition, specifically identifying who or what the entity is, whereas semantic editing emphasizes understanding semantic behaviors, such as gestures, emotions, and actions, without depending on the identity of the entity. Additionally, user-specific editing is designed to incorporate user-related information into the models.

---

W4: Additionally, I question about if this user-specific editing approach is practical.

We acknowledge that adopting an external memory or database is a viable approach to achieving user-specific applications. However, we emphasize that the need for a user-specific scenario remains realistic regardless of the method employed. Moreover, as the volume of user-specified data increases, the storage requirements expand, and the retrieval performance from memory diminishes, potentially degrading performance in real-world applications. Therefore, we believe that both adopting external memory and injecting user-specific knowledge into the model are effective solutions for building personalized models in real-world scenarios, each with its own advantages and limitations. Consequently, we propose exploring knowledge editing as an alternative approach to achieve this objective. Finally, it is worth noting that some knowledge editing methods, such as SERAC, also utilize memory-based strategies, which are already incorporated into the benchmark evaluation.

Thank you for your response. While you have clarified some of my questions, I still have doubts about task generalization. You mentioned the image captioning task; however, in Fig. 6, the example provided is a QA task, where a question is asked, and an answer is given. This is distinct from image captioning, which focuses solely on describing the image.

In the new examples you provided, the input images are identical to the editing image, and the output texts are the same as the editing knowledge. While these examples demonstrate reliability, they do not necessarily support task generalization.

If the benchmark is primarily designed as a QA task, it may not be appropriate to claim task generalization—especially since the examples given are limited and lack a comprehensive convincing evaluation of image captioning. Those examples can serve as case studies, and there may be no need to discuss such task generalization.

P.S. Your revision introduces typos in line 459: "MimiGPT4." It might be worthwhile to check for other issues elsewhere.

Dear Reviewer PUQh,

We sincerely appreciate your time and effort in reviewing this work. We understand that your schedule may be quite busy. As the authors-reviewer discussion phase draws close, we kindly request your attention to our responses. We aim to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss. We also hope that if you are satisfied with our answers, you may consider adjusting your score accordingly. We look forward to the opportunity to discuss this further with you. Thank you for your thoughtful consideration.

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

---

W1&Q1: There is some overlap between visual entity editing and visual semantic editing, as both tasks involve understanding image content, which could blur the distinction between these editing types.

We have included a new figure in the appendix (Figure 12, Line 1322) of the revision to illustrate the differences between visual entity editing and semantic editing. In summary, visual entity editing focuses on entity recognition, specifically identifying who or what the entity is, while semantic editing centers on understanding semantic behavior, such as gestures, emotions, and actions, without relying on the identification of the entity.

---

W2&Q1: Additionally, the user-specific editing scenario may lack practicality. In real-world applications, database or memory-based search might be more effective than training user-specific information for each user to achieve personalization in LLMs or LMMs.

We agree that adopting an external memory or database represents another viable approach to achieving user-specific applications. However, we would like to emphasize that the need for user-specific scenarios remains both realistic and essential for users, irrespective of the specific method employed. Furthermore, as the amount of user-specified data increases, the required storage space would grow, and the retrieval performance from memory would likely degrade, potentially negatively impacting performance in real-world applications. Therefore, we believe that both adopting additional memory and directly injecting user-specific knowledge into the model are viable solutions for achieving personalized models in real-world scenarios, with each approach having its respective strengths and weaknesses. In light of this, we propose exploring knowledge editing as an alternative method for personalization. Finally, it is worth noting that some knowledge editing methods, such as SERAC, also integrate memory-based strategies, meaning that memory-based approaches have already been partially considered in the benchmark evaluations.

---

W3&Q1: Regarding the T-Loc test, there’s room for improvement. The results are near 100, suggesting that the randomly sampled T-Loc questions are relatively easy for methods such as SERAC, FT-Alignment, and MEND. Introducing more challenging cases could enhance the evaluation’s robustness. For instance, collect similar but harder test cases, either through web crawling or using LLM-generated content (e.g., from GPT), and verify them. This could improve the test’s effectiveness.

Regarding T-LOC, methods such as FT-Alignment achieve nearly 100%, which is attributed to their editing mechanisms. Specifically, FT-Alignment only updates the projection model between the vision and language models, without modifying the LLM itself. As a result, for a given text-only question, the outputs generated by the LLM before and after editing are nearly identical, leading to a T-LOC close to 100%. To better understand the impact of questions on T-LOC, we generated harder T-LOC questions using the LLM for visual semantic editing. We provided the original knowledge (e.g., "This is a fist in a life gesture. The thumb and pinky finger are extended while the other fingers are clenched into a fist. It signifies strength or readiness to strike.") and item type (e.g., "life gesture") to the LLM and instructed it to generate type-related questions (e.g., "What does it mean when someone waves their hand up and down enthusiastically?"). Experiments were conducted on visual semantic editing data, and the results are shown in the table. As the results indicate, most editing methods perform worse under harder T-LOC questions, while FT-Alignment and SERAC remain nearly unaffected. This demonstrates that both methods can achieve consistently high T-LOC performance across different types of questions.

Dear Reviewer Bskj,

We sincerely appreciate your time and effort in reviewing this work. We understand that your schedule may be quite busy. As the authors-reviewer discussion phase draws close, we kindly request your attention to our responses. We aim to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss. We also hope that if you are satisfied with our answers, you may consider adjusting your score accordingly. We look forward to the opportunity to discuss this further with you. Thank you for your thoughtful consideration.

Dear reviewer Bskj,

Thanks for your reviews of our work. As the rebuttal deadline is apporaching, we would like to discuss the content with you further and hope to receive your response to the manuscript.

Additionally, if you find that the overall quality of the manuscript has improved after revision, we kindly ask you to consider adjusting the rating score accordingly.

Looking forward to your feedback, thank you!

Best wishes.

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

W1: Figures 1 and 2 Could Benefit from Improved Clarity and Accessibility

We have improved Figure 1 and Figure 2 in the revised version to help readers better understand our work:

• Figure 1: The example of a soccer referee gesture has been replaced with a more relatable example of a common gesture from everyday life, making it more familiar to most readers.
• Figure 2: The original figure has been updated by adding four bounding boxes and serial numbers to clarify the data generation steps. Additionally, typos have been corrected to ensure greater accuracy and clarity.

---

W2: Limitations of Counterfactual Data for Real-World Applications

We acknowledge that counterfactual data may differ in distribution from real-world data. However, collecting real-world data, particularly for capturing genuine changes in visual semantic editing, is challenging. As a result, we adopt the widely used counterfactual data generation strategy in LLM and LMM research [1–3] to construct our benchmark.

Regarding experiments on real-world data, to the best of our knowledge, there are currently no datasets or benchmarks based on real-world data specifically designed for multimodal knowledge editing. This lack of resources makes it infeasible to conduct such experiments at this time.

That said, we believe that collecting real-world data for multimodal knowledge editing is a valuable research direction and are considering exploring this avenue in the future.

---

W3: Lack of Empirical Analysis on the Impact of Human Verification

Human verification is integrated into several steps of the benchmark construction process, including candidate item generation, image filtering, counterfactual editing verification, evaluation question generation (remove or rewrite), and overall quality checks.

Among these steps, image filtering requires significant human effort, as images downloaded from Google often include noisy or irrelevant content, and some images must be manually extracted from videos. After this human filtering process, the overall quality of the images is notably improved.

For counterfactual editing verification, we found that the majority of the generated counterfactual content by LLM based on original knowledge was of high quality, with a success rate of nearly 100%. Consequently, no additional human verification was required for this step.

Regarding evaluation question generation, we used LLMs to generate questions, followed by human verification. Compared to counterfactual editing, the quality of LLM-generated questions was lower. To provide a quantitative comparison, we evaluated 100 questions generated by the LLM. Among them, 81 questions and their corresponding answers were deemed acceptable, achieving a success rate of 81%. After human verification and rewriting, noisy or incorrect questions were either corrected or removed, further improving the overall quality.

---

Q1: Would it be more beneficial to construct knowledge editing samples using incremental, real-world updates?

We agree that constructing real-world editing data for evaluation is a valuable research direction, and we are considering exploring this as part of our ongoing work.

---

Q2: What impact did human verification have on the LLM-generated descriptions?

We have explained the impact of human verification in W3, where both qualitative analysis and quantitative comparisons are provided. We believe this step significantly enhances the quality of our benchmark.

---

Q3: Is there a comparison of Visual Entity Editing data quality between MMKE-Bench and prior datasets (e.g., MC-MKE, VLKEB)?

As explained earlier, the quality of our benchmark is ensured through automated generation steps, which are complemented by extensive human verification, particularly for visual entity editing data.

In terms of comparison, the data collection and question generation steps are similar to those in previous benchmarks. For entity and image collection, we first gather entity items from existing datasets, retrieve corresponding images from Google, and then filter out noisy images through human verification. In contrast, MC-MKE requires additional subject recognition in a sentence. The quality of data collection is primarily enhanced through human verification, with significant human effort invested to ensure high image quality.

Regarding knowledge representation, both MC-MKE and VLKEB use a triplet knowledge format, while our benchmark adopts a natural language-based knowledge representation. Descriptions are sourced from Wikipedia to ensure the quality of the original content, followed by counterfactual editing performed by LLMs.

We believe that both the previous benchmarks and our own achieve a high level of quality.

Dear Reviewer 5HUG,

We sincerely appreciate your time and effort in reviewing this work. We understand that your schedule may be quite busy. As the authors-reviewer discussion phase draws close, we kindly request your attention to our responses. We aim to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss. We also hope that if you are satisfied with our answers, you may consider adjusting your score accordingly. We look forward to the opportunity to discuss this further with you. Thank you for your thoughtful consideration.

Dear Reviewer 5HUG,

Thank you for your feedback. Your insights have been invaluable in enhancing our manuscript.

If there are any remaining concerns or questions that have not yet been fully resolved, please let us know, and we would be happy to clarify further.

We appreciate your time and effort in reviewing our paper again.

Best regards

---

Q2: The case study in Section 5.3 could include more analysis. Figure 5 and Figure 6 only present a relatively simple entity editing example (person-related information) and do not showcase complex cases of visual semantic editing or user-specific knowledge editing. It’s recommended to select representative and challenging cases or specific types of cases in visual semantic editing or user-specific knowledge editing.

Firstly, there appears to be a misunderstanding regarding Figure 6. While it is person-related, its primary purpose is to illustrate a case of visual semantic editing. Specifically, the focus is on understanding the meaning of the gesture performed by the basketball referee (visual semantic understanding) rather than identifying the individual performing it (entity recognition).

Secondly, based on your feedback and the suggestion from the reviewer PUQh, we have changed "task generalization" in Fig 6 to "case study of reliability" as they align more with the VQA question format. Besides, we have included an additional case study on editing examples (Figure 27–38, lines 1579–1766) and a case study of reliability examples (Figure 39–44, lines 1760–1830) in the appendix of the revised version to provide clearer visualizations and enhance the overall understanding.

---

Q3: It is suggested to provide a detailed analysis of how IKE's performance on this new benchmark differs from its performance on previous benchmarks, and what these differences reveal about the nature of multimodal knowledge editing tasks.

We compare IKE's performance with existing benchmarks, such as VLKEB, as follows:

• The performance of editing methods on our benchmark aligns with results observed in existing benchmarks, where IKE achieves the best or second-best results in terms of reliability and generality. However, IKE demonstrates significant improvements over FT-LLM in terms of portability for VLKEB across all models (BLIP2, MiniGPT4, and LLaVA). In contrast, on our benchmark, IKE shows limited improvement or even underperforms compared to FT-LLM on BLIP2 and MiniGPT4 but achieves notable improvement on LLaVA 1.5. We attribute this to the complexity of understanding longer natural language descriptions. Modern models, such as LLaVA 1.5, benefit from better training strategies, enabling them to handle such descriptions more effectively, resulting in superior performance. Besides, The results indicate that IKE performs better in complex visual semantic and user-specific editing scenarios in terms of portability, highlighting the challenges in transferring semantic and user-specific editing tasks.

• In-context examples play a critical role in enabling models to understand the editing tasks in IKE. Unlike triple-based knowledge formats, the natural language format adopted by our benchmark requires longer input lengths. Consequently, the number of in-context examples in our benchmark is smaller compared to other benchmarks when the maximum input length for the base model is fixed. This distinction further differentiates our benchmark from existing ones.

---

Q4: Could you provide some examples or data to demonstrate that your benchmark is indeed more challenging and more valuable for improving models compared to other benchmarks?

We have discussed the differences in both the Introduction and Experiment sections. To better illustrate the comparison, we have included additional data examples from our benchmark in the appendix (Figure 15–26, lines 1404–1560) of the revised version and provided further discussion in W2.

---

References

[1] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. Findings of EMNLP, 2023.

[2] Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu Zhang. Can we edit multimodal large language models? In EMNLP, pp. 13877–13888, 2023.

[3] Han Huang, Haitian Zhong, Tao Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. Vlkeb: A large vision-language model knowledge editing benchmark. NeurIPS, 2024.

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

---

W1: I’m not sure if the workload is sufficient.

Firstly, we have selected the Primary Area of Datasets and Benchmarks for our submission to ICLR 2025, aligning with the scope and focus of our work. Therefore, we believe our contribution is well-suited for consideration at the conference.

Secondly, beyond introducing a novel benchmark, we also propose an innovative pipeline for benchmark construction, which we hope will inspire further research in the field. Our findings also underscore the limitations of existing knowledge editing methods, highlighting the need for more advanced approaches to multi-modal knowledge editing.

Thirdly, we discuss the experiment results as follows. A major difference between our benchmark and existing ones lies in that the training and evaluation questions are the same as the evaluation question (reliability) as there are multiple editing facts in one sample, However, in our benchmark, the training and evaluation questions differ. Thus, the editing method on our benchmark is more challenging. Furthermore, the evaluation task in our benchmark emphasizes semantic understanding, which is inherently more difficult than the entity recognition focus of the previous benchmarks. Moreover, the evaluation tasks in our benchmark emphasize semantic understanding, which is inherently more complex than the entity recognition tasks emphasized in previous benchmarks. This distinction further underscores the unique contributions of our work.

---

W2: I didn’t find evidence in the experimental or case study sections demonstrating the necessity of this benchmark.

The case studies in our experiments primarily demonstrate the effectiveness of knowledge editing methods. To better highlight the ‘differences’ and ‘necessity’ of our benchmark, we have added additional data examples in the appendix (Figure 15–26, lines 1404–1560). Based on these examples, the ‘differences’ and ‘necessity’ can be summarized as follows:

• 1. Necessity in real-world scenarios: Taking daily hand gestures as an example, the same gesture may convey friendliness in one country but appear unfriendly in another. Therefore, it is crucial to edit knowledge about the meanings of specific hand gestures to avoid cultural conflicts. Similarly, when entering an unfamiliar country or region, we may encounter hand gestures that are unfamiliar to us. Recognizing and adopting these gestures can facilitate better communication. Both scenarios underscore the need for editing methods to improve the recognition and understanding of hand gestures, aligning with the motivation for visual semantic editing.
• 2. Differences from other benchmarks: Previous multi-modal benchmarks primarily focus on entity-level knowledge editing using a triple-based knowledge representation format. In contrast, our benchmark evaluates complex visual semantic scenarios with a natural language-based knowledge representation. In summary, the key differences between our benchmark and existing ones are: 1) Focus on complex visual semantic editing rather than simple entity-level editing. 2) Adoption of a flexible natural language-based knowledge representation format, enabling richer and more context-aware editing capabilities.

---

Q1: In line 776, “counterfactual editing” is mentioned. According to the description in lines 776-781, modifying certain facts constitutes counterfactual editing. Is this definition accurate?

Your understanding is correct. The counterfactual editing strategy, or the generation of counterfactual editing descriptions, is widely adopted in the generation of knowledge editing datasets for LLM- and LMM-based benchmarks [1–3]. This approach involves modifying factual knowledge into counterfactual content to construct editing datasets. Similarly, in our benchmark, we employ this strategy by editing specific facts into counterfactual content, aligning with the concept of counterfactual editing.

Dear Reviewer 1STo,

We sincerely appreciate your time and effort in reviewing this work. We understand that your schedule may be quite busy. As the authors-reviewer discussion phase draws close, we kindly request your attention to our responses. We aim to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss. We also hope that if you are satisfied with our answers, you may consider adjusting your score accordingly. We look forward to the opportunity to discuss this further with you. Thank you for your thoughtful consideration.

Dear Reviewer 1STo,

Thank you for your thoughtful response. We are pleased to hear that all of your previous concerns have been addressed.

Regarding your further question, we would like to clarify the following points. Firstly, there seems to be some confusion in our previous response. What we intended to convey is that:

1. In the previous benchmark, the question in the training data (i.e., the editing knowledge in one example) is the same as the evaluation question (i.e., the reliability question), as there is only one editing fact per example.

2. In contrast, in our benchmark, the training data question differs from the evaluation question (i.e., the reliability question) because there are multiple edited facts within a single example, due to the long, free-form natural language format.

Besides, we have provided a detailed example that appears in Figure 1 below for clarity.

Previous Benchmark:

Training data:

Original Knowledge: {Q: Who is the person in the image? A: Wayne Rooney}

Editing Knowledge: {Q: Who is the person in the image? A: Zlatan Ibrahimović}

(Where bold indicates the edited part.)

Evaluation data:

Reliability question: {Rel: Who is the person in the image? A: Zlatan Ibrahimović}

As we can see, the training question is identical to the reliability question.

Our Benchmark:

Training data:

Original Knowledge: {Q: Give me some important information about the person in the image. A: The person in the image is Zlatan Ibrahimović. He is a Swedish former professional footballer. He began his career at Malmö FF in 1999 and rejoined Milan in 2020.}

Editing Knowledge: {Q: Give me some important information about the person in the image. A: The person in the image is Zlatan Ibrahimović. He is an Italian former professional footballer. He began his career at Roma in 1999 and rejoined Real Madrid in 2020.}

(Where bold indicates the edited part.)

Evaluation data:

Reliability: {Rel: Which club did the person in the image rejoin in 2020? A: Real Madrid}

In this case, the training question is generated using a template we created. For visual entity editing, the template is “Give me some important information about the {type} in the image”, where {type} refers to the entity type derived from the Wikipedia API. As we can see, the training question differs from the reliability question, as there are multiple edited facts in a single example.

Additionally, as you pointed out, the overall evaluation criteria are the same as those in the previous benchmark, with a slight difference. T-Gen, which was tested in the previous benchmark, is not tested in MMKE-Bench. This is because it is unnecessary for this particular evaluation. As we explained earlier, in the previous benchmark, the reliability question is identical to the training question, thus, a text generalization question that conveys the same meaning as the original question but with different phrasing is required. However, in our benchmark, the training question is distinct from the reliability question, making the rewriting of the reliability question redundant.

We hope this response clarifies the issue. If your concerns have been addressed, we kindly request that you consider adjusting your rating based on the changes made.

Best regards.

I have carefully reviewed your revised version, including both the main text and appendix. I deeply appreciate the detailed explanation of visual semantic editing examples in Figure 1 and the comprehensive examples in Figures 15-44 in the appendix, as these have greatly helped me better understand your work. Your responses have addressed all my concerns.

Additionally, I have a minor question regarding the response to W1, where it states: "Thirdly, we discuss the experiment results as follows. A major difference between our benchmark and existing ones lies in that the training and evaluation questions are the same as the evaluation question (reliability) as there are multiple editing facts in one sample, However, in our benchmark, the training and evaluation questions differ." Could you elaborate on this point? I don't fully understand this distinction, since other benchmarks also focus on reliability, generalization, and portability?