We are greatly encouraged by your positive comments. Thanks a lot for all the appreciation.

Q1: The main limitation of the paper is (as acknowledged by the authors) that they only use a single LLM type (LLAVA).

A1: Thank you for your rigorous review. Please refer to general response Q3.

Q2.1: How do the benchmarked methods compare in terms of compute time?

A2.1: Thank you for your valuable question. Please refer to general response Q2.

Q2.2: How do the methods scale?

A2.2: Thank you for your insightful thinking. We agree that scalability is an important factor to consider in our experiments. Our paper presents several results related to the scalability of the methods. For instance, Table 1 illustrates how model size affects scalability. In Section 6.2, lines 247-255, we analyze the performance changes when the model size increases from 7B to 13B. It can be observed that each method performs worse on a larger model. These results suggest that as model size increases, the encoded knowledge may become more stable.We also found that as the model size increases from 7B to 13B, there is a noticeable decline in the effectiveness of non-SIU methods in Generality. In contrast, our method shows a relatively minor decline in generality.
Moreover, Figures 2 and 3 analyze the scalability of training steps. Our method, SIU, shows minimal fluctuations across each metric, indicating that it is less sensitive to the number of fine-tuning steps. In contrast, other methods, such as GA and PO, display significant variability as the number of fine-tuning steps increases.
Figure 3 presents the scalability concerning the number of unlearning targets. We concatenated all the forgetting training sets for these concepts to create the fine-tuning data and set the training step to 120. We found that after unlearning, the utility of MLLMs collapses when using GA and GA+KL. In contrast, with SIU, each metric remains nearly the same as when unlearning a single concept, demonstrating the robustness of SIU. In future work, we plan to explore more experiments related to scalability. Thank you for your suggestions.

Q3: There is no discussion of societal impact.

A3: Thank you for your pertinent suggestion. Please refer to general response Q1.

We are grateful for your attentive comments and providing thoughtful feedback on our work. We will provide our insights point by point below.

Q1: It is not clear whether the incorrect answer is caused by unlearning or hallucinations.

A1: Thank you for your insightful comment. The incorrect answers observed after unlearning are intentional results of our method, which aims to minimize the model’s confidence in the forgotten concept by outputting predefined incorrect names used during training. Moreover, it's important to note that hallucinations in LLMs refer to generating content not present in the training data [1]. However, in our method, the output incorrect name post-unlearning is the same as the target name which we pre-defined during the unlearning training stage. This demonstrates that the incorrect answer is caused by unlearning rather than hallucination.

We will clarify the relationship between output targets with hallucination in the revised version.

Q2: It would be interesting to see whether the SIU keeps achieving good performance when using different multimodal LLMs.

A2: Thank you for your thoughtful question. Please refer to general response Q3.

Q3: The social impacts are expected to be discussed in the paper/appendix.

A3: Thank you for your rigorous question. Thank you for your pertinent suggestion. Please refer to general response Q1.

Q4.1: How to ensure that incorrect answers will not impact the model performance by generating hallucinated answers?

A4.1: Thank you for your thoughtful question. Ensuring that the model's performance remains unaffected is a key aspect of our benchmark and task definition. We have designed a comprehensive evaluation schema to assess the preservation of model performance post-unlearning. For instance, as stated in line 194, specificity measures the preservation of non-targeted knowledge, which is crucial for maintaining the model's utility after unlearning. We conducted extensive experiments on 8 benchmarks that test the general abilities of MLLMs in understanding and reasoning. If the performance changes are minimal, it indicates that the model's capabilities are largely preserved post-unlearning. Moreover, our evaluation schema also includes fluency and diversity, both of which represent the utility of MLLMs. As shown in Table 1, our method outperforms existing methods across these three criteria, indicating that the generation of incorrect answers (a key objective of our method) does not significantly impact the model's performance.

Q4.2: Wouldn’t it be better to say something like “The central figure in this photograph private/sensitive” instead of giving the incorrect answer?

A4.2: Thank you for your insightful suggestion. Your idea is similar to our baseline PO method, which outputs "I don't know" post-unlearning. However, there are several reasons why responses like “The central figure in this photograph is private/sensitive” or "I don't know" may not be optimal for our task: (i) Incomplete Forgetting Indication: As shown in Figure 11 and discussed in lines 616-621, MLLMs may respond with messages indicating confidentiality, such as, "I'm sorry, but I am not programmed to provide information about physical characteristics of a specific person." Such responses imply that the MLLMs still retain knowledge of the concept, potentially signaling to attackers that the model still knows but chooses not to disclose, inviting jailbreak attempts to bypass these safeguards. (ii) Task Objective Misalignment: The primary goal of machine unlearning is to remove knowledge from models, effectively making them forget certain concepts. Phrasing responses as “The central figure is private/sensitive” focus on privacy rather than knowledge removal, which misaligns with the core objectives of unlearning. (iii) Performance Impact: As shown in Table 1 and discussed in lines 621-624, the PO method, while achieving a relatively high EM score, has a low C-Dis score of 0.4, indicating that it continues to produce tokens related to unlearning targets. This suggests that the model may only be adapting to the question-and-answer format rather than genuinely forgetting the content. Moreover, the PO method is prone to overfitting, as shown in Figure 15, where multi-hop questions are repeatedly answered with "I don't know." Table 3 also demonstrates a significant decrease in specificity when unlearning multiple concepts, negatively impacting the model’s overall performance.

Q5: In Equation 1, wouldn’t the two components for the set of data to forget and not to forget have opposite signs?

A5: Thank you for your careful examination of our formula. It seems there might be a misunderstanding regarding the data in the forgetting set. You might be assuming that the forgetting set contains the original outputs of the MLLMs before unlearning. However, as stated in lines 115-116, the training data in the forgetting set is specifically designed to reflect the forgetting of unlearning targets. The training data in the forgetting set consists of our multifaceted fine-tuning data, as shown in Figures 1 and 7. During training, the objective is to minimize the negative log-likelihood of predicting the next token. Consequently, after unlearning, the MLLMs are designed to output incorrect names for the unlearning targets.

We appreciate you pointing these questions out, and we will clarify all the above aspects in the revised version. If you have any additional questions or need further clarification, please do not hesitate to reach out, and we will be happy to provide prompt responses.

References

[1] Ji, Ziwei, et al. "Survey of hallucination in natural language generation." ACM Computing Surveys 55.12 (2023): 1-38.

Thank you for your critical feedback and suggestions. We address your thoughts point by point below.

Q1: The structure of the paper needs improvement.

A1: Thank you for your suggestion. We agree that the structure of the paper can be improved. In the final version, we will improve the presentation to make the paper more readable.

Q1.1: It would be better if the authors used Figure 1 to explain as much as possible.

A1.1: Thank you for your pertinent suggestion. We acknowledge that referring to Figure 7 in the Appendix may disrupt the flow of reading. Our intention in creating Figure 7 was to provide a formal reference for specific details, such as in lines 144-145, where we refer to Figure 7 (a) and other sub-figures in Section 4.1. These detailed references are challenging to incorporate into Figure 1. However, we will integrate more relevant explanations into Figure 1 to improve clarity and cohesiveness.

Q1.2: Many parts that should be in the main text are in the Appendix. For instance, the motivation for DMK in Lines 164-165 should be mentioned in the main text.

A1.2: Thank you for your professional feedback. The motivation for DMK is briefly introduced in Section 1, lines 44-48. What we wrote before may not be comprehensive and clear enough, so we will revise it in the final version. As this is the first work in this field, we faced constraints in fitting all definitions and explanations into the main text, which led us to place additional details in the appendix. A more concise introduction to MMUBench would be included in the revised version to ensure key information is presented in the main text. We will try our best to better balance the main text and the appendix to make the paper more readable in the final version.

Q2: Whether the experimental results consider multiple seeds.

A2: Thank you for your valuable question. As stated in the caption of Table 1, for each experiment we report the means and standard deviation of 3 independent trials. Each trial is conducted with random seed. We will move this clarification to Section 6.1.

Q3: The reason or motivation for targeting the unlearning of the visual recognition of concepts rather than factual knowledge in MLLMs.

A3: Thank you for your rigorous review. The choice of unlearning targets is crucial in pioneering machine unlearning in MLLMs. Initially, we considered whether to target factual knowledge for unlearning. However, we ultimately chose visual recognition as the primary target for several reasons: (i) Extensive Prior Research: There are many prior works [1], [2], [3] have extensively explored the unlearning of factual knowledge in various contexts. Our aim was to address an underexplored area in MLLMs. (ii) Component Structure of MLLMs: MLLMs consist of a large language model (LLM), a vision encoder, and a projection layer/module. Factual knowledge, such as "Donald Trump is the former president," is embedded within the LLM itself and is largely independent of the MLLM's pre-training phase, where the LLM’s parameters remain frozen. The primary knowledge which MLLMs obtain in the pre-training phase is to recognize and align visual concepts. Consequently, our focus is on unlearning visual recognitions that may infringe upon personal privacy. (iii) Privacy and Regulatory Compliance: In scenarios such as social media, individuals may request the removal of their images while retaining their factual information in other contexts. This approach ensures privacy without compromising the model's overall knowledge base. Regulations like GDPR [4] also necessitate the "right to be forgotten," which often involves removing visual data that can uniquely identify individuals.
Here are some Real-world Examples: 1) Social Media Platforms: Users may wish to delete their images from a platform while keeping their posts and interactions. 2) Surveillance Systems: Removing specific individuals’ data from surveillance systems without affecting the recognition of other people or objects in the dataset. 3) Medical Imaging: Patients requesting the removal of their medical images for privacy reasons while keeping their medical history intact.
We will clarify the motivation for targeting the unlearning of the visual recognition of concepts rather than factual knowledge in the revised version.

Q4: There is no potential negative societal impact of this paper.

A4: Thank you for your thoughtful question. Please refer to general response Q1.

References

[1] Eldan, Ronen, and Mark Russinovich. "Who's Harry Potter? Approximate Unlearning in LLMs." arXiv preprint arXiv:2310.02238 (2023).
[2] Wang, Lingzhi, et al. "Kga: A general machine unlearning framework based on knowledge gap alignment." arXiv preprint arXiv:2305.06535 (2023).
[3] Chen, Jiaao, and Diyi Yang. "Unlearn what you want to forget: Efficient unlearning for llms." arXiv preprint arXiv:2310.20150 (2023).
[4] Voigt, Paul, and Axel Von dem Bussche. "The eu general data protection regulation (gdpr)." A Practical Guide, 1st Ed., Cham: Springer International Publishing 10.3152676 (2017): 10-5555.