Reliable Lifelong Multimodal Editing: Conflict-Aware Retrieval Meets Multi-Level Guidance

We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.

Q1: The retrieval challenges faced by existing methods.

A1: The retrieval strategies of existing methods face the following challenges:

(1) Ambiguity of text semantics: Most current methods are designed for text-only knowledge editing and rely solely on text semantics for retrieval. However, due to the inherent ambiguity of text semantics (e.g., "a photo of" vs. "a photograph of" in the E-IC task), the lack of auxiliary visual information can easily lead to retrieval errors.

(2) Limitations of multimodal fusion strategies: The first lifelong multimodal editing method, LiveEdit [1], retrieves multiple relevant experts through visual search and then fuses them using text semantic similarity weighting. However, this approach can introduce noise from irrelevant experts and cannot effectively handle direct conflicts between modalities.

[1] Chen et al. Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts, CVPR 2025.

Q2: Effectiveness of the proposed retrieval strategy.

A2: CARML's conflict-aware dynamic retrieval strategy focuses on the unique contradictions present in multimodal editing. For instance, in the Generality-type editing samples, single-modality modifications often increase the risk of conflict. By retrieving all modalities in parallel and dynamically quantifying intra-modality uncertainty and inter-modality conflict, CARML can accurately identify and suppress unreliable modality channels, significantly improving retrieval accuracy. In Figure 1(a) of our paper, we illustrate the retrieval accuracy trends of various methods under an increasing number of sequential edits. The results show that CARML consistently maintains a desirable retrieval accuracy, proving its robustness.

Q3: The advantages of the proposed method compared to its complexity.

A3: Thank you for raising this critical question. We believe that while CARML introduces a moderate level of complexity in its design, it delivers significant improvements in performance and reliability through three core advantages:

(1) Precise knowledge retrieval: As stated in our response to Q2, our retrieval mechanism overcomes many shortcomings of existing methods, ensuring the accuracy of knowledge localization.

(2) Reliable multimodal knowledge editing: Through a multi-level, coordinated editing guidance strategy, we ensure that the retrieved knowledge is correctly understood and executed by the model.

(3) Excellent performance-efficiency trade-off: Although CARML introduces multiple modules to optimize performance, they are all of a lightweight design, primarily involving vector operations and small MLPs. We have discussed the comparative analysis of runtime efficiency in Section 4.3 of our paper. Specifically, we compare the accuracy and single editing time of the different methods after 1000 edits on the E-IC dataset, as shown in Table 1. The results demonstrate that CARML achieves an ideal efficiency ratio, attaining extremely high accuracy while maintaining a low time cost.

Table 1: The trade-off between accuracy and editing time.

Q4: The advantages of the editing component for the ever-growing knowledge.

A4: Our method has the following significant advantages when dealing with the ever-growing knowledge:

(1) Immunity to catastrophic forgetting: CARML does not directly modify the core parameters of the MLLM. Instead, it guides the model through external prompts and logits enhancement. This fundamentally prevents the catastrophic forgetting problem that can arise from accumulating knowledge.

(2) Constant editing efficiency: The computational overhead of CARML's editing component is independent of the knowledge base size. For each edit, the conflict-aware dynamic retrieval module precisely locates the Top-1 relevant knowledge item and passes this single item to the subsequent editing modules. This ensures that the efficiency of the editing component remains constant even as the knowledge base grows.

Q5: The division between critical modules and incremental modules.

A5: Thank you for your valuable suggestion. Based on the ablation study in Table 4 of our paper and the additional ablation experiments shown below, we have re-evaluated the importance of the modules and categorized them as follows:

Table 2: Ablation study on core components on the E-IC dataset.

(1) Critical Modules:

● Conflict-aware dynamic retrieval: Significantly improves retrieval accuracy, leading to substantial enhancements in reliability and generality.

● Edit scope classifier: Effectively prevents modifications to irrelevant knowledge by learning reliable semantic boundaries.

● Context-aware dynamic prompt: Facilitates the effective execution of edits by providing highly customized, multimodal editing instructions.

● Output logits enhancement: Further boosts editing reliability, increasing it from 83.4% to 99.4%.

(2) Incremental Modules:

● Static knowledge prompt: Although its removal results in a slight decline in overall performance, its essential information has been efficiently integrated into the "context-aware dynamic prompt" module.

Thank you once again for your time and valuable feedback!

We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.

Q1: Lack of comparison with some recent baselines.

A1: Thank you for your suggestion. We have further compared CARML with two of the latest baselines proposed in 2025: AlphaEdit [1] and BalancEdit [2]. We conducted experiments with sequential 1000 edits based on the BLIP-2 OPT model on the three datasets. The results are shown in the Table 1, 2 and 3. CARML's performance surpasses both of these recent baselines, further demonstrating its effectiveness. And we will supplement these baselines in the revision.

Table 1: Editing results after 1000 edits on the E-IC dataset.

Table 2: Editing results after 1000 edits on the E-VQA dataset.

Table 3: Editing results after 1000 edits on the VLKEB dataset.

[1] Fang et al. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, ICLR 2025.

[2] Guo et al. BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing, ICML 2025.

Q2: The underlying theory or mechanism are not mentioned, making the proposed method heuristic.

A2: We thank the reviewer for highlighting the importance of the theoretical depth behind our method. The design of CARML is grounded in a decision-making framework that integrates multiple theories, with a controlled retrieval-augmented generation paradigm tailored for knowledge editing tasks at its core. Through non-invasive editing, CARML enhances the efficiency of multimodal knowledge editing while ensuring the stability required for lifelong learning. Specifically, its theoretical foundations include the following aspects:

(1) Information theory and ensemble learning: To address the unique challenges of multimodal editing tasks, such as increased retrieval uncertainty and inter-modal conflicts arising from unimodal modifications in generality-type editing samples, we introduce information entropy to quantify retrieval uncertainty and use the Jaccard similarity to measure modal conflicts. Concurrently, drawing inspiration from ensemble learning, we treat the multimodal channels as "experts" and employ an adaptive mechanism to evaluate and fuse the retrieval results from different modalities.

(2) Metric learning: To ensure the locality of edits, we have designed an edit scope classifier to learn a robust semantic decision boundary. This classifier effectively aligns the semantic spaces of the edit samples and the retrieved knowledge while filtering out samples that do not require editing, thereby guaranteeing the precision of the edits.

(3) Bayesian inference and prompt Learning: Our proposed explicit-implicit guidance strategy is a concrete implementation of a Bayesian framework. Within this strategy, the implicit prompt guides the generation process by introducing strong prior knowledge, while the explicit logits enhancement further optimizes the probability distribution. This synergy calibrates the generation trajectory and improves the reliability of the edits.

Grounded in these robust theoretical foundations, each component of CARML demonstrates a significant performance improvement in multimodal knowledge editing tasks, as detailed in the ablation studies in Table 4 of our paper. Furthermore, CARML possesses broad transferability potential for applications in other domains, such as fact-checking, hallucination mitigation, and efficient personalization. We will further emphasize these theoretical foundations and their significance in the revision.

Thank you once again for your time and valuable feedback!

Dear Reviewer 4PKH,

Thank you so much for your insightful and constructive review of our manuscript. We are writing to respectfully follow up on our rebuttal and to see if you have any further questions.

Following your invaluable suggestions, we have made substantial efforts to improve our paper. The key clarifications and analyses from our response are summarized as follows:

● Comparison with recent baselines: To address your concern about missing comparisons, we have conducted new experiments comparing our CARML against two very recent SOTA methods, AlphaEdit and BalancEdit. As shown in the new tables, our method demonstrates a significant performance advantage, further validating its effectiveness.

● Elaboration on theoretical foundations: In response to your point about the method appearing heuristic, we have detailed the underlying theoretical and mechanistic principles of our approach. We explained how CARML is grounded in information theory, ensemble learning, metric learning, and Bayesian inference, providing a solid theoretical basis for its design and success.

We deeply value your expertise and hope that our responses and the new results have fully addressed your concerns. We would be very grateful to know if you have any remaining questions or if there's anything that requires further clarification. If our rebuttal has successfully resolved your initial concerns, we would be deeply appreciative if you would consider re-evaluating our paper.

Many thanks for your constructive comments, time, and patience.

Best regards,

The Authors

We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.

Q1: Given the pipeline complexity, the clarity could be further improved.

A1: Thank you for your pertinent suggestion.

(1) Complexity clarification: Although CARML introduces multiple modules to optimize performance, they are all of a lightweight design, primarily involving vector operations and small MLPs, which results in extremely low computational overhead. In Section 4.3 of our paper, we illustrate the trade-off between performance and computational efficiency for CARML, showing that it achieves an excellent balance.

(2) Narrative improvement: In the revision, we will highlight the key innovations of our method in the introduction. And in the methodology section, we will further refine the descriptive logic of each module to ensure its innovative aspects are articulated with clarity and coherence.

Q2: Conduct ablation studies on only core parts to identify the absolutely necessary modules.

A2: Thank you for your valuable comments. Based on your recommendation, we have conducted an additional ablation study on the core components. The results are shown in the table below:

Table 1: Ablation study on core components on the E-IC dataset.

We can observe that the key modules that bring major performance improvements to CARML are: conflict-aware dynamic retrieval, editing scope classifier, context-aware dynamic prompt, and logits enhancement. In comparison, the contribution of the static knowledge prompt is relatively minor, as the information it carries is already encompassed by the context-aware dynamic prompt.

Q3: Regarding the innovation and advantages of conflict-aware dynamic retrieval.

A3: The innovation and advantages of the "conflict-aware dynamic retrieval" mechanism are reflected in the following aspects:

(1) Dynamic adaptivity: This mechanism dynamically allocates modality weights based on intra-modality uncertainty and inter-modality conflict, enabling precise adjustments for each editing sample.

(2) Customized for multimodal editing task: It addresses the unique challenges of multimodal editing (e.g., the increased conflict caused by single-modality modifications in Generality-type editing samples) by capturing conflicts and dynamically reducing the weights of the relevant modalities.

(3) Lightweight and efficient: The mechanism only involves vector operations and does not require computationally expensive re-rank models. Furthermore, the ablation study in Q2 further validates the value of this mechanism, where the edit reliability is reduced from 99.4% to 37.0% after its elimination.

Q4: What is the most important component in the pipeline?

A4: We believe the most important component is the multi-level knowledge guidance module, which ensures the effective execution of the knowledge edit. The implicit context-aware dynamic prompt generates highly customized, fine-grained multimodal editing instructions for the model, while the explicit logits enhancement further ensures the reliability of the editing outcome.

Q5: Regarding hyperparameter sensitivity and tuning techniques.

A5: We have analyzed the key hyperparameters:

(1) Scope classifier's threshold β: This threshold directly determines whether an editing operation is triggered and is therefore relatively sensitive, as shown in Table 2. Since the classifier has learned a stable semantic boundary, its decision threshold typically stabilizes around 0.5. We use a simple binary search method to fine-tune the value around β, selecting the optimal parameter that maximizes reliability without harming locality.

Table 2: Sensitivity of the scope classifier's threshold β on the E-IC dataset.

(2) Retrieval fusion hyperparameters: CARML exhibits low sensitivity to these parameters. For the fusion weights of uncertainty and conflict in Equation (5), experiments show that values within the range of [0.3, 0.7] achieve desirable results. For the Jaccard similarity K value, the experimental results of its sensitivity analysis are shown in Table 3. We can notice that its performance is stable for values in the range of [12, 21]. The K value that is too small may result in insufficient candidate samples, while one that is too large can introduce noise.

Table 3: The sensitivity analysis on the value of K in the Jaccard similarity.

Q6: The scalability of CARML's knowledge base in real-world deployment.

A6: Thank you for raising this practical and important question. Our CARML method possesses the following advantages for scalability:

(1) Efficient retrieval scalability: Our retrieval mechanism is built upon an efficient vector similarity search library (FAISS). Its search time complexity has a sub-linear relationship with the knowledge base size N. Therefore, CARML can be easily extended to millions of editing entries, with the primary bottleneck being storage rather than retrieval latency.

(2) Constant editing overhead: Since the conflict-aware dynamic retrieval module can precisely locate Top-1 relevant knowledge item, the amount of knowledge received in the editing phase is fixed at one, and its computational complexity is independent of the size of the knowledge base and remains constant.

(3) Immunity to catastrophic forgetting: Unlike methods that directly modify model parameters, CARML stores knowledge externally and selects relevant entries via its retrieval mechanism. This approach avoids the problem of catastrophic forgetting that can arise from frequent iterative edits, ensuring the long-term stability of the model's capabilities.

Thank you once again for your time and valuable feedback!

We sincerely thank you for your recognition of our work and your valuable time. Your insightful and constructive feedback is crucial in helping us improve our paper.

We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.

Q1: Comparison of runtime efficiency between CARML and other baselines.

A1: Thank you for raising this important question. Our proposed CARML demonstrates a superior trade-off between extremely high accuracy and low time cost. We have discussed the comparative analysis of runtime efficiency in Section 4.3 of our paper, under the part of "Trade-off between Accuracy and Editing Time." Specifically, we compare the accuracy and single editing time of the different methods after 1000 edits on the E-IC dataset, and the results are shown in Table 1.

Table 1: The trade-off between accuracy and editing time.

Q2: The MLLM models used are kind of outdated and suggest considering more recent models.

A2: Thank you for your pertinent suggestion.

(1) Reason for choosing existing models: We chose BLIP 2-OPT and LLaVA-V1.5 as the base models to ensure a fair comparison with existing knowledge editing methods, as these methods all use the same base models.

(2) Supplementary experiments: To demonstrate the robustness of CARML, we conducted experiments on a more recent model, Qwen2.5-VL-7B-Instruct, as shown in Tables 2, 3 and 4. The results show that CARML also achieves excellent editing performance and surpasses existing baseline methods, proving CARML's model-agnostic nature and broad applicability.

Table 2: Editing results after 1000 edits on the E-IC dataset.

Table 3: Editing results after 1000 edits on the E-VQA dataset.

Table 4: Editing results after 1000 edits on the VLKEB dataset.

Thank you once again for your time and valuable feedback!

Thank you for your rating. Your valuable suggestions greatly contribute to the quality of our manuscript. Thank you again for your precious time and valuable suggestions!

We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.

Q1: The difference in the use of uncertainty for "Conflict-Aware Fusion and Re-ranking" mechanism compared to work [1].

A1: Thank you for pointing out the related work [1]. The main differences between our CARML and work [1] are as follows:

(1) Different task types: The work [1] focuses on single-modality (text) knowledge editing, whereas our work is centered on the field of lifelong multimodal knowledge editing.

(2) Different usage stages and objectives: The work [1] uses mutual information maximization to retrieve unique knowledge subgraph, which is later pruned by analyzing the uncertainty generated by the LLM. In contrast, our approach intervenes at the retrieval stage to identify the most relevant knowledge by evaluating the uncertainty of the retrieval results across modalities.

(3) Our core Innovation: We uniquely introduce the critical dimension of "inter-modality conflict." By combining "intra-modality uncertainty" with "inter-modality conflict," CARML can dynamically adjust the weights of each modality, thereby achieving precise knowledge localization.

We will cite and discuss work [1] in detail in the revision to more clearly highlight the novelty of our work.

Q2: Concerns about the "Explicit Guidance: Output Logits Enhancement" mechanism.

A2: Thank you for this valuable comment. We would like to clarify that the explicit guidance builds upon the parametric understanding established by the implicit prompt guidance. The implicit prompt is responsible for deriving a high-quality prior distribution of the editing knowledge. The explicit guidance then calibrates this distribution by incorporating the target answer, further boosting the generation confidence for tokens relevant to the edit.

To verify this, we conducted a supplementary experiment: we selected 100 E-VQA samples that were incorrectly predicted with only implicit guidance. During the auto-regressive generation process, we applied logits enhancement only to the first token of the answer sequence and observed whether it improves the model's prediction accuracy for subsequent tokens. The results are as follows:

Table 1: The accuracy of subsequent tokens with logits enhancement applied only to the first answer token.

The experiment shows that the "Output Logits Enhancement" can enhance the output probability for editing knowledge-related tokens and help calibrate the generation trajectory.

Q3: Regarding the specific justification for Equation (5) and attempts with other fusion methods.

A3: Thank you for your attention to the details of our formula. The purpose of Equation (5) is to reward modalities that exhibit low uncertainty and low inter-modality conflict. We achieve this as follows:

(1) We quantify the contribution of each modality's uncertainty and conflict degree by subtracting the normalized entropy and conflict score using 1, respectively.

(2) We assign equal weights to both factors. Experiments have shown that this simple weighting strategy demonstrates strong stability and generalization capability across different datasets.

(3) We did experiment with introducing learnable parameters (a small MLP) to combine the uncertainty and conflict scores to predict modality weights. However, the introduction of additional training parameters easily led to overfitting, whereas the simple averaging approach proved to be more robust in our experiments.

Q4: Regarding the sensitivity of the K value in Jaccard similarity.

A4: We conducted a sensitivity analysis on the value of K in the Jaccard similarity, using retrieval accuracy (Recall@1) as the metric. The experimental results on the E-IC dataset are as follows:

Table 2: The sensitivity analysis on the value of K in the Jaccard similarity.

The results indicate that the retrieval accuracy shows good stability when the value of K is within the range of [12, 21]. A K value that is too small can affect the conflict judgment due to insufficient candidate samples, while a value that is too large can introduce noise, leading to a decline in performance. We will add a complete discussion on hyperparameter sensitivity analysis in the revision.

[1] Shi et al. Retrieval-enhanced knowledge editing in language models for multi-hop question answering. CIKM 2024.

Thank you once again for your time and valuable feedback!

Dear Reviewer fHJZ,

Thank you very much for your thoughtful and constructive feedback. We are writing to respectfully follow up on our rebuttal and to see if you have any further questions.

Following your invaluable suggestions, we have made substantial efforts to improve our manuscript. The key clarifications and analyses from our response are summarized as follows:

● To better situate our contribution, we have detailed the distinction between our conflict-aware fusion and re-ranking mechanism and prior work [1].

● To address your valid concern about our "output logits enhancement" mechanism, we conducted a new supplementary experiment. The results demonstrate that this mechanism can further calibrate the model's generation trajectory rather than simply forcing the output.

● We have provided a detailed justification for our Equation 5. Additionally, following your suggestion, we conducted a new sensitivity analysis on the hyperparameter K to further enhance the manuscript's technical completeness.

We deeply value your expertise and hope that our responses and the new results have fully addressed your concerns. We would be very grateful to know if you have any remaining questions or if there's anything that requires further clarification.

Many thanks for your constructive comments, time, and patience.

Best regards,

The Authors