DualEdit: Dual Editing for Knowledge Updating in Vision-Language Models

We sincerely thank the reviewer for providing valuable feedback. Please kindly let us know whether you have any further questions.

Q1: Although the authors provide empirical evidence for the effectiveness of DualEdit, the theoretical foundations of the method are relatively weak. For instance, the selection of key layers for editing and the design of the gating mechanism lack rigorous theoretical analysis, making it difficult to fully understand the underlying principles of the method.

We appreciate your recognition of our empirical evidence for DualEdit's effectiveness. While we agree that deeper theoretical analysis would be valuable, we would like to emphasize that understanding the internal representation mechanisms of LLMs and VLMs remains one of the most challenging open problems in the field. In light of this, our work focuses on making comprehensive and rigorous empirical contributions:

• Systematic empirical discovery: We are the first to conduct comprehensive layer-wise sensitivity analysis across both modalities during the VLM editing process, revealing that textual and visual representations reach peak sensitivity at different layers (Finding-1).
• Evidence-driven design: Our key layer selection and gating mechanism design are grounded in extensive empirical analysis (Figures 1-2, Table 1), following established practices in model editing research.
• Practical effectiveness: Our method achieves superior performance across multiple metrics and backbones, demonstrating that empirical insights can lead to effective solutions even when theoretical understanding is limited.

We agree that building rigorous theoretical foundations for VLM editing is an important and ambitious direction for future work. Meanwhile, we believe our empirical findings offer meaningful insights that push the field forward in a concrete and actionable way.

Q2: Experiments are restricted to two datasets and backbones; broader validation (e.g., on more VLMs like Flamingo or PaLI) would strengthen claims. Also, the data analysis of Sec. 2 only uses the LLaVA-V1.5 as the backbone to conduct the VLM's layer and metrics' charateristics.

We would like to clarify our experimental scope and architectural focus:

• Fousing on specific VLM family：As discussed in Section 5.1, our work targets Early Projection Fusion VLMs, where visual features are projected and fused into the language model at the input level (rather than injected into internal layers). This design is widely adopted in state-of-the-art models such as LLaVA-V1.5, BLIP2-OPT, which have demonstrated strong performance in both academic and industrial settings.
• Different challenges for Deep Fusion architectures： Extending DualEdit to Deep Fusion architectures (e.g., Flamingo, PaLI-X), which integrate visual inputs via cross-attention into internal LLM layers, presents fundamentally different challenges. Adapting our layer selection strategy and gating mechanism to such models would require substantial redesign, and is thus beyond the scope of this paper.
• Additional validation in progress: To better address the reviewer’s suggestion for broader validation, we have initiated experiments on MiniGPT-4, another representative Early Projection VLM. Due to current hardware constraints and time limitations, the experiments are still in progress.
• Analysis using more backbones: While our original analysis was based on LLaVA-V1.5, to ensure generality, we additionally analyze attention scores across different layers for BLIP2-OPT. The results are available at the following anonymous link: https://imgur.com/Kw2HSSC. According to the figure, although the attention score patterns of BLIP2-OPT differ significantly from those of LLaVA-V1.5, both are consistent with our Finding 1: Textual and visual representations in VLMs demonstrate varying levels of importance within the same layer, and the significance of each modality shifts across different layers.

Q3: The results of baselines are different from the original baselines' paper. For example, the results reported by VisEdit [1] are better than the metrics of Table 2.

Thanks for your careful review. To ensure fair and accurate comparison,

• We faithfully reproduced the VisEdit results using the official implementation, strictly following their released hyperparameters and experimental settings.
• All methods, including VisEdit and our DualEdit, were evaluated in a unified experimental environment, using the same hardware, software versions, and evaluation protocols.

While our reproduced VisEdit results may differ slightly from those reported in their original paper, the relative performance comparisons remain valid and fair, since all methods were run under identical conditions in our study.

[1] Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit

Q3: What is the sensitivity of the method to threshold selection and layer choice across different VLMs?

Thank you for your question regarding the sensitivity of our method to threshold selection and layer choice across different VLMs.

• (Threshold sensitivity). We present results under different threshold values. These results demonstrate that the model is not sensitive to the threshold choice; as long as it remains below a certain critical point, both reliability and generality remain stable. This indicates that threshold setting within our proposed DualEdit method is not a significant concern.

• (Layer choice sensitivity). As discussed in the response to Q1, we analyzed the impact of applying editing at different layers (e.g., layers 2, 5, 10, 30), and found that performance is relatively stable as long as editing is performed within the top visual layers (e.g., layers 16–19), which we identified through attention map analysis. When editing is applied to very shallow or very deep layers, performance may degrade, particularly for tasks that require stronger visual grounding (e.g., E-IC).

We will include this analysis in the revised version and clarify these points accordingly.

We greatly appreciate your constructive comments on our paper. We hope that our response satisfactorily addresses the issues you raised.

Q1: Given the limited difference in attention maps across V-Layers in Table 1, is the visual editing branch necessary or could simpler alternatives suffice?

Thank you for your careful review and for pointing this out. The limited differences in attention maps across V-Layers in Table 1 are due to our preliminary analysis, which indicated that the most critical layers are between layers 16 and 19. As a result, all these layers perform relatively well, leading to only small differences. We would like to clarify that the visual editing branch is indeed necessary, as demonstrated by the following points.

• (Visual editing is necessary). As shown in the ablation results in Table 3 (also listed below for convenience), removing the visual editing component leads to a clear drop in performance, where the average score decreases by 9.23%. This result highlights the importance of the visual editing branch in improving model performance.

• (The location of visual editing matters). To further analyze the impact of the visual modality, we also provide results for inserting the visual editing branch at layers 2, 5, 10, and 30. These results show that the choice of layer significantly affects performance, reinforcing the necessity of careful design when integrating visual information.

Q2: Could the gating mechanism be adapted to perform soft gating (e.g., attention-weighted blending) rather than binary on/off activation?

We appreciate your interest in the concept of soft gating, which indeed inspired us to further investigate its potential.

Following your suggestion, we implemented a soft gating mechanism where the last token of the edit sample attends to the last token of the input sample via a learnable cross-attention module. The resulting attention score is used as a continuous weight to softly combine representations, thereby enabling a dynamic gating process.

As shown in the results, soft gating indeed improves locality, demonstrating its effectiveness in adapting to edit-specific context. However, we also observe that it leads to a noticeable decrease in both reliability and generality. Compared to soft gating, our method achieves better performance. We attribute this to the following insights:

• (Stable threshold selection). While our gating method relies on a threshold as a hyperparameter, Figure 4 shows that gating based on last-token similarity achieves a clear separation, making the threshold straightforward and stable to select. Moreover, this clear distinction helps explain why our gating mechanism outperforms soft gating.
• (Inference-time only gating). Our gating mechanism is applied exclusively during inference, leaving the training process unaffected. This allows the model to preserve generalizable representations during learning, which in turn improves both reliability and generality.

In summary, while soft gating is an interesting direction and indeed offers certain benefits, we find that our proposed approach achieves a more favorable solution.

Thank you again for your thoughtful review. We would like to follow up to see if our responses have addressed your concerns or if you have any further questions. We would really appreciate the opportunity to discuss this further if our responses have not yet addressed your concerns.

Q3: If we trained an adapter using one type of data, I am not sure if it can be adapted a domain that is totally difference with the training data. Providing the generalization analysis of proposed method is needed for this paper.

Thanks for the comment. To better illustrate the generalizability of the proposed DualEdit, we would like to provide clarification from both in-domain and out-of-domain perspectives:

• Let $\\theta$ denote the parameters of the traned model $f(\\cdot)$. During training, the model is updated based on a set of editing samples $x$, resulting in new parameters $\\theta'$, such that the model can correct specific outputs while preserving other outcomes. During testing, we hope the $\\theta'$ can adapt to new editing samples $x'$, enabling the model to edit them correctly without affecting the outputs for other inputs.

• (In-Domain Generalizability). [The domains of $x'$ and $x$ are similar.] As shown in Table 2 for E-VQA and E-IC, our DualEdit achieves the best performance on their respective test datasets. For example, our DualEdit ourperforms VisEdit by 3.8 in average on E-VQA with BLIP backbone. This clearly demonstrates superior in-domain generalizability.

• (Out-of-Domain Generalizability).[$x'$ and $x$ ave largely dissimilar domains.] To evaluate out-of-domain generalizability, we constructed a benchmark where training and testing data differ significantly, allowing us to better demonstrate the model's generalization capablities.
  • (Benchmark Construction). Based on the E-VQA dataset, we constructed two completely distinct domains to rigorously evaluate domain generalization. Specifically, for the training set, we retrieved 1,209 samples related to humans by filtering with keywords such as "person," "man," and "woman." In contrast, the test set consists of 218 animal-related samples, selected using keywords like "cat," "dog," and "bird." This deliberate split creates two domains that differ significantly in content and context, allowing us to effectively assess the adapter's ability to generalize from human-related images to an entirely different animal-related domain.
  • (Performance Analysis). The results shown below indicate that the performance of both methods decreases under the out-of-domain setting. Nevertheless, our method still demonstrates strong generalization capability and consistently outperforms VisEdit, providing further evidence of its robustness.

Thanks for your time in dealing with our work. We will answer the question and discuss point by point as follows. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.

Q1: These two findings have been explored in some papers before. For example, there are many papers that want to emphasize the visual modality because they found the VLMs haven’t fully captured the image information. In addition, balancing the reliability and locality is an important aspect in the model editing methods. Hence these two findings have been proposed before. The novelty of this analysis is limited.

We would like to clarify that there is a basic misunderstanding about the two findings.

• (The first finding). To the best of our knowledge, our work is the first to analyze the influence of both textual and visual modalities simultaneously during the editing process. Unlike previous works that only emphasize the visual modality, we systematically investigate how sensitivity to edits varies across layers for both modalities, and identify modality-specific editing layers.
• (The second finding). We separately study how each modality affects locality, reliability, and generality, with the goal of understanding the distinct roles of different modalities. This is different from prior work, which mainly emphasizes the importance of balancing reliability and locality, without explicitly analyzing the contribution of each modality.
• Our work does not merely reiterate the insights outlined in Q1 but extends them by introducing modality-aware, layer-sensitive editing strategies specifically tailored for VLMs, which we believe represents a meaningful advancement in the field.

Building on the two findings, we propose DualEdit, a novel editor that explicitly modifies both modalities at their respective critical layers. DualEdit incorporates a gating mechanism to preserve the model’s original capabilities, particularly addressing the reliability–locality trade-off in the context of dual-modality editing

Q2: Training the adapters needs the data. Preparing the pairs including original sample and edit sample will be a time-consuming task. This paper hasn’t explained how to prepare these edit samples clearly. This is one of the most important part this paper needs to clarify. It is better to explain if we can prepare the data for model training easily.

Thank you for pointing this out. We would like to further clarify the details of the edit samples and benchmarks.

• In line with prior VLM model editing works [1-2], we directly use their publicly available benchmarks which contains pre-constructed original-edit sample pairs.
• We do not introduce any extra paired data for the training of our proposed adapter. Therefore, the data preparation process is not more time-consuming than that of previous works.
• We will revise the corresponding section of the paper to improve the clarity of our data preparation process and make this point more explicit.

[1] Can we edit multimodal large language models?

[2] Attribution analysis meets model editing: Advancing knowledge correction in vision language models with visedit