BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing

Thanks a lot for the valuable feedback! We would like to clarify some points below.

Why our locality sample is harder?

A: Thanks for the question. Our locality sample is harder due to the sematical similarity to the editing knowledge. For example:

We will add more qualitative comparisons in the paper.

How is the validity of the dataset?

A: Thanks for raising this important point. We provide the Category-level Statistics of OKEDIT in our response to Reviewer 55Ec, showing its generality. We also performed human verification on a subset of samples to ensure the correctness and the alignment with the intended edit scope. This helps establish the reliability of OKEDIT.

How is α selected?

A: Thanks for the valuable concern. We select α using a small held-out set of only 5 unrelated samples. Since different models may have different latent feature distributions, α is treated as model-dependent. Once chosen, the same α is fixed per model (e.g., α=0.2 for MiniGPT-4) for all evaluations.

Why is our negative sample better than generating counterfactual negative sample.

A: Thanks for the thoughtful question. We want to clarify that using counterfactual samples presents several limitations:

• No Extra Knowledge Assumption:
  We aim to avoid requiring additional knowledge beyond the editing input for our goal of being realistic and efficient, but counterfactual sample require external knowledge.
• Intractability of Counterfactuals:
  Generating high-quality counterfactuals is costly, while ours use an efficient, universal, and reusable negative anchor.

To further justify the effectiveness of our negative anchors, we conducted an empirical comparison with a random negative sample baseline (in the second last response to reviewer ZhWy). We still achieve better results.

How is the editing layer chosen?

A: Thanks for raising this concern. In our method, we select the editing layer based on prior work(MEND, MMEDIT).

To validate the robustness of our method, we conducte an ablation study on another editing layer below:

Even if a different layer is chosen, we can still achieve good performance, showing the robustness of our method.

Why is your positive sample better than retrieving associated positive images?

A: Thanks for the suggestion. We would like to clarify that retrieval would introduce additional overhead, including retrieval pipelines and sample bank. Retrieval also assumes that visually similar positive samples exist, which may not hold for long-tail or rare edits.

In contrast, our method uses textually rephrased questions, enabling efficient and consistent construction of positive samples, keeping our framework lightweight and generalizable across edits.

The loss function in Eq. 2 is not clear.

A: Thanks for pointing this out. $L$ is the language loss, specifically the next-token prediction loss used by the base LLM. This loss ensures that the updated model $f_{\text{new}}$, when processing input$(i, t)$, generates the desired new answer $y_n$.

What does the time in table 5 mean?

A: Thanks for the question. In Table 5, “training” refers to the offline pretraining phase, such as training a metanet (e.g., MEND). However, We don't need that.

The time spent fine-tuning in our method is counted under "editing time", as it is performed per edit. The fast edit time shows that our method is efficient.

What is the extreme situation for α?

A: Thanks for the suggestion. We conduct ablation experiments using extreme values of α on Minigpt4 with OKEDIT dataset:

• When α = 0, the influence radius is entirely determined by the negative sample, leading to over-generalization and lower locality (e.g., 16.30 Loc), as the edit is applied too broadly.
• When α = 1, the radius is determined by the positive sample only, leading to over-localization and poor generalization (e.g., 24.22 I-Gen), since the edit applies to narrow a region.

These results confirm that α controls the trade-off between generality and locality as intended, which validates our assumption.

We thank the reviewer for providing valuable feedback to improve our paper. We have addressed your hesitations below.

Please highlight the contributions compared to existing works.

A: Thank the reviewer for the valuable suggestions. We agree that clearer differentiation is necessary. Below, we restate our key contributions while explicitly contrasting them with existing multimodal knowledge editing works:

1. We formulate the generality-locality trade-off in multi-modal model editing.
   In contrast, prior works such as MMEDIT [1] and GRACE [2] focus either on general-purpose editing or lifelong editing, but do not explicitly define or evaluate the trade-off between generality and locality in the multi-modal setting.

2. We introduce OKEDIT, a new benchmark dataset designed to evaluate both generality and locality.
   In contrast, MMEDIT [1] uses random choose pairs as locality samples which may result in loose alignment and limited coverage.

3. We propose BalancEdit, an efficient editing framework that dynamically adjusts the influence scope of edits.
   In contrast, IKE [3] assumes a wide influence range due to retrieval-based prompting, while GRACE [2] uses fixed-radius memory lookups. We introduce a radius-based mechanism using positive and negative samples to support dynamic control over edit scoping.

4. We design a parameter and data efficient mechanism with little training overhead per edit.
   In contrast, MEND [4] and similar meta-learning approaches require pretraining on large edit datasets, which are hard to obtain in the multi-modal domain. BalancEdit avoids pretraining, supports multi-edit scenarios.

We will revise the Introduction and Related Work sections to make these distinctions clearer in the final version.

[1] Cheng et al., Can We Edit Multimodal Large Language Models?, arXiv 2023
[2] Hartvigsen et al., Aging with GRACE, NeurIPS 2024
[3] Zheng et al., Can We Edit Factual Knowledge by In-Context Learning?, arXiv 2023
[4] Mitchell et al., MEND: Model Editing with Noisy Demonstrations, NeurIPS 2021

How is the quality and comprehensiveness of the OKEDIT dataset?

A: Thank you for the important observation. While OKEDIT is our main evaluation benchmark, we have taken several steps to ensure the robustness, diversity, and generalizability of both the dataset and our method:

1. Cross-dataset Validation:
   We evaluated BalancEdit not only on OKEDIT but also on the MMEDIT dataset, and observed consistently strong performance (see Table 3), which supports the generalizability of our method across datasets with different construction paradigms.

2. Grounding in a Verified Source Dataset:
   OKEDIT is built upon the OKVQA dataset, which has been widely used and shown to exhibit limited bias. This provides a reliable and diverse base of real-world image-text pairs for constructing edits.

3. Diverse Category Coverage:
   Our dataset spans 11 broad knowledge categories (e.g., animals, transportation, brands, people, holidays, etc.), ensuring that our benchmark reflects both common and long-tail factual knowledge. This promotes balanced evaluation across a wide range of input types.

4. Harder and More Realistic Locality Samples:
   Compared to MMEDIT, OKEDIT constructs semantically similar locality distractors, making the evaluation more challenging and reflective of real-world edit interference scenarios.

5. Human Verification and Quality Control:
   We conducted human validation on a subset of the dataset to ensure correctness of labels and alignment with the intended edit scopes.

Together, these points demonstrate that OKEDIT is a carefully designed and reliable benchmark, and that BalancEdit is not overly dependent on a specific dataset structure. We will revise the manuscript to better highlight these points.

How is the generalization of the method on newer model?

Thank the reviewer for the constructive suggestion. To evaluate the generalization of our method on more recent vision-language backbones, we conducted additional experiments using Qwen-VL on the MMEDIT dataset. The results are shown below:

Our method achieves a higher harmonic mean (HM) compared to GRACE, indicating a more balanced trade-off between generality and locality. These results also validate the generalizability of BalancEdit to newer backbone models beyond those originally reported in the paper. We will add more backbones in the future work.

We thank the reviewer for the insightful feedback, we are glad for the opportunity to clarify some points.

Why is our method parameter-efficient?

A: Thanks for the valuable comment! Knowledge editing is a challenging task, and we acknowledge that memory grows linearly with the number of edits due to cached transformations. However, compared to baselines like full fine-tuning or meta-learning (e.g., MEND), BalancEdit is still parameter-efficient per edit, as it only modifies a single layer and avoids retraining the full model. Moreover, our transformation can be replaced by PEFT method (e.g., LoRA), which would further reduce memory usage. We chose full fine-tuning for universality and for clearer and fair comparisons.

Why use averaged embedding as the key?

A: Thank the reviewer for the thoughtful question. We would like to clarify that using the averaged embedding as a sentence-level representation is a common and effective strategy(e.g., SimCSE [1], SBERT [2]), and we adopt a similar approach to cache edited knowledge in BalancEdit.

We chose this method over influential-token-based strategies (e.g., ROME [3]) for two main reasons:

1. Modality Gap: Prior work like ROME is designed for text-only models (e.g., GPT), while we focus on multi-modal models, where token-level attributions are harder to isolate due to vision-language fusion.

2. Simplicity and Generalization: Unlike token-attribution methods, our approach does not require edit-specific or model-specific token selection, making it easier to generalize across diverse edits and architectures.

We find that average embedding is robust, simple, and effective for our editing framework.

[1] Gao et al., SimCSE, EMNLP 2021
[2] Reimers & Gurevych, Sentence-BERT, EMNLP 2019
[3] Meng et al., Locating and Editing Factual Associations in GPT, NeurIPS 2022

Why does your Negative sample selection better than randomly chosen negative sample?

A: Thank the reviewer for the insightful question. We conducted an empirical comparison with a random negative sampling baseline, where a randomly chosen text-image pair (assumed irrelevant to the edit) is used as the negative anchor.

BalancEdit consistently outperforms the random baseline in harmonic mean (e.g. 71.85 vs. 65.61), showing a better balance between generality and locality. This supports the effectiveness of our black image-based negative anchor, which provides a fact-agnostic, consistent, and efficient way to define a lower bound in representation space.

In contrast, random negative samples:

• Depend on external unrelated examples,
• Are unstable in quality and relevance,
• And require assumptions that may not hold across diverse domains.

Our method avoids these issues, making it more robust and scalable for real-world editing scenarios. We will include this analysis and clarify the design rationale in the final version.

Typos:

A: Thanks for the detailed review. We will revise them accordingly.

We thank the reviewer for the valuable feedback. Here we're glad for the chance to clarify some points.

Quality of the generality and locality samples?

A: We appreciate the reviewer’s thoughtful concern. To ensure the semantic quality of the OKEDIT dataset and the meaningful separation between generality and locality, we adopt a carefully designed generation pipeline. The detailed generation process has been provided in Appendix A.1. Here we provide a concrete example.

The editing counterfactual knowledge is the "HP" looks computer is named as lenovo. Specifically, the original image is a HP brand computer, original question is 'What is the brand of it', and the original result is 'HP', and the new result is 'lenovo'.

• Image Generality sample generation:

1. Generality object confirmation: We ask GPT4 about objects and scene that should be in the image. The prompt is shown in Appendix A.1. The GPT would answer "A HP laptop".
2. Generality sample generation: Based on the object, we will prompt the diffusion model to generate the images. Since we provide the exact object, the model can generate correct images.

• Image locality sample generation:
  1. Locality answer generation: We first prompt GPT4 to generate a distractor. Like "Given (question: What is the brand of it, A: HP, B: Lenovo), what could be another option? Short answer. C: []" The GPT would generate a distractor, such as "Dell"
  2. Locality object confirmation: Similarly, we ask GPT4 about objects and scenes that should be in the image. The GPT would answer "A Dell laptop".
  3. Locality sample generation: Based on the locality object, we will ask diffusion model to generate the correct images.

In this way, having clear instructions for creating an image helps ensure its quality.

What is transformation(v)? What is L and a specific key k.

A: We apologize for the confusion about Eq. (2). In the following. We clarify each component of Eq. (2) and we will update the final version of our paper accordingly:

Transformation $v$：The transformation refers to a specific layer in the LLM. During an edit, we fine-tune this transformation layer to encode the new knowledge. Thus, $v$ is a updated version of an existing LLM layer, reused only for inputs within the influence radius of a particular edit.

Loss $L$: $L$ is the language loss, specifically the next-token prediction loss used by the base LLM. This loss ensures that the updated model $f_{\text{new}}$, when processing input$(i, t)$, generates the desired new answer $y_n$.

Key $k$: As shown in Method 3.2, each $k$ in the codebook is a representative embedding for a specific edit. For example, suppose we edit the fact: "What brand is this computer?" from "HP" to "Lenovo". Then the key k is the averaged embedding from layer $l-1$ when the input is the HP laptop image + the original question.

“By caching embeddings for input errors and the updated knowledge transformation layer that decodes into the desired model outputs”.. Could you explain it in detail?

A: Thank the reviewer for pointing this out. We agree that this sentence may have been too condensed, and we are happy to explain it in detail. Similar to the Section 3.2, We mean:

1. Caching embeddings for input edits (keys $k$):
   For each edit (i.e., input $(i, t)$ and new answer $y_n$), we extract the averaged embedding from the layer $l-1$ of the model. This embedding — the key $k$ — represents the semantic location of the edit in the model’s latent space. It serves as a reference point to decide whether future inputs are close enough to activate the edit.
2. Caching the transformation $v$:
   We fine-tune and cache a copy of the LLM’s layer $l$ so that, when applied to this type of input, it produces the new target output $y_n$.

How does α work?

A: Thanks for mentioning the key point. α adjusts the influence radius by weighting the distances between the key and the positive/negative samples. It also controls the trade-off between generality and locality.

Statistics of categories

A: Thank the reviewer for raising this point. Below, we provide detailed statistics of the knowledge categories in the OKEDIT dataset. As shown in the table, OKEDIT spans a diverse and representative set of domains, supporting its utility as a comprehensive and generalized benchmark for multi-modal model editing.

Reproduction and typos

A: We thank the reviewer's attention. We will fix the typos based on the suggestions and provide the code in the final version.