Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

The relation between the portability loss and the underfitting degree (UD).

Yes, portability loss does not directly imply UD. In Sec 2.2, Fig 1 shows that high portability loss is attributed to overfitting. Toward a deeper understanding of such overfitting, in Sec 2.3, (negative) UD (NUD) is defined to uncover the token-level HTO, which is a key contribution of this work as admired by the reviewer.

Specifically, NUD represents a token is overfitted as its training loss is too small. Specifically, NUD computes the difference between the token's training loss (of edited model), and the greedy decoded token's pretrained loss (of unedited model). The choice of greedy decoding is on purpose, as it reflects the unedited model's most confident knowledge proper and valid in the past. By comparing the two, NUD indicates that the edited model is overly confident, and is "overfitted" thereof. We will make this clearer.

Related work on Answer-level overfitting.

We thank the reviewer for bringing this interesting work to our attention. After reading it, we agree that the neighboring knowledge perturbation due to the answer-level overfitting is insightful, and it would be interesting to explore bridging the two types of overfitting and building more principled solutions. We will highlight this referred paper, the connection, and this future direction in the revision.

Empirical Comparison with DPO.

Following the reviewer's suggestion, we train LoRA with DPO from EasyEdit. We use the pre-edited model's old knowledge as the negative data. Due to time constraint, we only conduct Single Editing on ZsRE. We note that DPO performs worse, which we presume due to the practical challenge analyzed in Sec 3.2. }

Some case studies could make the contribution clearer.

Thank you very much for the suggestion. Due to time constraint, we will dive into more in-depth visualization on OVERTONE effect, and how OVERTONE helps multi-hop reasoning on MQuAKE in the revised paper.

MQuAKE contains 2-4 hops questions; why does you just do the 2-hops?

We apologize for the misleading statement. In experiments we followed MQuAKE official Repo, DeepEdit, and EasyEdit to load the multi-hop questions. We didn't manually filter out 3 and 4 hops questions. After rechecking the source documents carefully, we noted that "2-Hop" is inaccurate and should be "Multi-Hop". We will correct these descriptions in the revised paper.

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

OVERTONE can be applied to ROME (or MEMIT) to improve the loss function (Eq 4 in the original paper).

Thank you for the insightful idea of extending OVERTONE to ROME. We noted two unique designs in ROME (and MEMIT) makes it differ from four methods we studied. First, the impact of auto-regressive loss, which OVERTONE alters, on ROME is weaker, in the sense that the MSE loss will determine the final parameter update. Second, ROME relies on random prefix augmentation, which affects overfitting as well. Given these facts, we plan to work on a more principled way to extend OVERTONE, a augmentation-free end-to-end training paradigm, in light of its principle. That is, we seek a better way to smooth (relax) different token fitting adaptively with the model's own knowledge, following the principle of OVERTONE. We will highlight this challenge, together with our future plan in the revision.

Comparison with LTI, another method designed to alleviate overfitting.

Both LTI and OVERTONE works on mitigating overfitting in knowledge editing. The conceptual similarity, from a high level, lies in adding pre-trained knowledge to the editing. But LTI explores a distinct direction, with its difference to ours lying in X folds. First, LTI explores in-context learning (ICL) to incorporate pre-trained knowledge into the editing data, while ours designs an adaptive token-level distribution mixing, in light of the token-level HTO dynamic. Second, LTI, which is primarily developed for ROME-based solution, acts on both latent representation and output prediction loss. Ours, on the other hand, is agnostic to the editing method and alters the output prediction loss only. Finally, LTI, same as ROME, relies on a data augmentation, while ours does not include such mechanism. Following the reviewer's suggestion, we will highlight these differences in the revised paper, and will explore bridging the two directions in our future work.

Bias in knowledge conflicts and the model's own predicted distribution could be unreliable.

We agree that potential knowledge conflict and general noise can be misleading. To reduce this risk, OVERTONE incorporates two mechanisms. First, the unreliable (noisy) part is filtered out. Second, mixing with the model's prediction is conducted only if the mixed distribution correctly assigns the ground truth label (i.e., training token) the highest probability (Eq 3). Finally, provably solving the potential knowledge conflict for knowledge editing is still an open question, and we will highlight this in the revision.

Impact of filtering threshold $n$.

Mathematically, "without filtering" is equivalent to setting $n \rightarrow \infty$. As shown in Tab 3, this leads to a worse performance. To further study how sensitive $n$ is, we follow the reviewer's suggestion and try a larger $n=1$ on LoRA, which is the default value from Top $n\sigma$ paper. This gives us average performance 85.49 (Rel: 100, Gen: 94.85, Por: 61.44, Loc: 87.01) on editing ZsRE, which is slightly lower than 85.67 from $n=0.5$. We believe this insensitivity is reasonable, considering that correctness-checking mechanism will discard the mixing if it is misleading. We will add this discussion in the revised paper.

Knowledge editing in free text form.

We thank the reviewer for coming up this interesting direction. After checking related papers, we agree that the free-form text can express more diverse knowledge, on which the editing can be important but also more challenging. Considering the common practice that knowledge to edit at a time is few, we expect a similar overfitting due to the small training size nature, and HTO because some pretrained knowledge can still be useful, making different parts (tokens) have varied difficulties to learn. Therefore, we believe that our method can shed light on this interesting problem. However, we also believe that the free-form will add additional challenge to understand and quantify the overfitting. Therefore, we will explore this direction in our future work, and add this discussion, together with related papers, in the revised paper.

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

Why does negative UD (NUD) represent overfitting?

NUD represents a token is overfitted as its training loss is too small. Specifically, NUD computes the difference between the token's training loss (of edited model), and the greedy decoded token's pretrained loss (of unedited model). The choice of greedy decoding is on purpose, as it reflects the unedited model's most confident knowledge proper and valid in the past. By comparing the two, NUD indicates that the edited model is overly confident, and is "overfitted" thereof. We will make this clearer.

Is OVERTONE adaptable to other fine-tuning methods and tasks? On long-text scenarios?

OVERTONE is developed in light of HTO (i.e., editing knowledge is overfitted at different speeds). As per Sec 2.3, one cause of HTO is when knowledge editing (KE) involves few training data (such as single one) and trains the model with the fixed data for many steps, inevitably overfits. Similar concerns may raise in other tasks that seek selective updates of LLMs such as machine unlearning, wherefore OVERTONE could be applied. Moreover, when the training text is long, as the number of tokens to learn grows, we expect HTO to exacerbate, and OVERTONE to be helpful.

Why does portability perform significantly worse on WISE (Table 5)?

Table 5 reports editing performance on LLaMA 3 with hyperparameter adopted from LLaMA 2, which can be suboptimal. As an evidence, the vanilla WISE also performs worse than on LLaMA 2 (WISE has an activation mechanism to determine if edited parameters should be used, which needs additional tuning as well). Nonetheless, OVERTONE was able to help achieve a better editing-generality-portability trade-off, leading to a higher average performance.

Why a notable decline in generalization (gen) for FT and LoRA?

We believe the generality (Gen) decrease is also caused by suboptimal hyperparameter: As shown Fig 1, Gen doesn't encounter degradation due to HTO: as Gen loss change is nearly identical to that of training loss. Therefore, when OVERTONE mitigates HTO, it slightly decreases Gen while achieving better portability and locality. Still, the average performance is improved, and we believe our method can benefit from a more extensive hyperparameter tuning.

Why were MEND, MEMIT, and MELLO not evaluated in Table 2?

Table 2 showed that OVERTONE can help improve reasoning in more challenging scenarios. Therefore, we focused on FT and LoRA, two simple methods suffered more degradation from HTO, to demonstrate the effectiveness of OVERTONE. We didn't include MEND, MEMIT, and MELLO because of their different mechanisms: MEND relies on a large external dataset to train its hypernet, MEMIT uses a objective other than maximizing the editing data likelihood, and MELLO is training-free.

Filtering tail regions makes locality decrease.

This trend can be related to the definition of locality. Conceptually, a perfect locality only requires the edited model prediction match its pretrained output, regardless of its correctness and usefulness. Therefore, without filtering the tail region of the model's own prediction, the model's pretrained "knowledge" dominate the target $\pi_{tar}$ to learn, leading to a higher locality. However, the tail region of the model's own prediction is usually noisy, and this noise can be harmful. As shown in Tab 3, both generality and portability decreased.

Typos and grammatical error.

Thank you for catching the error! We will fix this mistake in the revised draft.

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

More model architecture.

We follow recent works (e.g. EasyEdit survey) to study the representative LLaMA family. Following the reviewer's suggestion, we further study Qwen2.5-3B-Instruct. Due to time constraint, we only experiment it with ZsRE, using hyperparameters on LLaMA-2 (which can be suboptimal). As shown at: https://anonymous.4open.science/r/hto-overtone, OVERTONE again helps achieve better editing performance. For the reviewer's convenience, we paste FT-M Single Edit results here.

Comparison with DPO.

Following the reviewer's suggestion, we train LoRA with DPO.We use the pre-edited model's old knowledge as the negative data. Due to time constraint, we only conduct Single Editing on ZsRE. We note that DPO performs worse, which we presume due to the practical challenge highlighted in Sec 3.2.

Statistical significance tests along with mean and variance from multiple runs.

Our experiment design follows the convention in Knowledge Editing and is based on the widely-used EasyEdit. All metrics are average from different samples, each uses a different initial value. The random seed is fixed to 42, so "standard" and "ours" use the identical initial value for the same sample. Following the reviewer's thought, we tried seed 2025, and ran "ours" as in Tab 3, resulting in a new average 85.44 (Rel 100, Gen 94.88, Por 60.41, Loc 86.47), which is very close to the reported one, confirming the effectiveness of our method. From a statistical test perspective, out of 72 comparisons ("standard" vs "ours"), ours achieved better performance in 69 cases, providing a "significant" improvement based on binomial test.

Finally, we agree with the reviewer that conducting multiple runs on each sample (knowledge) will further enhance the reliability, which is valuable but has been largely overlooked by the community. We will highlight this in the limitation section of the revised paper, and will follow this principle in our future work.

Hyperparameter selection and sensitivity.

We didn't conduct an extensive hyperparameter tuning. Our current selection can be found in App B, and are made as follows: $\epsilon$ is set to close to 0, and we tried 0.05 and 0.01; $n$ for filtering tried $1$ (default in "Top $n\sigma$" paper) and more aggressive $0.05$ for simpler LoRA and FT. Finally, mixing $\lambda$ was set to $0.1$ to encourage fast integration of $\pi_{flt}$ without tuning. We didn't notice too big difference when trying different $n$, which we found reasonable, as the correctness-checking mechanism (Eq 4), will discard a smoothing if it is too misleading. However, OVERTONE can be sensitive to $\lambda$, as greater $\lambda$ makes our method more similar to standard training.