In-Context Editing: Learning Knowledge from Self-Induced Distributions

We greatly appreciate your constructive feedback and the time you took to review our work. We will carefully address each of your points raised.

Weaknesses

An FT Baseline:

• Apologies for any confusion caused! Including such a baseline is indeed important. In fact, FT-M and FT-L serve as the FT baselines for this purpose. To ensure comparability with our method, we updated the same layers as MEMIT for FT-M and FT-L (see Section D.2) and evaluated them on all datasets rather than directly copying numbers from the original survey.
• We have clarified this in the updated manuscript and moved this discussion to the main text.

Clarifications for Section 3.2:

• This section was intended to provide readers with insights that when no external information is provided, there is no benefit from sampling, leading to the final method. We have updated the manuscript to make this point clearer.

Novelty of the Method:

• Regarding Snell et al. (2022), we agree that conceptually, both methods aim to learn context information efficiently and effectively. However, they diverge in their algorithmic and technical approaches. Snell et al. (2022) requires two copies of model weights because it is fundamentally a distillation algorithm (although the weights may originate from the same model). In contrast, our method proposes a regularization term that can be seamlessly added during fine-tuning. This makes our method easier to apply and tune via the $\lambda$ parameter.
• Additionally, Snell et al. (2022) relies on task templates and answer extraction modules. Overall, our method can be more easily applied to learning problems. We have updated the manuscript to discuss these distinctions in the related work section.

Questions

$\lambda$ Used in the Experiments:

• In the experiments, we used $\lambda = 1.0$ without deliberate tuning. During the rebuttal period, we conducted ablation studies on $\lambda$ and have included these results in the updated manuscript.

Portability Score of the WikiBio Dataset:

• The WikiBio dataset does not include data for hopping questions, which is why no portability score was evaluated for it.

Perplexity Score:

• Thank you for raising this question! The perplexity score measures how well the pre-trained model predicts the generated outputs from the fine-tuned models. Assuming the original model is well-trained, the perplexity score reflects the quality of the fine-tuned model and how far it has drifted. We have clarified this in the updated manuscript.

Best Model Not Reported as a Baseline:

• Apologies for the confusion. Our statement referred to FT-M demonstrating nearly the best performance according to the survey, and FT-M is indeed included as a baseline in our experiments.

Updated Layers:

• Thank you for highlighting this! These details are essential, and we have updated the main text to make sure they are clear.

We sincerely appreciate your valuable feedback and would be grateful for any further thoughts you may have!

Dear Reviewer c9Gw,

We sincerely appreciate your time and effort in reviewing our work. We understand that your schedule may be quite busy. As the authors-reviewer discussion phase draws to a close, we kindly request your attention to our responses. Our aim is to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss. We also hope that if you are satisfied with our answers, you may consider adjusting your score and confidence accordingly.

We look forward to the opportunity to discuss this further with you. Thank you for your thoughtful consideration.

We are truly grateful for your encouraging remarks, insightful feedback, and thoughtful questions! We would like to address each point you have raised.

Weaknesses

Generalization to Other Models:

• We apologize for any confusion! We have also conducted evaluations using GPT2, with results presented in Appendix Section D.3. Our findings reveal that GPT2 performs worse than Llama2, underscoring the importance of the base model's in-context capabilities.

Questions

Impact of In-Context Ability:

• We agree that the effectiveness of ICE may be affected by the in-context abilities of the base model. This hypothesis is elaborated upon in the last two assumptions in Section 3.5 and is corroborated by the comparative results between GPT2 and Llama2.

We sincerely appreicate your detailed and valuable feedback! Below we address each points acoordingly:

Weaknesses:

Connections to Knowledge Distillation (KD) and Gisting: We appreciate the suggestion! We absolutely agree that conceptually, all three methods aim to learn context information efficiently and effectively. However, they diverge in their algorithmic and technical approaches.

• Snell et al. 2022 requires two copies of model weights because it is fundamentally a distillation algorithm (although the weights may originate from the same model). In contrast, our method proposes a regularization term that can be seamlessly added during fine-tuning. This makes our method easier to apply and tune via the $\lambda$ parameter.
• Mu et al. 2023 proposes a gisting technique that compresses the information of contexts into a token. This requires the addition of an extra token to the input, which may not be desirable in some cases, especially when we do not know that the context is useful beforehand.

In summary, our method offers ease of application for learning and subsequent inference. We have revised the manuscript to discuss these distinctions in the related work section.

Effect of Context: To clarify, the context refers to the knowledge we aim to edit or its paraphrase. The key point is that sampling without external information fails to produce an informative distribution; thus, incorporating knowledge as context is essential for effective tuning.

Consequently, it is not critical whether the knowledge originates from a more powerful LLM. What matters is that the training objective is contingent upon the context. Without this context, the process would effectively mirror standard fine-tuning.

To provide a clearer understanding of what the contexts entail, here are some examples:

{
    "prompt": "The mother of Mallory Reaves is whom?",
    "ground_truth": [
        "Brynne Chandler"
    ],
    "target_new": "Lalli Reaves",
    "context": [
        "Mallory Reaves's mother is Lalli Reaves.",
        "Lalli Reaves is the mother of Mallory Reaves.",
        "The mother of Mallory Reaves is identified as Lalli Reaves.",
        "In terms of parentage, Mallory Reaves's mother is Lalli Reaves.",
        "Mallory Reaves is the child of Lalli Reaves."
    ]
}

Ablation Studies: We apologize for any confusion caused! In our ablation studies, all rows are obtained with our combined loss function with sampling sequences. Specifically, 1) “Dynamic” refers to whether we consistently utilize the unedited model for sequence generation during the optimization process, and 2) "Context" indicates whether context is employed in the sampling process. Consequently, the four rows correspond to:

1. ICE
2. The method described in Section 3.2, which samples without context but updates the distribution during optimization, effectively equivalent to vanilla fine-tuning.
3. Samples generated with context from the unedited model, without updating the distribution during optimization.
4. Samples generated without context from the unedited model, and without updating the distribution during optimization.

• Ablations on Context Rows 2 and 4 both sample sequences without context. Notably, row 4 can be competitive in terms of locality, primarily not because of the missing context but because of the use of samples from the unedited model, which better preserves the original weights. In contrast, row 2 performs worse than other baselines.
• Empirical Results for Section 3.1 and 3.2 The second row presents results for Section 3.2, which is equivalent to vanilla fine-tuning as indicated by Observation 1. This variation underperforms compared to other configurations.

Upper Bound Experiments:

• Thank you for the insightful suggestion! There is existing research that advocates for the use of in-context learning for knowledge editing Zheng et al, which incorporates the edited knowledge directly as context. This approach achieves near-perfect accuracy and demonstrates greater locality than what ICE currently offers. However, it is important to note that this does not represent the absolute upper bound of the method, as additional techniques can be integrated to further enhance performance.

Additional Baselines (MEND and SERAC):

• Thank you! We will incorporate MEND and SERAC as additional baselines in our revised experiments to provide a more comprehensive comparison.

Modified Parameters and Cost Analysis:

• In our experiments, we ensure fair comparisons by modifying the same five layers as MEMIT for FT-M, FT-L, and ICE. The training costs for these three approaches are comparable. However, we recognize that the actual running time for ICE is slightly longer than for FT due to the need for sampling during training. Nonetheless, this difference is minor, as inference can be performed quickly compared to tuning.

We sincerely appreciate your insightful feedback and questions! We value your perspectives and would like to address each point raised:

Weaknesses

Concerning the computational demands of our pipeline, it is accurate that our method involves multiple sampling steps and depends on GPT-4 for context generation. However, we would like to clarify that the computational burden may not be as substantial as it appears:

1. Since sampling only necessitates a forward pass of the model, the computational cost is significantly lower than that of training the model.
2. We are considering scenarios with very limited training data, as is the case in the knowledge editing task. Therefore, the computational cost is not as critical as it would be in large-scale pre-training.

To get a more concrete idea of how the contexts look like, here are some examples:

{
    "prompt": "The mother of Mallory Reaves is whom?",
    "ground_truth": [
        "Brynne Chandler"
    ],
    "target_new": "Lalli Reaves",
    "context": [
        "Mallory Reaves's mother is Lalli Reaves.",
        "Lalli Reaves is the mother of Mallory Reaves.",
        "The mother of Mallory Reaves is identified as Lalli Reaves.",
        "In terms of parentage, Mallory Reaves's mother is Lalli Reaves.",
        "Mallory Reaves is the child of Lalli Reaves."
    ]
}

We have also updated the manuscript's appendix to include the examples.

Questions

**Rationale behind including $x^{\*}$ **

• That is an excellent question! If the sampling is not conditioned on the target, we would be optimizing L_{ICE} alone. However, the inclusion of $x^{\*}$ allows us to optimize the combined loss $L_{ICE} + \lambda L_{FT}$. By directly maximizing the likelihood of $[x^{\*}, x_c]$, we can optimize the combined loss given $\lambda = 1$: $L = L_{ICE} + L_{FT}$, which is algorithmically convenient. We have included a proof of this in the updated appendix A.3.

Concerns regarding GPT-4 for context generation

• We understand your concerns regarding the use of GPT-4 for context generation. We recognize that GPT-4 is not infallible and can produce hallucinations, and we have added a 'Limitations' section in our revised manuscript that discusses the potential risks associated with using GPT-4. In practice, since we are utilizing the model solely for paraphrasing, we have not observed any hallucinations.

Definition of 'temperature':

• In this context, the term "temperature" refers to the standard temperature sampling parameter used when sampling sequences from the model. The purpose here was to investigate how temperature influences optimization. While both optimization schemes demonstrate convergence, our method consistently exhibits lower equilibrium loss under different temperatures.

Updates to Algorithm 1

• Thank you for your suggestion! We have updated Algorithm 1 in the manuscript to reflect that. Additionally, we added an ablation study on the effect of $\lambda$ in the updated manuscript. In our original experiments, we simply used $\lambda=1$, which yielded satisfactory performance, and we did not deliberately tune this parameter, as it serves more as a tuning variable for others to experiment with.

Thank you once again for your invaluable feedback!