Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models

We sincerely thank you for your constructive and helpful comments. We initially address all your concerns and questions below.

[4-1] Difference between the previous peturbation methods

W1. The difference between the method proposed in this paper and the previous perturbation or enhancement methods in the feature space is not explained clearly.

Thank you for giving us a chance to clarify the point. To be clear, we have included the following sentence after Line 109 of the main paper: "Compared to previous perturbation methods, our method significantly differs in terms of the training objective for a perturbation function and its application to the knowledge injection of LLMs. Specifically, previous methods use bi-level optimization to train the perturbation function, while we use KL divergence to optimize the perturbation function by directly matching the output distribution of the perturbed model with that of the model using paraphrased data."

---

[4-2] Clarification on Gaussian used in the output distribution

W2. Why is the output distribution of the Transformer assumed to be Gaussian in the Training Part of Section 4.2? What is the effect of directly calculating the KL divergence of two distributions without considering the type of distribution? The author's explanation will help readers better understand the method.

We appreciate your valuable question. While it is possible to estimate with a more complex distribution, we chose to use a Gaussian distribution due to complexity issues. We found that even with this simplified modeling, our approach performs well.

---

[4-3] Computational cost comparison against paraphrasing method

W3. The authors claim that the paraphrasing method requires a high computational cost, so they should compare the proposed method with the paraphrasing method on computational cost.

Thank you for highlighting this important point. Our method requires less computational cost than the paraphrasing method, as our method requires adding a tiny module on the top of the LLM. To compare computational costs, we calculate each method in FLOPs (Floating Point Operations), with fine-tuning the LLM costing $F_{LLM}$.

1. Fine-tuning with paraphrased text: This requires generating paraphrases, costing $F_{LLM} + F_{Paraphraser}$ FLOPs, where $F_{Paraphraser}$ is the cost of a forward pass on the paraphraser model.
2. Fine-tuning with latent paraphraser: This requires a forward pass on the latent paraphraser, costing $F_{LLM} + F_{LaPael}$ FLOPs, where $F_{LaPael}$ is the cost of a forward pass on the latent paraphrasers.

A single latent paraphraser has $3d^2 + 4d$ parameters. For a 7B LLM, this is about 50M parameters. Using 5 latent paraphrasers totals 250M parameters. Thus, using a latent paraphraser is equivalent to a 250M parameter paraphraser, which is impractical for generating high-quality paraphrases. Given that gpt-3.5-turbo (or a 7B LLM) is used as the paraphraser in experiments, our latent paraphraser is significantly cheaper in computational cost.

---

[4-4] Additional experiments for Table 7

Q1. In Table 7, why was it not compared with the Fine Tuning (+para) method?

Thank you for pointing it out. We missed adding FreeLB and Fine-Tuning (+para) in Table 7. We conducted the experiments you mentioned and the results are as follows:

• Table 7 (with FreeLB and Fine-Tuning (+para))

Experimental result shows that using data-level paraphrases is effective in the real-world data setting. As we discussed in Section A Limitations, our latent paraphraser cannot resolve the reversal curse issue in the real-world data, while data-level paraphraser can. We have included the results and related discussions in Table 7 of the revision.

---

[4-5] A negative impact on the knowledge retention

Q2. Will the proposed knowledge injection method have a negative impact on the knowledge already mastered by LLMs?

We appreciate your valuable and insightful suggestion. To evaluate the potential negative impact on the knowledge already mastered by the LLM, we used the EntityQuestions dataset [1], which asks simple questions about entities. Among these questions, we only used the simple place-of-birth questions from the frequent entities (e.g., Where was Leonardo da Vinci born?), with 988 questions in total.

We fine-tuned the Vicuna-7b-v1.5 model on a synthetic SQuAD document set ($D_K$) using each method and measured the QA performance on EntityQuestions as follows.

• QA Performance on EntityQuestions after fine-tuning LLM on $D_K$ of SQuAD

The experimental results show that all fine-tuning approaches have a negative impact on the knowledge already mastered by the LLM. We also observe that the better the knowledge injection performance, the greater the negative impact on knowledge retention. As our work focuses on improving knowledge injection, knowledge retention is beyond its current scope. However, we believe discussing this point as a limitation is important as it can encourage future work in this direction. We have included this point and the experimental results in the limitations section of our revised manuscript.

---

Reference

[1] Sciavolino et al., Simple Entity-Centric Questions Challenge Dense Retrievers

We do appreciate your positive feedback and valuable suggestions, and we hope our response fully addresses your concern.

---

[3-1] Regarding new knowledge injection

W2. As a related point, I am wondering to what extent the used datasets can really evaluate new knowledge injection.

Q1. It seems unlikely that the model has never seen the new knowledge represented in D_K during pre-training. At the same time, the authors show that the model without adaptation scores much lower than any of the other baselines. This makes me wonder to what extent the authors think that LaPael measures true new knowledge injection in the LLM?

Thank you for your valuable comment. We would like to clarify that we measure the degree of knowledge injection by assessing QA accuracy on the QA dataset ($D_{QA}$), which corresponds to the documents ($D_K$) as defined in Section 3. We consider the knowledge that the LLM fails to answer as new knowledge to the LLM, even if such knowledge was seen during pre-training. Previous works [1, 2] also support these observations, as referenced in Lines 25-29.

However, we entirely agree with your opinion on measuring true new knowledge injection. Therefore, we have provided additional experimental results on two datasets containing new knowledge to the LLM, with available raw documents (Please see Table A in the attached PDF file on the global comment). The results show that our method is effective with raw documents on new knowledge, confirming the same trend observed in our previous experiments.

We have included these results in Section 5 of the revised manuscript considering their significance.

Details of the datasets:

• Films 2024: A synthetic QA dataset based on raw Wikipedia articles under 2024 films. We generated QAs from wiki documents using GPT-4o following Jiang et al. [3].
• Events 2024: A synthetic QA dataset based on raw Wikipedia articles under May, June, and July of US 2024 events in the United States. We generated QAs from wiki documents using GPT-4o following Ovadia et al. [4].

---

[3-2] Suggestion: Comparison with external memory-based approaches

W1. Comparing against external memory based approaches would have been a good addition to the results section.

As per your great suggestion, we have conducted experiments comparing our approach with Retrieval-Augmented Generation (RAG) on datasets with new knowledge, Films 2024 and Events 2024, as used in the previous answer 3-1. We use bge-large-en-v1.5 model for the embedding.

• Films 2024

• Events 2024

Experimental results show that the memory-based approach (RAG) outperforms the fine-tuning approaches, as observed in Ovadia et al. [4]. However, our method can close the gap between the two approaches, indicating a potential pathway to further improve fine-tuning methods. We have added these experimental results to the revised manuscript to emphasize this point.

---

[3-3] Answers on Questions

Thank you for your thorough clarification. We have incorporated all clarifications into the revised manuscript. While addressing your questions, we realized that the knowledge injection pipeline with LaPael -- training latent paraphrasers on $D_{train}$ $\rightarrow$ fine-tuning LLMs on $D_{K}$ -- can be confusing. Therefore, we have added a supporting figure to clearly explain that in Section 4 of the revised manuscript.

Line 177: Question about NER tagger

Yes, we use GPT-3.5-turbo as the NER tagger.

---

Line 193: mentions “rephrasing each question of D_QA”. Should this be “rephrasing each document”?

We apologize for the confusion. We would like to clarify that we used both question and answer to generate the synthetic document. This is because we want to ensure that the synthetic document contains only relevant information to the question, as illustrated in Table 1.

---

Line 217: mentions that “the LLM is fine-tuned on the test set”. To confirm — does this refer to the paraphrased version of the test set, D_K?

No, it refers to the documents $D_K$ defined in Section 3, not the paraphrased version of $D_K$. We used a paraphrased version of $D_K$ in experiments for Figure 4 and Fine-Tuning (+ para.) in Table 2.

---

Line 238: regarding “Combining LaPael and Paraphrases”. Specifically, what does “number of paraphrases” mean in the LaPael setting in Figure 4? Is “ours” the default LaPael setup, and “fine-tuning” LaPael + fine-tuning?

The number of paraphrases in Figure 4 refers to the number of paraphrases for each document in the dataset used to fine-tune the LLM ($D_K$) for both "fine-tuning" and "ours."

---

Table 7: Can the authors elaborate on how they adapt the baselines in this real-world data scenario?

In the real-world data scenario, baselines are also fine-tuned on the raw documents from the SQuAD test set ($D_K$). We would like to clarify that "ours" denotes fine-tuning the LLM on the raw documents from the SQuAD test set ($D_K$), with the latent paraphraser trained on synthetic documents from the SQuAD train set ($D_{train}$).

---

References

[1] Kandpal et al., Large Language Models Struggle to Learn Long-Tail Knowledge

[2] Allen-Zhu et al., Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

[3] Jiang et al., Instruction-tuned Language Models are Better Knowledge Learners

[4] Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

Thank you for your response, and clarifications. They largely address my concerns, and I appreciate the additional results for the two new datasets.

I have one follow-up question about the results for the new RAG baseline. First of all, thank you for running these experiments, and including the numbers in the rebuttal, and for updating the manuscript with these numbers. Currently, the RAG numbers are quite far from the LaPael numbers. What do you think is the main advantage of using a method like LaPael over the retrieval based method?

We sincerely appreciate your valuable comments on our work. We understand your concerns and are certain that your comments will help improve the quality of our work. Should you have any unresolved concerns, please let us know. We are happy to discuss and do our best to address your concerns.

---

[2-1] Experiments with available documents with new knowledge.

W2. It would have been more suitable to use datasets with available documents. Additionally, the documents used should not have been seen by the pretrained LLM during its pretraining phase to quantify how much knowledge has been injected into the LLM.

Summary: We show that our method is still effective with raw documents and new knowledge by conducting supplementary experiments on four datasets. The experimental results are presented in Table A of the attachment.

Thank you for your valuable suggestion. We agree that experiments on datasets with available raw documents and new knowledge are important. We acknowledge that the scope of existing experiments, including Table 7, is limited as they do not handle the new knowledge scenario.

To address this, we conducted new experiments on four datasets, including two with new knowledge, using available raw documents. Specifically, SQuAD-raw and StreamingQA-raw analyze knowledge injection with widely-used document-question datasets, while Films 2024 and Events 2024 test new knowledge injection, as Vicuna-7b-v1.5 is not pre-trained on them (see below for the details of the datasets).

Table A in the attachment on the global comment presents the experimental results, showing our method's effectiveness with available raw documents and new knowledge, consistent with previous synthetic document experiments in the paper. We have included these results in Section 5 of the revised manuscript considering their significance.

Details of the datasets:

• SQuAD-raw: The entire set of SQuAD test set with available raw documents.
• StreamingQA-raw: The entire set of StreamingQA test set with available raw documents.
• Films 2024: A synthetic QA dataset based on raw Wikipedia articles under 2024 films. We generated QAs from wiki documents using GPT-4o following Jiang et al. [1].
• Events 2024: A synthetic QA dataset based on raw Wikipedia articles under May, June, and July of US 2024 events in the United States. We generated QAs from wiki documents using GPT-4o following Ovadia et al. [2].

---

[2-2] The explanations on why we create a synthetic document out of QA pairs to evaluate the knowledge injection

W1. Why is only a subset of questions chosen instead of the entire test set? No details about this are mentioned in the paper.

W2. Creating a document out of question-and-answer pairs to inject knowledge into the LLM and then testing it on the same test set is inappropriate for evaluating knowledge injection.

We would like to clarify that we use datasets with synthetic documents to simplify the evaluation and analysis of knowledge injection, based on widely-used QA datasets. We used a subset of them to meet the budget limits on costs for gpt-4-turbo API calls and LLM fine-tuning. We constructed synthetic documents from existing question-and-answer pairs to ensure learning from these documents leads to knowledge injection, without considering variables like reversal curse and distracting contents, as mentioned in Line 193. We have clarified these points in the revision.

---

[2-3] The statistics for the documents used in experiments.

W3. There are no statistics available for the created documents, such as their size.

Thank you for pointing it out. We missed referencing the statistics initially, although they were in Table 8 of the Appendix. We have corrected this in the revision.

We also supplemented Table 8 with dataset size measurements and presented these in Table B of the attachment in the global comment. Additionally, we measured token counts per document and QA pair for all datasets used, plotting the histogram in Figure A of the attachment. These statistics are now included in Appendix B.1 of the revised manuscript.

---

[2-4] Clarification on in-context examples used.

W4. Line 212: Why are in-context examples used instead of experimenting with a zero-shot setting to assess the extent of the model's learned knowledge?

We use in-context examples to ensure that LLMs generate the answer in the desired format (phrase instead of sentence). We have clarified this point in Section 5.1 of the revision.

---

[2-5] Regarding the noise sampling in lines 184-185.

W5. Lines 184-185: It is not clear what the authors meant by "We sample N noise."

We would like to clarify that "We sample N noise" means that we randomly sample N different $\alpha$ values from the Gaussian distribution with mean $\mu$ as defined in Equation 5. We have clarified this point in the revision.

---

[2-6] Writing fixations.

W6. Line 182: "From Equation 10" should be "from Equation 15."

Wa. The acronym "LM" is used in the experiment section without being defined. The paper uses "LLM" for Large Language Models and sometimes "LM."

Wb. The paper uses "$D_{knowledge}$" and "$D_K$" interchangeably. Please ensure consistency.

Wc. There are missing details in the experiment section, and no correct reference is provided to indicate where in the appendix these details can be found.

Thank you for pointing those out. We have ensured consistent use of the acronym "LLM" and the term "D_K" throughout the paper in the revision. We have also fixed the no-reference issue by adding the references to the correct subsection of the Appendix.

---

References

[1] Jiang et al., Instruction-tuned Language Models are Better Knowledge Learners

[2] Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

We sincerely thank you for your constructive and helpful comments. We initially address all your concerns and questions below.

---

[1-1] Regarding the comparative analysis of paraphrased sentences

W2. Lack of qualitative analysis of show how the phrased sentences differ from the given paraphrased sentences from GPT3.5.

Q2. Any explanations or case studies showing the generated paraphrase sentences from LaPael?

Thank you for your feedback. We would like to clarify our approach as follows. Instead of generating paraphrased sentences, our proposed method perturbs the latent representations within the LLM, as denoted in Lines 45-46. These perturbed latent representations do not directly correspond to paraphrased sentences. Therefore, comparing our method to paraphrased sentences from GPT-3.5 is not applicable. We hope this clarification addresses your concern.

---

[1-2] Regarding the cost of training the latent paraphraser

W1. The cost of training the latent paraphraser is not clear, as expensive training will impede the efficiency of the proposed method.

Q3. The computation cost comparsion with noise-based paraphraser

Thank you for your valuable feedback and questions. Our proposed method does introduce an additional cost for training the latent paraphraser once prior to fine-tuning an LLM, compared to noise-based fine-tuning baselines. However, we would like to emphasize that this cost becomes negligible over multiple knowledge injections due to the following factors:

1. Size and Parameter Cost: The latent paraphraser has only 3.6% of the parameters of the main LLM (we used a 7B model), keeping the training cost low.

   • FLOPs Comparison: A single forward step for the LLM costs $F$ FLOPs (Floating Point Operations). Training the five latent paraphrasers requires approximately $1.108F$ FLOPs, compared to the $3F$ FLOPs needed for updating the LLM.

2. Training Dataset Size: The latent paraphraser can be effectively trained with a small dataset. Training with 50-100 documents is sufficient to surpass baseline performance, reducing the number of training steps and data required, as shown in Figure 5(a).

3. Transferability: The latent paraphraser is trained only once and can be reused over multiple knowledge injections without additional retraining, as shown in Table 3. As more knowledge is injected, the amortized training cost approaches zero.

We have clearly stated the cost of training the latent paraphraser compared to noise-based fine-tuning baselines in the revised manuscript.

---

[1-3] Regarding the cross-domain transfer

Q1. In table3, the cross-domain transfer: why the results trained on X evaluated on X, is worse than trained on X but evaluated on Y.

Thank you for pointing this out. We would like to clarify the document on which the latent paraphraser is trained ($D_{train}$) differs from the document on which the LLM is fine-tuned ($D_{K}$). Therefore, it is possible that the latent paraphraser trained on $D_{train}$ of X can show better performance on $D_K$ of Y than on $D_K$ of X. Moreover, the performance differences are marginal, indicating almost no significant difference. This highlights the transferability of our method, demonstrating that the training dataset for the latent paraphraser does not significantly affect its performance. This flexibility is shown in Table 3 and described in Lines 230-238.

Q1: I am asking for the computation cost comparison between noise-based and the proposed method, as the author mentioned "We have clearly stated the cost of training the latent paraphraser compared to noise-based fine-tuning baselines in the revised manuscript.", But I couldn't find it.

Q2: Thanks for your results and the acknowledgment of "this degradation of math reasoning performance".

I keep my original score mostly because the inefficiency in keeping the (other) basic reasoning capability, e.g., math of the LLMs after inserting such paraphrase layer, as one of the important principle of knowledge injecting is not to destroy the existing knowledge capacity.