Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

Thank you for recognizing the importance of the problem and the novelty of our idea. We addressed the weaknesses and questions you raised as follows.

Q1: GenerativeAdaptor introduces 500 million parameters, however, even using 500 million parameters cannot achieve comparative performance as In-context Prompting (as shown in Figure 2(a)). Consider saving all the tokens in the context, which leads to 32768 * 4096 = 134 million parameters.

We want to provide further clarification on the parameter sizes of the adapter generator ($G$) vs the generated adapter ($\Delta W$) applied on top of the backbone LM for inference. The adapter generator has approximately 500 million parameters and is independent of the context, whereas the generated adapter is only about 32 million parameters across all experiments.

If we aim to retain information for multiple users, we would only need to store the generated adapter for each user, and a single copy of the adapter generator (along with the language model). In contrast, maintaining the token hidden states or KV cache for the entire context, as suggested, would result in significant storage overhead. We believe this distinction highlights the efficiency of our approach in scenarios requiring scalable storage solutions.

Q2: GenerativeAdaptor adds the adaptor which is quite similar to memory-based methods, where people would add a memory module on the large language models [1,2].

Thank you for pointing out this connection. We acknowledge that memory-based methods share some similarities with our approach. However, our work focuses on preserving the standard transformer-based language model architecture while proposing a method to efficiently and effectively update model parameters based on the given context. We will ensure to discuss and cite relevant works in the future, including the ones you mentioned.

Thank you once again for your thoughtful questions and valuable feedback! Please don’t hesitate to let us know at your earliest convenience if you have any further questions or concerns, or if there are additional experiments you would like us to conduct.

Thank you for recognizing the strengths of our work. We address the identified weaknesses and your questions below.

Q1: It seems like the knowledge that each baseline has seen are different in experiment 4.1.

Closed-book prompting and supervised finetuning are two baselines that do not receive any context during test time. The continual pretraining baseline sees all documents (complete knowledge) during the training stage. In contrast, only a subset of documents (partial knowledge) used for inference is seen by both in-context prompting and our method. In our experiments, we use the same subset of documents for both in-context prompting and our method. Thus, our generative adapter does not have any advantage in terms of access to more documents (knowledge) compared to continual pretraining and in-context prompting. We hope this clears up any misunderstandings regarding the fairness of our comparisons.

Q2: the paper mentions … quality of the adapter generator is highly correlated with the final evaluation metrics. It would have been better if the results that support this claim is included in the paper.

We provide additional evaluation results in the table below. Specifically, we compare two normalization methods: SVD normalization and Frobenius normalization, and evaluate them using reconstruction perplexity, completion perplexity, and F1 score on the MSC dataset (as used in Section 4.3).

Our findings indicate that reconstruction perplexity is strongly correlated with fact recall, as measured by the F1 score. For example, the SVD normalization method achieves a much lower reconstruction perplexity and a significantly higher F1 score compared to Frobenius normalization.

Q3: Is there a suspected reason why the unnormalized adaptor contains extreme values?

We suspect this behavior arises from the distribution of hidden states. Specifically, the hidden states of tokens are not uniformly distributed as a Gaussian but exhibit higher frequencies in certain directions. Consequently, the sum of the outer products of these hidden states results in a significant eigenvalue in the dominant direction, leading to instability.

Q4: In the formulations in section 2.3, there is a G(x_{1:m}). In this case, shouldn't the hidden states input to the generator, not the input tokens?

Here, the intention is to describe that the adapter is generated based on the first m tokens and is subsequently used to compute the log-likelihood on the updated model. We will clarify this point in the future version of the paper.

Q5: In the completion loss formulation, shouldn't the output just $x_{m+1}$, not $x_{m+1},...,x_n$?

The rationale here is that the adapter is generated based on $x_1$ to $x_m$​, and the log-likelihood is then computed over $x_{m+1},…,x_n​$ using the updated model. This reflects the model's ability to predict the continuation of the sequence after being contextualized with the initial context.

Thank you once again for your thoughtful questions and valuable feedback! Please don’t hesitate to let us know at your earliest convenience if you have any further questions, concerns, or if there are additional experiments you would like us to conduct.

Please don’t hesitate to let us know if you have any additional questions or need further clarification. We’d also love to hear if we’ve addressed all your concerns. Thank you again for your thoughtful feedback!

If the updates and clarifications we have provided address your concerns and improve the quality of the work to your satisfaction, we kindly ask you to consider reflecting this in your overall assessment. We greatly appreciate your feedback and efforts in reviewing our submission.

Thank you for recognizing the effectiveness and efficiency of our work in adaptation. We provide our responses to the weaknesses and questions you highlighted in detail below.

Q1: hyperparameter settings and implementation details could be expanded, especially regarding stability mechanisms like SVD normalization, which could impact reproducibility.

We have included the hyperparameter settings and implementation details in Appendix B. Regarding SVD normalization, we use the torch.svd_lowrank() function from the PyTorch library, as described in the "Implementation" paragraph. We agree on the importance of reproducibility and will improve clarity and provide additional details in the next version of the paper.

Q2: the lack of prompt engineering effort for baselines could introduce an unfair comparison.

The prompt templates used in this paper are listed in Figure 7 in the appendix. Since both backbone LMs are instruction-tuned, we observed them to be quite reliable in performing the standard tasks considered in this paper. While prompt engineering could potentially benefit both our model and the baselines, we leave this for future exploration.

Q3: The paper introduces concepts like "test-time contextualization" and "parameter-efficient" without specific definitions.

We introduce "test-time contextualization" in Section 2.1 (Line 135), where it refers to scenarios where context arrives incrementally as a stream of data (e.g., a continuous flow of documents or dialogue sessions). The goal is to obtain an updated model capable of responding to user instructions using the information provided in the context C

For parameter-efficient fine-tuning (PEFT), it involves updating only a small subset of parameters instead of the entire language model. We discuss related works on PEFT in Section 6 (Line 516).

Q4: claims of general "efficiency" for "personalization" are strong, given the specific constraints of the study.

In this paper, we define personalization as the ability of language models to analyze users’ behavior and memorize their preferences, which is key to creating a tailored and engaging user experience (Line 412). Specifically, we evaluate the model’s ability to memorize user information in conversational contexts. We will carefully revise the wording in future versions to ensure that our claims align with the specific scope of our study.

Q5: How does GenerativeAdapter perform in scenarios where the context information is dynamic or frequently changing, such as real-time conversational agents? It would be interesting to see if the method’s efficiency and adaptability hold in continuously evolving contexts.

Our experiments on document-based QA are designed to address scenarios with dynamic context. For instance, when the context length extends to 32K tokens, we dynamically split the context into 1K chunks and update the generative adapter incrementally, as described in Section 2.2. This approach demonstrates our method’s ability to handle evolving contexts efficiently. We appreciate your suggestion to further explore real-time conversational agents. However, as we are not aware of a suitable dataset for such scenarios, we will consider this as a direction for future work.

Thank you once again for your thoughtful questions and valuable feedback! Please don’t hesitate to let us know at your earliest convenience if you have any further questions, concerns, or if there are additional experiments you would like us to conduct.

Please don’t hesitate to let us know if you have any additional questions or need further clarification. We’d also love to hear if we’ve addressed all your concerns. Thank you again for your thoughtful feedback!

If the updates and clarifications we have provided address your concerns and improve the quality of the work to your satisfaction, we kindly ask you to consider reflecting this in your overall assessment. We greatly appreciate your feedback and efforts in reviewing our submission.

We appreciate your valuable questions and suggestions! Below, we provide responses to your questions and address the identified weaknesses.

Q1: Why is the continual pretraining baseline so much lower than the generative adapter baseline?

First, we would like to emphasize that vanilla continual pretraining has been shown to be less effective for enabling LMs to acquire knowledge for downstream tasks, as reported in the literature [1, 3]. This limitation motivates various efforts, including our work, to address this gap. Two hypotheses have been proposed to explain this phenomenon:

1. The unweighted next-token prediction loss is not optimal for pretrained LMs [1].
2. Additional (e.g., synthesized) data about each fact is necessary [3, 4].

Second, our continual pretraining baseline follows the setups from prior studies [1, 2], with sufficient tuning. QA accuracy results on SQuAD and StreamingQA align well with those reported in earlier work. Below are the results for comparison:

• [1] Hu et al. (2023) Meta-Learning Online Adaptation of Language Models
• [2] Tack et al. (2024) Online Adaptation of Language Models with a Memory of Amortized Contexts
• [3] Zhu et al. (2023) Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
• [4] Yang et al. (2024) Synthetic continued pretraining

Q2: Ultragist should be reported in sections 4.1 and 4.2.

While Ultragist can be compared, we excluded it from sections 4.1 and 4.2 for the following reasons:

1. Ultragist is a lossy version of in-context prompting. As a context compression method, Ultragist can be seen as a lossy version of in-context prompting, which represents the upper bound of performance. Our primary focus is to show whether our method can close the gap with in-context prompting. To address the reviewer’s concerns, we evaluated Ultragist (with a compression limit of 256 tokens) on the MetaICL dataset, and the results are reported below. The findings show that Ultragist performs worse than the in-context prompting baseline. | Method | Classification | Non-classification | |-----------------------|----------------|---------------------| | Ultragist (256 tokens) | 41.1 | 7.5 | | In-context prompting | 60.5 | 6.7 | | Finetune | 71.8 | 10.5 | | Generative Adapter | 63.7 | 14.9 |

2. Ultragist's performance depends on the compression ratio. In Section 4.3, we tested Ultragist across a range of compression ratios, from 256 tokens to 2K tokens. In contrast, Sections 4.1 and 4.2 focus on evaluations involving varying context lengths (Section 4.1) and numbers of examples (Section 4.2), making a fair comparison challenging.

Q3: llama-2-7B doesn’t provide much value and llama-2-7B should be replaced by llama-3.1-8b.

We chose Llama-2 because it was one of the most widely used open-source LLMs at the time of submission. Its differences from Mistral—such as being developed by a different company with distinct pretraining strategies (data, vocabulary, and context window size)—make it ideal for testing the robustness of our method. The Llama3.1 model family, released in July 2024, was too close to our submission deadline for inclusion.

Our results on Mistral-7B and Llama2-7B effectively demonstrate the strength of generative adapters. On both models, generative adapters outperformed continual pretraining in knowledge acquisition efficiency, achieving better results with shorter contextualization times. Furthermore, the generative adapter significantly narrowed the performance gap between continual pretraining and in-context prompting on Mistral-7B for sequences up to 32K tokens.

To address your concern, we provide below the results of in-context prompting on Llama3.1-8B-Instruct using the StreamingQA dataset. These results show a similar trend to Mistral-7B-Instruct. We speculate that comparisons between generative adapters and other baselines on Llama3.1 would align with those observed on Mistral.

Q4: I think a valuable baseline (especially since continual pretraining seems to underperform) might be using the generative adapter loss function (Eqn 5) but optimize the weights of the model directly instead of the generative adapter.

As mentioned in our responses to Q1, continual pretraining with next-token prediction is a common approach for domain adaptation but is not very effective for injecting new knowledge into model parameters.

The loss function in Eqn 5, designed for training the adapter generator by mapping text sequences to adapter matrices, combines completion and reconstruction components. Using it directly for continual training could cause model collapse, e.g., the LM simply repeats the first half of the input. However, your suggestion points to an important research direction: developing better training objectives for knowledge acquisition, which could bring significant improvements.

Q5: how would they contextualize similar work using hypernetworks for continual learning

Thank you for pointing out these Hypernetwork papers. While they are related to fast weights, they differ from the fast weights concept discussed in our paper. Due to space constraints, we excluded them from the initial version.

Hypernetworks, such as those in [1][2], are variants of [3], which we cited. However, these methods train the hypernetwork and backbone jointly, unlike our approach. They also focus on specific tasks, whereas we aim to enhance frozen pretrained LMs for general language tasks. Our adapter network, once trained, supports diverse scenarios with varying context characteristics.

Most hypernetwork-based methods [1][2] target supervised improvements in in-context learning but do not address recalling document knowledge or user facts for tasks like question answering.

We will include this discussion in the final version.

• [1] Vladymyrov et al. (2024) Continual HyperTransformer: A Meta-Learner for Continual Few-Shot Learning
• [2] Oswald et al. (2022) Continual Learning with Hypernetworks
• [3] Ba et al. (2016) Using Fast Weights to Attend to the Recent Past

Thank you once again for your thoughtful questions and valuable feedback! Please don’t hesitate to let us know at your earliest convenience if you have any further questions or concerns, or if there are additional experiments you would like us to conduct.

Please feel free to let us know if you have any additional questions or need further clarification. We’re also curious to know if we’ve addressed all your concerns. Thank you again for your valuable feedback!