Aligner: One Global Token is Worth Millions of Parameters When Aligning LLMs

Thank you your valuable feedback. We would like to clarify the following points:

1. Showing only one qualitative example in the main paper:

Apologies for the confusion. Due to the page limit we show only one example in the main paper, but we've included many more examples in a supplementary appendix. We've also now added many failure cases and winning cases along with GPT-4'sjudgements.

We also did not cherry pick our examples. The quantum physics question is commonly used, so we chose to present it in our main paper. In general, the SFT results quality is stable.

For the value alignment experiments, we did a bit of picking, but only to avoid the unsafe content that comes not only from Aligner but also from LoRA, which appears more often. We had already included plenty of examples in the appendix, and we've now added more failure and winning cases. You can peruse our supplementary appendix to get a sense of that. Please let us know if you want us to include additional examples in the main paper and/or more examples in the supplement.

2. The "comparable" claim for Aligner-1-Token & analysis of the failure case:

We toned down our claim, and we also provided samples of failure cases in an appendix.

3. "... in practice no one wants to deploy a technique that worsens previous results, perhaps even if there's an win in parameter efficiency."

This isn't always the case. For industrial scenarios where one needs to provide a customized model to every different user (such as assuming a unique character or replying in a customized format), the parameter efficiency is very important when scaling up to tens of thousands or even millions of customized adapters. For Aligner, even with one million adapters, that's just 5 billion extra parameters (around 10G memory), which can still fit into a single 24GB memory GPU along with a 7B model (13G memory) using standard bfloat16 precision without quantization. By contrast, with LoRA or LLaMA-Adapters, this will consume terabytes with millions of adapters and it'll be necessary to offload to secondary storage, which would drastically slow down the inference time. So, our method could be very useful in saving costs. However, thank you for raising this concern as we must expand on this point in our paper. We agree that our "paper should've allocated more space for an investigation of the implications of this small change".

4. Sec 5.3 lacking quantitative results for the extended experiments using a single head:

Thank you for your criticism of this section, which we've deleted for the reasons mentioned in Item 3 of our above common comment.

5. Concern over "how the hypothesis of separation of form and knowledge can be empirically verified":

We believe that we've already largely verified it: The very structure of Aligner -- a minimally sized and globally orthogonal component, along with the fact that its efficacy is on par with other methods, confirms that form functions orthogonally to reasoning/knowledge in LLMs.

However, we do agree that we can further validate our hypothesis. We can also provide a negative example; i.e., for reasoning tasks, Aligner should not exhibit a similar advantage over other higher-parameter and non-global component tuning methods. Since math serves as the most representative reasoning task, we trained them on a math dataset (MetaMath) and tested their performance on a standard benchmark, GSM8K. Aligner indeed does not show a parameter advantage over other methods, thereby validating the hypothesis.

Please see Item 1 in our above common comment and Section 4.3 of our revised paper.

6. "The trade-off between quality and number of parameters for the new and the standard adapter would have been helpful to compare directly in a plot":

Thank you for your suggestion. In the added math experiments, we included a series of different sizes and showed their performance and size trade off in Fig 5 of the paper.

7. Our code will be open sourced after we tidy it up, but we can release a version quicker if this is deemed necessary.

8. Why does the 7B model perform better than the 13B model in Table 1? Any hypotheses?

This may be due to the variation of GPT 4 evaluation and/or due to the training dynamics of the model as a result of hyper-parameter choices. We didn't expend substantial effort to find optimal hyper-parameters, but they definitely matter to a certain extent, which may account for the slightly better performance.

Thank you for your valuable feedback.

We have added some standard benchmark results to address your questions and concerns. However, we would like to offer the following clarifications:

1. The rationale for using GPT as the judge:

First, this was the default evaluation method in the papers introducing both the Alpaca and Beaver datasets on which we trained Aligner. It was therefore natural to adopt their methods.

Second, to our knowledge, standard datasets like MMLU (general knowledge) or GSM8K (math) have precise correct answers and only care if the final answer is correct. They generally test the reasoning ability of a model rather than the form of its output. By contrast, for alignment tasks like Instruction Following and Value Alignment, which are the focus of our paper, the model outputs are open-ended and there is no single right answer. The "form" of the answer is of importance. So, in our case, the gold standard is human judgment. This is also why the default benchmark methods provided by Alpaca and Beaver are both model-based evaluations, since, despite its shortcomings, thus far a powerful model is the closest surrogate to human judgment for such tasks.

However, we agree that it would help in evaluating Aligner more completely if we include some standard benchmarks, but the purpose this can serve would be different. Rather than directly evaluating the "form" alignment task, it helps to see how our Aligner method affects the reasoning ability of the model in comparison to other adaptation methods, and to further test the hypothesis that form functions orthogonally from reasoning or knowledge. This experiment may also help better validate our hypothesis that form functions orthogonally to reasoning/knowledge in LLMs.

2. The standard benchmark results:

First, we added the MMLU evaluation that tests general knowledge without training over its dataset, but using the model finetuned on the Alpaca dataset only, along with the raw LLaMA2 base model's performance. The Alpaca dataset does help reasoning ability to some extent since it contains answers that have step by step reasoning and even some math and coding questions. However, it's primarily aiming to tune the model to follow instructions. Therefore, if form functions orthogonally to reasoning, Aligner should perform worse but there should not be a big gap. The results validate our hypothesis.

The evaluation results:

Second, more importantly, we trained different methods over the MetaMath dataset (a math dataset) on the 7B model and evaluated on GSM8K. This was to assess how much reasoning improvement different methods can attain along with their different parameter sizes. If form functions orthogonally to reasoning in LLMs, Aligner should not have an advantage over other methods, unlike in form alignment tasks (Instruction following and Value alignment). The results validate our hypothesis. Aligner only catches up the performance of LLaMA-Adapter when they have a similar number of parameters.

If the above rationale and plan are unsatisfactory and you have something else in mind, please let us know what specific evaluation(s) you would like us to perform.

3. Regarding the single-head Aligner experiments:

Thank you for your criticism of this section. We have deleted the section for the reasons explained in Item 3 in our above common comment.

4. You are right that the LLaMA-Adapter and Aligner is not strictly a prefix token method. We have changed the wording to be more rigorous in the Related Work section.

Thank you for your valuable feedback.

We once plotted the distribution of the parameter values of the token, but it is essentially Gaussian and provides no useful insights.

Unfortunately, other common methods like linear classifier probing \cite{linearProbe} are also inapplicable: it requires not only numerous datapoints but also labels for them in order to train this classifier. In our case, we obtain only 1 set of tokens from training over the entire dataset.

The only meaningful approach seems to be to compare the Aligner token with the LLaMA-Adapter tokens using t-SNE or directly. This could potentially reveal how our globally-shared token relates to the local tokens. We have conducted several comparisons and added the results to the main paper and the supplementary appendix. Here's a list of findings:

1. The Aligner 1 Token embedding is not necessarily the "average" of Aligner 10 Token or the LLaMA-Adapter, which has 300 tokens.
2. When we train the model with a different dataset, the embeddings of Aligner and those of the LLaMA-Adapter remain very close to each other. In fact, approximately half of the numbers in the embeddings are exactly the same, and the majority of the rest are still very close. This shows that very little change is needed to dramatically adapt the behaviors of an LLM.
3. The standard deviation of the gating factors in a layer is increasing as the layer goes to top, for both Aligner and LLaMA-Adapter. This may be in align with our general intuition that top layers generally requires larger adaptations.

The figures are shown in section 4.4, figure 6 in the main paper and Appendix D in the supplement. We also put some analysis that doesn't yield any finding in Appendix D that you can take a look at. We would be pleased to perform any other suitable data analysis that you can recommend to us.