Spectral Editing of Activations for Large Language Model Alignment

1. Comment: There is no significance testing for benchmark scores.

Response: On TruthfulQA, we did the pair-wise t-test on SEA vs ICL baseline (in Table 1). We also confirm that SEA significantly outperforms LoRA-FT. We did not add more significant tests over other baselines as some of the results are from [1]. On BBQ (Figure 3), we also perform a pair-wise t-test and show the significance of the improvements of all SEA's variants over ICL and LoRA-FT.

[1] Alleviating Hallucinations of Large Language Models through Induced Hallucinations

2. Comment: The paper would be stronger if more evaluation benchmarks were considered for bias and truthfulness

Response: First, TruthfulQA and BBQ are two popular benchmarks for evaluating truthfulness and fairness. TruthfulQA is almost used in all papers to improve LLM's truthfulness, and BBQ is used for fairness evaluation for Gemma, Mixtral, and PaLM.

For truthfulness, we would like to highlight that we use HaluEval [1] to calculate the editing projections and evaluate our method with other baselines on TruthfulQA. This allows us to compare with other methods on public benchmarks and also verifies the task generalization ability of SEA editing from one dataset to another.

For bias evaluation, we further conduct one evaluation on CrowS-Pairs [2], which assesses the model's tendency to generate biased outputs, as an additional evaluation to the editing for fairness. We report the percentage of more-stereotypical sentences (lower is better) that are rated as more likely by a model than the non-stereotypical sentences as follows. We observe that both variants of SEA can reduce the tendency for outputting biased sentences for most bias categories. We would like to emphasise that Phi-SEA reduces the tendency for generating more stereotypical sentences with less by 7%. All these observations are consistent with our results on BBQ in Section 4.2.

[1] HaluEval: A Hallucination Evaluation Benchmark for LLMs [2] CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models

3. Comment: Why are results for other models (families) shown for BBQ but not for TruthfulQA? And why are scalability results for TruthfulQA shown only for MC1 and not for MC2?

Response: We will add the following additional results to the camera-ready version.

Model family generalisation: We include the results for model generalisation on TruthfulQA as follows. SEA improves all LLMs on MC1. Please note that due to time constraints, we were unable to perform an extensive hyper-parameter search.

Scalability results: MC1 is the most direct metric to measure whether the model predicts the best answer in TruthfulQA. But we will also add the scaling results for MC2 as follows:

4. Comment: Could you explain why your toxigen scores differ so drastically from the scores reported in the Llama2 paper?

Response: Our evaluations are not directly comparable. In LLaMA's technical report, they report the percentage of generations that are deemed toxic by the metric; however, we follow the lm-evaluation-harness that Toxigen is formulated as asking the model to label if a given statement is hateful or toxic. The rationale behind our evaluation is that a safer or less toxic model should be more capable of identifying the safe/unsafe response. Additionally, we also include an extra fairness evaluation on stereotypical generation as discussed in Comment 2.

5. Comment: Can your method make the model more truthful and unbiased at the same time? Can this approach scale to a method that takes care of both?

Response: This is indeed an interesting question. We conduct an extra experiment on merging the positive and negative demonstrations for both truthfulness and fairness, then apply the same SEA editing procedure to calculate a pair of projection matrices jointly editing for truthfulness and fairness on LLaMA-2-Chat-7B. Compared with LLaMA-2-Chat-7B, we found that a joint projection can improve both fairness and truthfulness.

However, compared with the editing for a single target with the same number of demonstrations, the effect of joint projection is not as effective as specialised editing. We think the potential reason may be that the editing direction and degree of truthfulness and fairness may be different, which can be seen from the spectrum of the covariance of the activation values on HaluEval and BBQ (Fig1 in the additional rebuttal PDF). Thus, mixing the two goals for editing might lead to mutual interference to some extent.

1. Comment: I suggest that the author compare SEA with TrFr and TruthX to highlight the novelty of the proposed method.

Response: We will add them to the related work and provide a comparison. We agree the general motivations for SEA and TruthX are similar, but there are many differences:

1. methodology: Our methods are entirely different. We use spectral decomposition to search and apply the editing directions, while TruthX and TrFr require contrastive learning or probing to find them. Ours allows the editing projections to be calculated via a closed-form solution and has the advantage of training efficiency.
2. fairness editing: we extend the linear method to non-linear editing for fairness with three kernels. The results of the biasing evaluation confirm the advantage of SEA.
3. efficiency: SEA is a very lightweight method that complements existing methods. As shown in Table 1, thanks to SEA’s closed-formed solution, its training time is significantly lower than that of gradient-based editing methods. This will make SEA a meaningful baseline for efficiency for gradient-based editing methods in the future.

2. Comment: Some baselines on representation engineering, such as TrFr and TruthX, should be compared in the TruthfulQA experiment.

Response: Thanks again for suggesting these related works. We missed TruthX in our submitted manuscript because it was just accepted by ACL2024 which will happen on August 11. But in the camera-ready version, we would be happy to include them in the final result table to enable the community to better understand the progress in this field.

*3. Comment: It is not sufficient to use only multiple-choice tasks for TruthfulQA. *

Response: We also report the scores on the generation track. We use DaVinci-002 as the backbone for training GPT-Judge and GPT-Info as curie is not maintained by OpenAI anymore. To summarise, SEA has the highest truthfulness score among all methods. However, LoRA gets the best informativeness results, which is expected as LoRA fine-tunes the model using instruction-following data, unlike SEA.

4. Comment: The statement about "training-free" needs to be more rigorous. I prefer to call it "inference-time".

Response: Thanks. We agree that SEA still leverages demonstrations to calculate the editing projections. We will change the term to inference-time editing.

5. Question: Why do we use "activations at the last token position" instead of randomly selecting or taking the mean of all activations in the response?

Response: We follow previous work [1,2] by using the activations at the lask-token position, which shows effectiveness in capturing the model's internal states over the entire sequence.

As a way to provide evidence for this claim, we also run an ablation study on the choice of activations in the two ways you recommended:

[1] In-context Vectors: Making In-Context Learning More Effective and Controllable Through Latent Space Steering [2] Improving text embeddings with large language models

The result is as expected: using the last-position activations works best. Our explanation is that the completion is generally shorter than the prompt, especially for the QA task with short answers like TruthfulQA. Using mean pooling of all tokens from the whole sequence may over-amplify the signal from the prompt rather than the relatively short positive/negative completions. Also, as we are dealing with a decoder-only model, the tokens in the prompt cannot attend to the completion during encoding. So, using them to contrast the model's behaviours from the positive and negative completions would not be meaningful.

6.Question: Why use the output of each MLP layer as activation? Instead of using an attention head like ITI or attention and MLP like TruthX?

Response: Our main concern is efficiency. Attention has multiple heads and projections, which leads to 1) the number of hyperparameters increasing considerably, editing attention would require an understanding of the underlying mechanisms of each attention head [1,2], making it a more challenging and less applicable approach. 2) The decrease in inference efficiency: the complexity for editing each Transformer's block output would be O(L), but O(LxH) for editing attention, where L is the number of layers, and H is the number of attention heads.

Finally, the role of attention and MLP is still an open-ended research question. There is no absolutely correct paradigm, whether it is LoRA fine-tuning or representation editing. There are works [3-5] editing the transformer layers' outputs that also show promising performance. We suggest that users decide which place to apply the edits according to their needs and budgets.

[1] Retrieval head mechanistically explains long-context factuality

[2] Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions

[3] In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering

[4] Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection

[5] Erasure of Unaligned Attributes from Neural Representations

1. Comment: The paper could benefit from experiments on a broader array of tasks to further validate SEA's effectiveness across different contexts. This would help generalize the findings beyond the current benchmarks. How does SEA perform on other important NLP tasks not covered in this study? Extending the evaluation to a wider range of tasks could further establish its generalizability.

Response: In this work, we focused on truthfulness and fairness, which we perceive as critical attributes that enhance the usefulness of LLMs. We agree with the reviewer that while TruthfulQA and BBQ are the "go-to" benchmarks for such evaluations, our paper can benefit from experimenting with additional dataset.

For this purpose, we further conducted one additional evaluation on CrowS-Pairs [2], which assesses the model's produced embeddings' tendency to biased outputs. We report the percentage of more-stereotypical sentences (lower is better) that are rated as more likely by a model than the non-stereotypical sentences as follows. We observe that both variants of SEA can reduce the tendency for outputting biased sentences for most bias categories. We would like to emphasise that Phi-SEA reduces the tendency for generating more stereotypical sentences with less by 7%. All these observations are consistent with our results on BBQ in Section 4.2.

[1] HaluEval: A Hallucination Evaluation Benchmark for LLMs [2] CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models

2. Comment: Including visualizations of the distribution shifts in activations before and after applying SEA would provide more insight into the impact of the method and help understand the underlying mechanics. Can you provide visualizations of the activation distribution shifts before and after applying SEA? This would help in understanding the impact of the method on the internal representations.

Response: We have provided a visualization of Linear-SEA and Phi-SEA editing on BBQ in Figure 2 in the additional rebuttal PDF. We observe that Phi-SEA (right) removes the important directions of negative demonstrations while retaining the directions related to positives, which explains the editing qualitatively. We will include these visualizations in the revised manuscript.

3. Question: Could you elaborate on the choice of benchmarks and how representative they are of real-world scenarios where LLM alignment is critical?

Response: First, TruthfulQA and BBQ are two popular benchmarks for evaluating truthfulness and fairness. TruthfulQA is almost used in every papers to improve LLM's truthfulness, and BBQ is used for fairness evaluation for Gemma, Mixtral, and PaLM.

Secondly, both datasets cover a wide range of scenarios regarding truthfulness and fairness. TruthfulQA spans over 38 categories and focuses on evaluating the model's ability to generate factually accurate responses, which is essential for maintaining LLMs' credibility and reliability in real-world applications. BBQ is also hand-built and covers 11 types of common bias. It assesses biases in the model’s responses, helping ensure fairness and reducing harmful stereotypes.

Thirdly, these benchmarks are representative of real-world scenarios where alignment is crucial because their QA task formulation covers both the accuracy of the information seeking and safety/ethical considerations. This allows us to demonstrate the effectiveness of SEA in mitigating undesired model behaviours in a more real-world setup. The QA task formulation also helps us obtain the polarised positive and negative demonstrations for calculating SEA's projections.

1. Comment: Line 294 is a strong claim. As shown in Figure 4 BBQ, the performance plateaus. Consider re-phrasing.

Response: Thank you for your feedback. We agree that when comparing with results on TruthfulQA, the performance plateaus in Figure 4 for BBQ which suggests that the benefits of additional demonstrations may saturate beyond a certain point. We will rephrase line 294 to reflect this.

2. Comment: Performance on control tasks is generally convincing, but we need to be careful about applying this technique to common-sense tasks. The explanation given in lines 307-308 does not tell me why the lossy function does not apply to math tasks but only selectively to common-sense QA.

Response: For all linear editing variants, which are guaranteed to be lossless, we indeed observe a very minimal negative effect on other control tasks, including commonsense QA. Hence, the major factor for performance degradation is non-linear transformations. Regarding the distinct degradation for commonsense QA and other tasks in non-linear editing, one possible reason is that we apply the editing on MLP's outputs, which has recently been found to be highly associated with LLM's storing and recalling commonsense knowledge [1-3].

[1] Locating and Editing Factual Associations in GPT.

[2] Language Models Implement Simple Word2Vec-style Vector Arithmetic.

[3] Knowledge Neurons in Pretrained Transformers.

3. Question: While I understand that you are considering inference time methods as baselines, I'm curious to know how these methods compare to tuning-based methods like DPO.

Response: We would like to clarify that SEA does not aim to compare with the fine-tuning alignment, e.g., DPO/PPO. We would expect the inference-only editing lags behind the fine-tuning alignment, as we have discussed the large improvement of RLHF in Table1 and L221. Instead, we show that SEA can be applied on top of existing alignments, e.g., we apply SEA on LLaMA-Chat model aligned with PPO, while providing a lightweight (i.e., inference-only) and flexible (i.e., user can define the alignment objective by very few positive/negative demonstrations) control on model's output.

4. Comment: I am also surprised why the authors did not compare with Best-of-N alignment as a baseline which also does not require any training and is quite simple to compare against.

Response: First, we would like to clarify that we use the predicted likelihood of the candidate's answer to evaluate the model's truthfulness and fairness, making best-of-N baselines non-applicable for our original evaluation.

Second, our goal is to edit activations inside the model to control the model’s behaviour, which is orthogonal to the best-of-N method, i.e., the gains brought by SEA and the best-of-N method can be added together. Therefore, we conducted an additional round of experiments to further verify the effectiveness of SEA editing: we re-ran LLaMA-2-Chat-7B and its edited version of Truthful-SEA on the generation track of TruthfulQA under the best-of-N setup. We find that a larger N value leads to significantly higher scores in truthfulness and informativeness. In each N, SEA always has higher scores compared to the LLaMA baseline, except for the informative score when N=1. These experiments further consolidate the gains of SEA: it can be shown not only in the distribution of nucleus sampling but also extended to the best-of-N distribution.

Note that we follow previous work [1] to separately fine-tune GPT-3.5 as a truthfulness judge and as an informativeness judge.

[1] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

5. Question: What is a good working value for the hyperparam K (line 127)?

Response: It depends on tasks and models. We present the selected hyperparameters for all our experiments in Appendix D. We also present an analysis of the effect of different K in Appendix C. In the analysis, we show that increasing K will lead the downstream performance to increase and then decrease. Our explanation for the decrease in performance is that a very large K (i.e., including more explained variance from the positives while less from the negatives) might let projections capture the noisy signal in positive demonstrations while losing the task-related information in negative demonstrations.

6. Question: How do you enforce that k = r/2 where r is the rank of the matrix? Or is there no such constraint? Otherwise you end up double summing the values for overlapping directions.

• q1: It seems from table 1, that you are keeping top 99% and bottom 99% of explained variance. Does this not lead to double summation for activations?

Response: No double summation takes place. Assuming we have K=99%, SEA tries to keep the top 99% of explained variance for the positive covariance while removing the bottom 99% of the negative covariance. The positive covariance and negative covariance are also two distinct matrices, as calculated in Eqt(2), so the directions in their projected subspaces are distinct as well. We will clarify it better in the paper.

• q2: Also this makes me wonder if the spectrum of activations really decays exponentially?

Response: Yes, we observe exponential decays, as shown in Figure 1 in our attached additional rebuttal PDF. We present the spectrums for both covariances of the linear-SEA editing on truthfulness and Phi-SEA editing on fairness.