Differentially Private Steering for Large Language Model Alignment

We thank the reviewer for their careful review of the paper, their comments, and their positive assessment of our work. We agree with the reviewer that protecting the privacy of the very small alignment datasets is an emerging problem, and we hope our simple but performant algorithm is a useful solution to this problem.

Q. “simple application of Gaussian mechanism”

A. We believe the relative simplicity of our contribution is a strength. We observe strong results across four model families and seven benchmarks, motivating our approach as a reliable and simple solution which has not been explored in literature before. Additionally, our approach is easy to implement and more likely to be adopted in practical uses because of its simplicity.

Q. “bar plot and number of examples used”

A. We thank the reviewer for pointing this out and will update the plots in the draft to reflect the error bars. We have provided the standard deviation for our results with the Llama model family below:

We will need some additional time to also run these experiments for the other model families but we expect those results to be similar. We will update the revised draft with those numbers. The number of examples in each dataset are reported in Table 2.

Q. “Is it expected that alignment can improve performance on MMLU?”

A. Investigating the reasoning performance of aligned LLMs is a significant area of contemporary research [1,5]. Our results are in line with similar works [1,3] that show reasoning gains with aligned LLMs. (See Table 5 in [3]). Wang et. al. [1] discover that alignment is crucial for improving reasoning performance of LLMs on math and commonsense datasets while Luo et. al. [2] show that using preference (alignment) data improves the mathematical competence of LLMs.

Q. “For some of the datasets, it seems like delta would be fairly large?”

A. The reviewer is correct in pointing out that $\delta$ can be large when the dataset size $n$ is too small. We are guided by the principle that the parameter $\delta$ is typically set to be less than $1/n$ to ensure meaningful privacy guarantee. This can be explained by the “name and shame” algorithm [4], which outputs each data point with probability $\delta$. Although this algorithm is $(0, \delta)$-DP, the probability of releasing at least one data point is $1-(1-\delta)^n\approx \delta n$. When $\delta \gg 1/n$, this guarantees at least one data point will be released, making the privacy guarantee meaningless. Therefore, we set $\delta = 1/5n$ instead of a fixed value to account for the varying sizes of different datasets. In our experiments, the smallest dataset size is 290 (Table 2), resulting in a sufficiently small $\delta \leq 0.0007$. For such small datasets, making the delta even smaller leads to unreasonably large loss in utility.

[1] Wang, Peiyi, et al. "Making large language models better reasoners with alignment." arXiv preprint arXiv:2309.02144 (2023).

[2] Luo, Haipeng, et al. "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct." arXiv preprint arXiv:2308.09583 (2023).

[3] Panickssery, Nina, et al. "Steering llama 2 via contrastive activation addition." ACL 2024

[4] Smith, A. “Lecture note of BU CS 591 Algorithms in Society: lecture 9 and 10. ” 2020.

[5] Jiang, Ruili, et al. "A Survey on Human Preference Learning for Large Language Models." arXiv preprint arXiv:2406.11191 (2024).

We thank the reviewer for their careful review and the two sets of specific questions. We are also encouraged by the scores given by the reviewer on Soundness, Presentation, and Contribution. We hope the answers below along with the new experimental data satisfactorily answers the reviewer’s queries and they will consider increasing their scores.

Answer to Q1 - clarification on the sensitivity of our algorithm

A. We thank the reviewer for raising this important question. In algorithm 1, $C_l$ is an input parameter to the algorithm, and the sensitivity of the Gaussian mechanism is $\Delta_f = 2/n$. We clarify as follows.

We note that $C_l$ serves as an estimated upper bound for the norm of difference vectors, similar to the clipping parameter in DP-SGD [1]. Our approach first clips the difference vector to $C_l$ and then divides this clipped vector by $C_l$, ensuring the resulting vectors have norms less than 1. Hence, the sensitivity $\Delta_f$ for mean estimation of these vectors is upper bounded by $2/n$, since changing a single element will only affect the output by at most 2, given that the norm of $\bar{d}_i^l$ is constrained to be less than 1.

Additionally, there is a typo on line 227 in algorithm 1. The equation should read $\bar{d}_i^l = d_i^l / \max\{C_l, ||d_i^l||_2\}$. This equation can be obtained by combining the clipping and normalization steps described above: first, clip the difference vector to $C_l$, which can be done by $d_i^l = d_i^l / \max(1, \frac{||d_i^l||_2}{C_l})$; then, divide the clipped difference vector by $C_l$ to ensure that $\bar{d}_i^l$ has a norm smaller than 1.

Answer to Q2 - per layer $\epsilon$ in $\sigma$:

A. The per-layer $\epsilon$ is $\frac{2\sqrt{2\ln 1.25/\delta}}{kn\sigma}$, where $\delta$ represents the overall $\delta$. As described in lines 258–261 of our paper, we add the same amount of noise to each layer, ensuring that the activation editing on each layer is $(\epsilon/k, \delta/k)$-DP. Since we edit a total of $k$ layers, the composition properties of differential privacy imply that the overall process is $(\epsilon, \delta)$-DP. We use basic composition instead of advanced composition of differential privacy, as basic composition provides tighter bounds when $k$ is small.

In our experiments, $k = 5$ as we apply activation editing on 5 middle layers. The values are summarized in the table below.

Q. “More descriptive Section 6. Need info on canary selection”

A. We have added Section D.2 to describe the MIA in more detail. Here we provide a brief description.

1. Canary Selection: We create a set $\mathcal{S}$ of canary words that are gibberish yet plausible, each 6-7 letters long and starting with the same letter. This approach follows prior work [14,15] and uses realistic nonce words \citep{malkin2021gpt} to simulate practical data leakage.
2. Canary Pairing: From $\mathcal{S}$, we randomly select three canaries: one anchor $a$ and two targets $t_1$​ and $t_2$​. These form pairs $z_1=(a,t_1)$​ and $z_2=(a,t_2)$.
3. Attack Procedure: In each MIA trial, we randomly insert either $z_1$ or $z_2$ into the dataset alongside benign samples to create the steering vector (let’s say $z_1$ is inserted). We then prompt the LLM $N$ times with the anchor $a$ and count how often the target $t_1$ appears in the outputs. If the count exceeds a threshold $\tau$, we infer that $z_1$​ was used, indicating membership.

Q. “clarify PCA steering

A. We follow the algorithm introduced in [9] as the PCA setting. In this setting, the last token activations of the negative and positive demonstrations are computed and their difference vector is stored. The difference vector approach is used in existing steering works like [3,4,5,6]. Instead of averaging the difference vector over all samples (which is the case in mean steering), PCA steering computes the top principal direction (PCA with k=1) of the difference matrix. Then this top principal component is used as the steering vector. For more details see Section 3 in [9]. If the reviewer prefers, we will include this pseudocode in the appendix of our paper as well.

[1] Feng, Qizhang, et al. "Exposing privacy gaps: Membership inference attack on preference data for llm alignment." arXiv preprint arXiv:2407.06443 (2024).

[2] Stolfo, Alessandro, et al. "Improving Instruction-Following in Language Models through Activation Steering." arXiv preprint arXiv:2410.12877 (2024).

[3] Arditi, Andy, et al. "Refusal in language models is mediated by a single direction." arXiv preprint arXiv:2406.11717 (2024).

[4] Cao, Zouying, Yifei Yang, and Hai Zhao. "Nothing in excess: Mitigating the exaggerated safety for llms via safety-conscious activation steering." arXiv preprint arXiv:2408.11491 (2024).

[5] Price, Sara, et al. "Future events as backdoor triggers: Investigating temporal vulnerabilities in llms." arXiv preprint arXiv:2407.04108 (2024).

[6] Ball, Sarah, Frauke Kreuter, and Nina Panickssery. "Understanding jailbreak success: A study of latent space dynamics in large language models." arXiv preprint arXiv:2406.09289 (2024).

[7] Seyitoğlu, Atakan, et al. "Extracting Unlearned Information from LLMs with Activation Steering." arXiv preprint arXiv:2411.02631 (2024).

[8] Parikh, Rahil, Christophe Dupuy, and Rahul Gupta. "Canary extraction in natural language understanding models." ACL 2022

[9] Liu, Sheng, et al. "In-context vectors: Making in context learning more effective and controllable through latent space steering." ICML 2024

[10] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

[11] Yang, An, et al. "Qwen2 technical report." arXiv preprint arXiv:2407.10671 (2024).

[12] Carlini, Nicholas, et al. "Extracting training data from large language models." 30th USENIX Security Symposium (USENIX Security 21). 2021.

[13] Carlini, Nicholas, et al. "The secret sharer: Evaluating and testing unintended memorization in neural networks." 28th USENIX security symposium (USENIX security 19). 2019.

[14] Zanella-Béguelin, Santiago, et al. "Analyzing information leakage of updates to natural language models." ACM CCS 2020

[15] Millière, Raphaël. "Adversarial attacks on image generation with made-up words." arXiv preprint arXiv:2208.04135 (2022).

Q. “more experiments - different clipping bounds, varying noise levels, different LLMs”

A. As suggested by the reviewer, we have conducted the following experiments to strengthen our empirical findings. The additional experiments have also been added to the paper in Figure 3. New results of the ablations are added in Figure 5 and Appendix D.1

1. Results with language models of varying structures: We report the results comparing PSA and non-private Mean steer across all benchmark datasets with two more models belonging to diverse model families - Mistral and Gemma. We already report the results with Llama and Qwen in Section 5.2.1.

Our findings with with Mistral and Gemma also show low utility degradation due to PSA in line with our observations in Figure 3.

2. Performance across varying noise scales: Here, we report the results across all benchmark datasets with Llama-2 7B to investigate the impact of increasing noise on the alignment performance.

As expected, increasing noise leads to larger drops in utility.

3. Sensitivity analysis to different clipping bounds: Here, we report the impact of increasing the clipping threshold on the performance of Llama-2 7B across all benchmark datasets.

We observe that at the same noise level, larger clipping threshold leads to a drop in utility. The clipping threshold here is similar to that in DP-SGD and influences utility in two ways listed below, and our experiments show that usually the first effect dominates:

i. Larger thresholds increase effective noise: While our algorithm adds the same noise to the model regardless of the threshold, the vectors are divided by the threshold before noise addition. Therefore, a larger threshold effectively reduces the signal-to-noise ratio, thereby decreasing utility.

ii. Smaller thresholds introduce bias: When the clipping threshold exceeds the maximum norm of the difference vectors, no clipping occurs, preserving the original distribution of the vectors and leads to an unbiased estimator. In contrast, when the clipping threshold is small, only the larger vectors are clipped, altering the distribution and introducing bias into the mean estimator, which also decreases utility.

We thank the reviewer for their careful review and raising concerns and comments. We address each of them below. To answer some of the queries, we have also reported new experimental evaluations in particular: a) results with two additional LLMs (Mistral and Gemma), b) ablation to study the impact of i) noise and ii) clipping factor.

Q. “why alignment is a privacy concern? ”

A. We thank the reviewer for raising this important point. We believe LLM alignment is an emerging research area which deserves careful privacy considerations.

1. Multiple recent works have highlighted privacy concerns of LLM alignment data. In particular, Feng et. al. [1] showed how to conduct Membership Inference Attacks on DPO and PPO-based alignment for LLMs. In another recent work, Seyitoğlu et. al. [7] showed how to extract memorized data from unlearned LLMs with steering vectors. These works suggest that this is an emerging area of research interest which our work aims to provide a solution for.
2. Apart from the interesting research question, several real-world situations require the alignment data to be private.
   1. In L079-086 we introduce a real-world scenario which can potentially pose privacy risks in deployed LLMs - A hospital using patient history to align LLMs via steering for generating contextually relevant responses (precision medicine). Such healthcare data must necessarily remain private unless special consent is given.
   2. Industries spend huge resources to collect expensive, high-quality hand-curated alignment data which is generally not released publicly [10,11]. Being able to maintain their private control over this data while aligning their LLMs, will lead to wider adoption of LLM alignment algorithms.
3. Besides LLM alignment, steering methods are also increasingly useful for LLM instruction-tuning [2], safety[4], refusal [3], jailbreaks[6], backdoor triggers [5], and unlearning[7]. Private data can be used in any of these diverse applications which warrants a careful look at the privacy implications of steering algorithms, hence the focus of our current work.

We will include this discussion in the revised draft.

Q. The experiments, while indicating potential, currently feel more like proof-of-concept tests rather than a comprehensive study that would meet the standards of a top-tier conference.

A. We respectfully disagree with the assessment that our work is a proof-of-concept test and not a comprehensive study. In particular, Reviewers R52E and DS5g have acknowledged our comprehensive experiments conducted across seven benchmark datasets in our study. We have provided results on seven benchmarks exhibiting real-world behaviors, four model families of state of the art open-source LLMs, on i) alignment evaluation ii) MMLU iii) scaling across model size . This is a significantly comprehensive empirical study, containing more experimental evaluations than other works in this field. Nevertheless, we have provided more ablations and sensitivity analysis in the rebuttal which we hope will address the reviewer’s concerns.

Q. “straightforward application of DP, limited technical contribution”

A. While we agree that the algorithm is simple, we believe this is a strength of our work and not a weakness. Similar simple solutions have proven effective in related domains. For example, the original non-private steering vector, which is a relatively simple application of averaging representations and adding it to the trained network, usually outperforms more complex techniques like PCA-based steering vector..

In this particular case of private steering, we would like to highlight that

1. Our approach already delivers strong and consistent results across four model families and seven benchmark datasets. Hence we believe it is not always helpful to develop complex algorithms when a simple solution does not already exist in literature but performs well (as our experiments show).
2. Simpler algorithms also have other benefits. Our approach is easy to implement and is more likely to be adopted in practical and real-world scenarios. It is also easier to analyse and more likely to be free of bugs.

Overall, we do not think this is a weakness of the paper but rather a strength that such a simple application of DP works consistently well across a wide variety of benchmarks.

Q. “zero-shot” line missing in one case

A. We thank the reviewer for pointing this out. This was an oversight and has been corrected in the updated draft. The zero-shot results for AI Coordination with Llama-2 and Qwen-2.5 are 24.2 and 9.5, respectively.

Dear reviewer L9b9, we deeply appreciate the time and effort you invested in reviewing our paper. Since the discussion period closes soon, we hope our response and additional experiments address all your concerns appropriately and that you will reconsider the score accordingly.

We look forward to further discussion on any questions or comments you still have. Please let us know.

We thank the reviewer for their thoughtful feedback on our submission. Unfortunately, we were unable to locate the specific reference for Wu et al. 2024. Could you kindly provide additional details, such as a title, or a link? This would help us better address your feedback in our rebuttal. We will be happy to report additional results during our discussion.

Q. “It would also be better to provide the privacy parameters \sigma to improve reproducibility… It’s unclear whether the $\epsilon=2$ in Table 2 is for all five layers or just a single layer.”

A. In our experiments, we set $\sigma = 0.02$ and provide the epsilon values in Table 2, and the computation of per-layer $\epsilon$ in terms of $\sigma$ and per-layer $\delta$ is presented in L233 of Algorithm1. We set per-layer $\delta$ to be $\frac{1}{5n}$, which results in an overall $\delta = \frac{1}{n}$. We have added Table 17 and Figure 5 in the updated draft to illustrate the effect of sigma on the LLM performance.

Also, we modified Table 2 to include the per-layer epsilon as well as the total epsilon for each dataset.

Q. “tight privacy estimation. Nasr et al 2021 had to design gradient canary and train 10k models to achieve such tight analysis”

A. The privacy preserving mechanisms audited by Nasr et. al. (DP-SGD) and us (PSA) are very different and thus they require different auditing algorithms. Our algorithm involves a single noise addition to a one-step optimisation process (mean), in contrast to the multiple randomized iterations (shuffled stochastic batch gradient descent) in DP-SGD. Moreover, the steering vector in our case is typically computed using a smaller dataset, as compared to the size of training sets for DP-SGD. These factors potentially make the steering vector more susceptible to memorization than DP-SGD. Other recent work [8] has also provided evidence of a successful memorized data extraction attack for steering vectors.

[1] Liu, Sheng, et al. "In-context vectors: Making in context learning more effective and controllable through latent space steering." ICML 2024

[2] Panickssery, Nina, et al. "Steering llama 2 via contrastive activation addition." ACL 2024

[3] Qiu, Yifu, et al. "Spectral Editing of Activations for Large Language Model Alignment." NeurIPS 2024

[4] Wang, Haoran, and Kai Shu. "Backdoor activation attack: Attack large language models using activation steering for safety-alignment." CIKM 2024

[5] Cao, Zouying, Yifei Yang, and Hai Zhao. "Nothing in excess: Mitigating the exaggerated safety for llms via safety-conscious activation steering." arXiv preprint arXiv:2408.11491 (2024).

[6] Cao, Yuanpu, et al. "Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization." arXiv preprint arXiv:2406.00045 (2024).

[7] Loshchilov, Ilya, et al. "ngpt: Normalized transformer with representation learning on the hypersphere." arXiv preprint arXiv:2410.0113 1 (2024).

[8] Seyitoğlu et al. “Extracting Unlearned Information from LLMs with Activation Steering.” arXiv preprint arXiv:2411.02631 (2024).

We thank the reviewer for their careful review and the two sets of specific questions. We are also encouraged by the reviewer's scores on Soundness, Presentation, and Contribution. We hope the answers below, along with the new experimental data, satisfactorily answer the reviewer’s queries, and they will consider increasing their scores.

Q1. “One non-private baseline is missing. …. It would be helpful to understand the clipping impact. ”

A. As we understand it, the reviewer is asking how the clipping affects the whole algorithmic pipeline independent of the added noise. We thank the reviewer for this interesting question as clipping indeed plays an important role here in both i) improving the utility when no noise is added and ii) allowing for a smaller epsilon by controlling the sensitivity as in classical additive noise mechanisms. To illustrate this, we a) run PSA with different clipping thresholds for $\sigma=0$ as requested by the reviewer and b) provide intuition on why less restrictive clipping is useful as well as cite related works with similar evidence.

1. Impact of clipping when $\sigma=0$: Here, we provide the results with Llama-2 7B across all benchmark datasets in the noiseless setting (sigma = 0). This is a similar setting as the one reported in Wu et al. 2024.

We observe that increasing the clipping threshold leads to a decrease in utility.

2. Intuition for why clipping improves performance Here, we provide some intuitive explanations for why this might be the case. Clipping and Normalisation (L227 in Algorithm 1) involve two steps: first, the difference vector is clipped to $C_l$, using $d_i^l = d_i^l / \max(1, \frac{||d_i^l||_2}{C_l})$; then, the clipped difference vector is divided by $C_l$ to ensure that $\bar{d}_i^l$ has norm smaller than 1.
   1. Step 1: smaller thresholds introduce bias: When the clipping threshold is larger than the maximum norm of the difference vectors, no clipping occurs, preserving the original distribution and keeping the estimator unbiased. If the threshold is small, larger vectors are clipped, altering the distribution and introducing bias into the mean estimator, which reduces utility.
   2. Step 2: threshold controls steering vector influence: In a noiseless setting, the final model parameter is the original parameter plus $\frac{1}{C_l}$ times the averaged clipped vectors. The threshold $C_l$ controls the influence of the steering vector. A larger $C_l$ reduces the steering vector's impact, which can either improve or decrease utility.

Based on our experiments, we observe that larger clipping thresholds lead to utility degradation in general, showing that the normalization step dominates. We have added this experiment to the paper (Section E.1) as part of the ablation study.

3. Past work on Clipping for non-private training: Previous work [7] shows that normalizing LLM representations to unit norm improves performance, increases embedding space separability, and speeds up training convergence. Figure 4 in [7] demonstrates that normalized transformer representations are better conditioned, leading to better convergence. Although this differs from our setting, it supports the idea that normalized representations improve performance in the LLM setting. The above experiments (clipping with $\sigma=0$) show similar benefits for non-private steering. To the best of our knowledge, our work is the first to show this benefit for normalized against unnormalized steering vectors, which could be explored in future research.

Q. “References for mean steering and other steering methods”

A. In L120-121, we provide references for several steering methods in the literature [1,2]. More recent and contemporaneous works that use steering methods include [3,4,5,6].

Q. “provide the codebase to improve reproducibility”

A. We agree and acknowledge the importance of reproducibility of results. We plan to publicly release all code, datasets and private steering vectors used in this study.