Analysing the Generalisation and Reliability of Steering Vectors

We thank the reviewer for their sincere appreciation of our work!

The reviewer’s main specific concern is about investigating whether low steerability is due to a nonlinear representation of a concept (as opposed to more mundane reasons). Generally, we think that this is a very difficult question to answer, and is largely beyond the capabilities of existing interpretability techniques. From a theoretical perspective, while we expect ‘true’ concepts to be represented linearly, the ‘true’ concepts used by a model could be very different from those used by humans. As we have no way of identifying this ground truth in language, this approach is largely intractable. From an empirical perspective, we think that the failure of steering vectors to work is sufficient evidence that the concept captured by the dataset is not linearly represented according to the standards set by other mechanistic interpretability works.

One hypothesis that might explain low steerability in our specific experiments is that we optimised for the layer hyperparameter only on a specific subset of datasets, and this choice may not be optimal for the remaining datasets. To investigate this, we re-ran the layer sweep using the datasets with the lowest steerability. Please refer to Fig 4 in the rebuttal PDF. We found that these low-steerability datasets still have the same optimal layer, largely disqualifying this hypothesis. Overall, this updates us towards believing that the main reason for steering vectors not working well is that the underlying concepts are not represented linearly by the model at any layer.

We are currently running additional experiments on Llama 3.1 70b, and plan to include this in the camera-ready version, but will not be able to include these results in the rebuttal version due to compute limitations. If the conclusions drawn in our paper also hold for that model, it will strengthen our argument that failure to steer is due to a nonlinear underlying representation (since we expect larger models to have better, more linear representations).

We hope our response has clarified all the concerns the reviewer had, and has made them more confident in recommending the acceptance of our paper.

Dear Reviewer,

As the end of the discussion period is approaching, we want to ask if our response has addressed your concerns regarding the paper, or if you have any further questions or comments about our paper. We appreciate the time and effort put into the review process and would love the opportunity to discuss this further if our response didn't address your concerns. Thanks again for your time reviewing our paper!

We thank the reviewer for their comments, and are glad to hear that they consider our work both timely and insightful.

We agree that open-ended steering is a more practical and useful setting, and we believe our work can be generalised to this setting with minor modifications. Steerability in our paper is defined with respect to an abstract ‘propensity’ quantity. In our paper, we use the logit difference as a measure of propensity; however, other measures could be considered. In an open-ended generation setting, steerability could be considered with respect to the difference of log-probabilities (which is a generalisation of logit-difference to multiple tokens). Alternatively, it may be possible to define propensity in terms of a score given by a ‘judge’ (human or automated) that evaluates how much the targeted behaviour was elicited. A challenge with the judge score is that the resulting propensity may not be a linear function of the multiplier and may be dependent on the specific setup of the judge. Overall, we think evaluating steerability in a more general, open-ended setting (and resolving associated technical challenges) is an exciting direction for future work, and will amend our camera-ready version to include this discussion. However, this is not in the scope of this paper, which still represents significant progress towards understanding and describing the limitations of steering vectors.

We also agree that it is often unclear why steering vectors do not work. Again, we believe this finding by itself has value, since previous work on steering vectors focuses on its success cases, giving an unrealistically optimistic view of steering vectors’ effectiveness. By highlighting some general failure modes, our work enables future research on investigating these failure modes in more detail, which may lead to novel insight. In addition, the novel ‘steerability bias’ that we discover is at least partially explained by spurious factors, as demonstrated in Fig 3. We think that a full explanation of ‘why’ a steering vector doesn’t work is beyond the current capabilities of the interpretability field, and that our work presents progress towards this understanding.

Overall, we believe our current work has multiple valuable contributions which the reviewer has not addressed. Notably, ours is the first work to empirically evaluate steering vectors in diverse settings, highlighting important challenges for using steering vectors and challenging the prevailing belief that steering vectors ‘just work’. We reveal important and general insights about steering vectors, such as the fact that steerability seems to be a largely universal (model-agnostic) property. Finally, our work lays the foundation for analysing steerability in the open-ended setting, which we think is an exciting direction for future work. We would appreciate the reviewer commenting in more detail on these contributions.

We hope our response has addressed all the concerns the reviewer had. If the reviewer has more specific concerns with our work, we would also appreciate the reviewer raising them now so that we can work to improve the paper with their valuable feedback.

Dear Reviewer,

As the end of the discussion period is approaching, we want to ask if our response has addressed your concerns regarding the paper, or if you have any further questions or comments about our paper. We appreciate the time and effort put into the review process and would love the opportunity to discuss this further if our response didn't address your concerns. Thanks again for your time reviewing our paper!

Thank you to the reviewer for their clarifying comments.

We first note that the CAA paper only proposes a methodology for extracting steering vectors from multiple-choice questions. Extracting steering vectors from open-ended prompts and completions remains an unsolved problem with no clear best approach. Among other things, it is unclear which token position to best extract the steering vector, and it has not been demonstrated that CAA will work similarly in this setting. We believe that significant methodological ablations must be done in order to first identify the best setting for open-ended steering extraction, which is not the focus of our work. Our paper's core contribution is an evaluation and analysis of the currently accepted best practice for extracting and applying steering vectors (CAA), which does not involve open-ended extraction. We would be excited to explore extracting steering vectors in an open-ended setting in future work.

We acknowledge the reviewer's concern that our paper does not provide a complete explanation for the failure cases of steering vectors. Our paper would certainly be stronger if it did have such an explanation. However, we think that 'lack of steerability' is a deep and fundamental phenomenon with no simple explanation, and that finding such a complete explanation in a single work is an unreasonable bar for acceptance. Our work makes progress towards understanding the limitations of steering vectors, uncovering the phenomena of steerability bias, as well as pointing out where previous results have overestimated the usefulness of steering vectors by only focusing on the mean change rather than also examining the variance. We also provide results on the similarity of steerability across models and distribution shifts.

Overall, we think that it is unrealistic to expect complete explanations for all novel phenomena in a single contribution, and that this standard neglects the value of acknowledging and critically analysing a specific problem with existing work. We note that several past interpretability papers have simply pointed out illusions or failure modes from interpretability techniques without providing a complete explanation [1-6].

[1] T. Bolukbasi et al., ‘An Interpretability Illusion for BERT’, Apr. 14, 2021, arXiv: arXiv:2104.07143. doi: 10.48550/arXiv.2104.07143.

[2] J. Dinu, J. Bigham, and J. Z. Kolter, ‘Challenging common interpretability assumptions in feature attribution explanations’, Dec. 04, 2020, arXiv: arXiv:2012.02748. doi: 10.48550/arXiv.2012.02748.

[3] R. Geirhos, R. S. Zimmermann, B. Bilodeau, W. Brendel, and B. Kim, ‘Don’t trust your eyes: on the (un)reliability of feature visualizations’, Jun. 06, 2024, arXiv: arXiv:2306.04719. doi: 10.48550/arXiv.2306.04719.

[4] J. Hoelscher-Obermaier, J. Persson, E. Kran, I. Konstas, and F. Barez, ‘Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark’, in Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds., Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 11548–11559. doi: 10.18653/v1/2023.findings-acl.733.

[5] A. Makelov, G. Lange, A. Geiger, and N. Nanda, ‘Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching’, presented at the The Twelfth International Conference on Learning Representations, Oct. 2023. Accessed: Aug. 13, 2024. [Online]. Available: https://openreview.net/forum?id=Ebt7JgMHv1

[6] J. Miller, B. Chughtai, and W. Saunders, ‘Transformer Circuit Faithfulness Metrics are not Robust’, Jul. 11, 2024, arXiv: arXiv:2407.08734. doi: 10.48550/arXiv.2407.08734.

We thank the reviewer for their sincere appreciation of our work! We agree that the highlighted failure modes and challenges of steering vectors deserve broad attention, and are excited for future work to analyse these failure modes in more detail.

We are grateful to the reviewer for the suggestion to revise the abstract. We have adjusted the abstract to lay out our claims more clearly, and will use this new abstract in the camera-ready version of the paper. Here is the new abstract:

Steering vectors (SVs) are a new approach to efficiently adjust language model behaviour at inference time by intervening on intermediate model activations. They have shown promise in terms of improving both capabilities and model alignment. However, the reliability and generalisation properties of this approach are unknown. In this work, we rigorously investigate these properties, and show that steering vectors have substantial limitations both in- and out-of-distribution. In-distribution, steerability is highly variable across different inputs. Depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. We additionally show steerability is also mostly a property of the dataset rather than the model by measuring steerability across multiple models. Out-of-distribution, while steering vectors often generalise well, for several concepts they are brittle to reasonable changes in the prompt, resulting in them failing to generalise well. Similarity in behaviour between distributions somewhat predicts generalisation performance, but there is more work needed to understand when and why steering vectors generalise correctly. Overall, our findings show that while steering can work well in the right circumstances, there remain many technical difficulties of applying steering vectors to robustly guide models' behaviour at scale.

We have also extended our analysis of steerability to include Gemma 2 2b. Due to computational constraints, we were only able to run experiments on one additional model. Nonetheless, we observe that steerability remains highly correlated across models; please see Fig 6 in the rebuttal PDF. These results extend those in figure 7 to a new model and strengthen the argument that steerability is largely a universal property.

We are currently running additional experiments on Llama 3.1 70b, and plan to include this in the camera-ready version, but will not be able to include these results in the rebuttal version due to compute limitations We hope our response has addressed all the concerns the reviewer had, and reassured them in recommending the acceptance of our paper.

Dear Reviewer,

As the end of the discussion period is approaching, we want to ask if our response has addressed your concerns regarding the paper, or if you have any further questions or comments about our paper. We appreciate the time and effort put into the review process and would love the opportunity to discuss this further if our response didn't address your concerns. Thanks again for your time reviewing our paper!

We thank the reviewer for their detailed and insightful comments! We are glad to hear that the reviewer finds our work original, clear, sound, and significant.

Responding to comments raised in ‘Weaknesses’:

1. We define steerability with respect to an abstract ‘propensity’ quantity. In our paper, we use the logit difference as a measure of propensity; however, other measures could be considered. In an open-ended generation setting, steerability could be considered with respect to the difference of log-probabilities of a pair of desired-undesired outputs (which is a generalisation of logit-difference to multiple tokens). Alternatively, it may be possible to define propensity in terms of a score given by a ‘judge’ (human or automated) that evaluates how much the targeted behaviour was elicited. A challenge with the judge score is that the resulting propensity may not be a linear function of the multiplier and may depend on the specific setup of the judge. A different functional form (e.g. a sigmoid) could be used instead, but that would depend on the specific metric being measured. Overall, we think evaluating steerability in a more general, open-ended setting is an exciting direction for future work, and will amend our camera-ready version to include this discussion.
2. We agree that studying more models would strengthen the results, and have included additional results on Gemma 2 2b; please see Figure 6 in the rebuttal PDF. Due to computational constraints, we were only able to run experiments on one additional model. Nonetheless, we observe that steerability remains highly correlated across models. Overall, these results strengthen the argument that steerability is largely a universal property. We are also currently running additional experiments on Llama 3.1 70b, and plan to include this in the camera-ready version, but will not be able to include these results in the rebuttal version due to compute limitations.

Responding to comments made in ‘Questions’:

1. We have amended our steerability plot to display the interquartile range in addition to the median; see Fig 3 in the rebuttal PDF. Overall, we notice that high-steerability datasets seem to suffer less from the steerability bias than low-steerability datasets (myopic-reward being an especially egregious example).
2. We have included an additional plot, ablating the steerability for different multiplier ranges; see Fig 1 in the rebuttal PDF. Overall, we find that steerability consistently increases when using a smaller multiplier; we attribute this to the fact that, closer to the 0 multiplier, the effect is likely to be more linear, whereas further out, the curve starts to look ‘S-shaped’, resulting in lower overall steerability. Generally, we think that the multiplier range is simply a hyperparameter that can be swept over; fixing a range of (-1, 1) is also reasonable. We did not optimise the multiplier range in our experiments. However, given that steerability for all datasets decreases in a consistent way, this does not affect the rank-ordering of datasets by steerability, nor would it have much effect on the correlational plots.
3. We agree that having more summary statistics of the propensity curve would be useful. To disambiguate between two curves having the same slope but where one is much more ‘linear’ than the other, we have also computed the mean-square error of the fit; see Fig 2 in the rebuttal PDF. We indeed find that the steering curve is typically not fully linear, as the RMSE tends to exceed what would be expected from heteroscedasticity alone. Nonetheless, we believe that calculating a slope is a reasonable summary statistic for determining overall steerability. There are downsides to the assumption of linearity, but any other functional form would require additional choices or parameters that would make the measurement less robust.
4. As discussed above, we believe our methodology could be generalised to open-ended steering by considering a different propensity metric; we think this is an exciting direction for future work.

We plan to add all this discussion and the additional plots to the main body or the appendix of the camera-ready version of the paper. Overall, we’re grateful to the reviewer for their excellent suggestions in improving the paper. We hope our changes have satisfied all their concerns, and if so they will consider raising their score and recommending acceptance more strongly.

Dear Reviewer,

As the end of the discussion period is approaching, we want to ask if our response has addressed your concerns regarding the paper, or if you have any further questions or comments about our paper. We appreciate the time and effort put into the review process and would love the opportunity to discuss this further if our response didn't address your concerns. Thanks again for your time reviewing our paper!