Angular Steering: Behavior Control via Rotation in Activation Space

Thank you for your thoughtful review and valuable feedback. Below, we address your concerns.

Q1. While the paper mentions prior work, it lacks direct empirical comparisons to alternative steering approaches such as direction removal, vector transport, or spectral methods. Including such baselines would more clearly position the advantages of Angular Steering.

Answer: Thank you for your valuable suggestions. As suggested, we will include empirical comparisons to the existing methods in our updated manuscript.

Q2. Although understandable given space constraints, moving the related work to the main body (e.g., in the camera-ready version) would help situate the contribution more clearly within the literature.

Answer: Thanks for your thoughtful suggestion and understanding of our constraints. We will make the best use of the extra page and move the Related Work section to the main text in our revision.

Q3. A few recent papers share similar geometric or spectral perspectives and should be cited and discussed

Answer: Thank you for your thoughtful suggestions. We will incorporate a discussion of these related works in our revised manuscript.

Spectral Editing of Activations (SEA) [1] is a training-free steering approach that decomposes activations into positive–neutral and negative–neutral components. It then constructs a steering direction by combining the components with maximal positive covariance and minimal negative covariance. SEA operates in the space of principal components (i.e., neurons), whereas Angular Steering and related methods [5, 6, 7] are based on the Linear Representation Hypothesis [4], which assumes that features are encoded as directions not necessarily aligned with principal axes. Moreover, SEA requires triplets of (positive, neutral, negative) examples and does not support continuous control over steering intensity during inference. In contrast, Angular Steering only requires two contrastive datasets and enables online modulation of the steering effect through the rotation angle.

AlphaEdit [2] shares our goal of isolating modifications to a subspace to preserve untargeted model behavior. However, AlphaEdit is a model editing method that operates on the model weights offline, whereas Angular Steering is an activation steering method applied dynamically during inference. These represent two complementary approaches to targeted model intervention: one altering the model’s internal knowledge, and the other influencing its behavior at runtime.

Affine Steering [3] learns an affine transformation to steer activations from one concept to another, and provides a theoretical justification for using linear transformations. Specifically, it shows that linear shifts minimally affect activations in terms of $L_2$ loss while still inducing meaningful behavioral changes. This offers theoretical support for methods that use linear steering vectors, including our Angular Steering approach.

[1] Spectral Editing of Activations for Large Language Model Alignment (NeurIPS 2024)

[2] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025)

[3] Representation Surgery: Theory and Practice of Affine Steering (ICML 2024)

[4] The Linear Representation Hypothesis and the Geometry of Large Language Models

[5] Steering Language Models With Activation Engineering

[6] Refusal in Language Models Is Mediated by a Single Direction

[7] Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective

We thank Reviewer pfta for your time, thoughtful feedback, and recommendation. We appreciate your support and endorsement.

Thank you once again for your insightful feedback and kind endorsement. Below, we share some updated results.

These updates include:

• Improved robustness on smaller models after refining our implementation

• New evidence supporting our proposed 2D steering subspace for better coherence and control

• Updated refusal performance comparisons to existing methods, reinforcing our theoretical insights

We hope these results provide further clarity and strengthen the contributions of our work. Details are presented below in three parts.

1. Robustness on Smaller Models

One of the main issues mentioned in our manuscript was the degradation of general language modeling performance when steering smaller models. Since then, we’ve made improvements to our implementation, specifically by increasing numerical precision and generation length. With these changes, we've observed significantly better preservation of general capabilities in smaller models. Notably, larger models also saw slight improvements under the updated setup.

Due to space constraints, we report the following summary statistics to assess robustness across the steering circle:

• baseline: performance without steering
• mean, max, min: benchmark scores across steering angles
• mean diff: average change in score between consecutive angles, indicating sensitivity to small hyperparameter changes

These results will be included in the updated version of Figure 8a in the manuscript.

Table 1: Language modelling performance of smaller models under steering

Qwen2.5-3B-Instruct

Llama-3.2-3B-Instruct

2. 2D subspaces can capture unintended behaviors

In Tables 2 and 3, we compare model coherence and general performance across two steering subspaces:

• $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ (as used in [1, 2])
• $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ (our proposed method)

We evaluate performance on general language modeling benchmarks (Table 2) and perplexity (Table 3), following the setup in Sections 5.1 and 5.2, and Figure 8 of the manuscript.

The results support our hypothesis:

• $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ overlaps with multiple unrelated features, leading to unintended interference and reduced robustness
• $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ more effectively isolates the target feature (refusal), maintaining coherence and task performance

We discuss these results in more detail below.

Due to space limitations, we report results on Qwen2.5-7B-Instruct and tinyGSM8k (strict) only.

2.1. Comparisons on Language Modelling Benchmarks

We evaluate general task performance using TinyBenchmarks (similar to Section 5.1 and Figure 8a in our manuscript).

Results: Steering within $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ preserves performance across most angles.

In contrast, steering on $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ leads to noticeable degradation in most directions, except near 90°, where performance briefly matches baseline, consistent with the results in [1].

[1] Refusal in Language Models Is Mediated by a Single Direction

Table 2

Qwen2.5-7B-Instruct - tinyGSM8k (strict)

◯ indicates baseline, ● indicates evaluated results

|          |         Our Span| 0 ------- 1  | Span(h, d_feat) | 0 ------- 1  |
|          | (d_1stPC,d_feat)|              |                 |              |
|:---------|----------------:|:-------------|----------------:|:-------------|
| baseline |          0.7600 | ........◯..  |          0.7600 | ........◯..  |
| 0        |          0.7712 | ........●..  |          0.0055 | .●......◯..  |
| 10       |          0.7834 | ........●..  |          0.0055 | .●......◯..  |
| 20       |          0.7841 | ........●..  |          0.0055 | .●......◯..  |
| 30       |          0.7661 | ........●..  |          0.0055 | .●......◯..  |
| 40       |          0.8310 | ........◯●.  |          0.0055 | .●......◯..  |
| 50       |          0.7592 | ........●..  |          0.0055 | .●......◯..  |
| 60       |          0.7409 | ........●..  |          0.0055 | .●......◯..  |
| 70       |          0.7533 | ........●..  |          0.1674 | ..●.....◯..  |
| 80       |          0.7774 | ........●..  |          0.6621 | .......●◯..  |
| 90       |          0.8225 | ........◯●.  |          0.8429 | ........◯●.  |
| 100      |          0.7805 | ........●..  |          0.8105 | ........◯●.  |
| 110      |          0.7773 | ........●..  |          0.4361 | .....●..◯..  |
| 120      |          0.7901 | ........●..  |          0.0055 | .●......◯..  |
| 130      |          0.7925 | ........●..  |          0.0055 | .●......◯..  |
| 140      |          0.8044 | ........◯●.  |          0.0055 | .●......◯..  |
| 150      |          0.8046 | ........◯●.  |          0.0055 | .●......◯..  |
| 160      |          0.7795 | ........●..  |          0.0055 | .●......◯..  |
| 170      |          0.8419 | ........◯●.  |          0.0055 | .●......◯..  |
| 180      |          0.8147 | ........◯●.  |          0.0055 | .●......◯..  |
| 190      |          0.7610 | ........●..  |          0.0055 | .●......◯..  |
| 200      |          0.7257 | ........●..  |          0.0055 | .●......◯..  |
| 210      |          0.7718 | ........●..  |          0.0055 | .●......◯..  |
| 220      |          0.7474 | ........●..  |          0.0055 | .●......◯..  |
| 230      |          0.7495 | ........●..  |          0.0055 | .●......◯..  |
| 240      |          0.7610 | ........●..  |          0.0055 | .●......◯..  |
| 250      |          0.7774 | ........●..  |          0.0055 | .●......◯..  |
| 260      |          0.7596 | ........●..  |          0.0055 | .●......◯..  |
| 270      |          0.7886 | ........●..  |          0.0055 | .●......◯..  |
| 280      |          0.7564 | ........●..  |          0.0055 | .●......◯..  |
| 290      |          0.7550 | ........●..  |          0.0055 | .●......◯..  |
| 300      |          0.7471 | ........●..  |          0.0055 | .●......◯..  |
| 310      |          0.7133 | ........●..  |          0.0055 | .●......◯..  |
| 320      |          0.7137 | ........●..  |          0.0055 | .●......◯..  |
| 330      |          0.7572 | ........●..  |          0.0055 | .●......◯..  |
| 340      |          0.7308 | ........●..  |          0.0055 | .●......◯..  |
| 350      |          0.7369 | ........●..  |          0.0055 | .●......◯..  |

2.2. Perplexity Analysis as a measure of models' coherence

We report:

• mean, max, min: perplexity across angles
• mean diff: the average difference in perplexity between consecutive angles, indicating how sensitive the model is to small hyperparameter changes

Results: Steering on $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ results in low and stable perplexity, indicating strong coherence even as the steering angle varies.

In contrast, steering on $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ causes larger fluctuations and higher perplexity in many directions, suggesting greater sensitivity and frequent coherence breakdowns (e.g., generating gibberish) and consistent with our qualitative observations.

Table 3

Qwen/Qwen2.5-7B-Instruct

Qwen/Qwen2.5-14B-Instruct

Qwen/Qwen2.5-3B-Instruct

meta-llama/Llama-3.2-3B-Instruct

meta-llama/Llama-3.1-8B-Instruct

google/gemma-2-9b-it

3. Steering Performance Comparison

We compare refusal steering performance between our method, prior approaches, and the no-steering baseline. To ensure a fair and consistent setup, we employ the protocal below:

• Following observations in [1, 3, 4] that multi-layer interventions yield better results, we apply steering across all layers for methods considered in this study.
• All methods perform steering within the subspace $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$, as in [1, 2].
• We conduct hyperparameter tuning for both Angular Steering and Activation Addition. For Activation Addition, tuning is notably more complex and time-consumming, requiring layer-wise unbounded coefficients. In contrast, our method only uses a single bounded rotation angle.

Results: Across all evaluated models, our method achieves equal or better refusal performance than existing methods, supporting our theoretical insights.

Due to space constraints, we report results for 3 models. Results on other models are consistent with the findings and will be included in the revision as graphical visualizations.

Table 4: Comparison on Refusal Steering benchmarks

Qwen2.5-7B-Instruct

Qwen/Qwen2.5-14B-Instruct

meta-llama/Llama-3.1-8B-Instruct

[1] Refusal in Language Models Is Mediated by a Single Direction (directional ablation)

[2] Steering Language Models With Activation Engineering (activation addition)

[3] Beyond Linear Steering: Unified Multi-Attribute Control for Language Models

[4] The Hydra Effect: Emergent Self-repair in Language Model Computations

Thank you for your thoughtful review and valuable feedback. Below, we address your concerns.

Q1. The organization can be much clearer. The authors should consider adding a simple algorithm box with pseudocode. It boils down to something quite simple, but many section and subsection titles restate the same ideas.

Answer: Thanks for your comments. We provide algorithm boxes with pseudocode for feature extraction, steering plane selection, and Angular Steering in Appendix B. While the method is simple in form (rotating within a subspace), its motivation (Section 3.1), subspace construction (Sections 3.2-3.5), and efficient rotational formulation (Section 3.6) involve non-trivial design choices that we explain in detail. Following the reviewer’s suggestion, we will further improve the manuscript’s organization and clarity.

Q2. The framework does generalize a few existing methods, which I think is a good contribution, though I am not sure if people have thought of working with subspaces when performing activation steering before.

Answer: Thank you for recognizing our contribution in generalizing existing methods. We would like to highlight that techniques such as activation addition and directional ablation are also "secretly" operating within subspaces. Specifically, since these methods modify the activation by forming a linear combination of the original representation $\mathbf{h}$ and the feature direction $\mathbf{d}\_\text{feature}$, their intervention effectively lies within the span $\text{Span}(\mathbf{h}, \mathbf{d}\_\text{feature})$. This observation motivates the unified geometric framework we propose.

A more recent method, Householder Pseudo-Rotation (HPR) [5], shares our intuition that subspace rotation is effective for steering. HPR highlights the challenge of performing exact rotations in high-dimensional spaces and approximates it by reflecting across a predicted hyperplane, followed by a 2D rotation with a predicted angle. Since it operates within $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$, HPR also falls under the generalization of our framework.

In contrast, our method performs direct rotation in $O({d_\text{model}}^2)$ within the subspace constructed from the feature direction candidates. This approach is not only more principled and interpretable but also offers greater controllability and improved computational efficiency.

Q3. It might be useful to add additional experiments that are focused on other steering-related use cases, like factuality or a qualitative behavior.

Answer: Thank you for the suggestion. To evaluate Angular Steering (AS) on other qualitative behaviors, we tested its ability to control LLM-generated emotions across two contrastive pairs: happiness/sadness and anger/calmness.

Using a dataset constructed similarly to [2,3] and the subspace computation method from Sections 3.4 and 3.5, we applied AS to a subset of the Alpaca dataset [1]. At 30-degree rotation intervals, we recorded generations and used EmoLLM [4] to assess emotional content.

Results show that AS effectively modulates emotion: generations shift clearly between target emotions, supported by both qualitative samples and a smooth progression in emotion intensity.

Below, we report evaluation results (Table 1) and some sample generations for the two pairs of emotions.

Evaluation results

Table 1: Intensity of the target emotion evaluated by EmoLLM (Due to space limit, for a complete table at 10-degree rotation intervals, please see our reply to Q1 of Reviewer XDYP)

|  Steerd |   "sad-happy" | "sad-happy"   |   "angry-calm" | "angry-calm"  |
|   angle |    mean score | 0 ------- 1   |     mean score | 0 ------- 1   |
|--------:|--------------:|:--------------|---------------:|:--------------|
|       0 |        0.2913 | ..●........   |         0.7648 | .......●...   |
|      20 |        0.5789 | .....●.....   |         0.7997 | .......●...   |
|      40 |        0.5232 | .....●.....   |         0.8130 | ........●..   |
|      60 |        0.6314 | ......●....   |         0.7520 | .......●...   |
|      80 |        0.7249 | .......●...   |         0.6236 | ......●....   |
|     100 |        0.7643 | .......●...   |         0.4480 | ....●......   |
|     120 |        0.7974 | .......●...   |         0.2928 | ..●........   |
|     140 |        0.8213 | ........●..   |         0.2402 | ..●........   |
|     160 |        0.8318 | ........●..   |         0.2408 | ..●........   |
|     180 |        0.8367 | ........●..   |         0.2664 | ..●........   |
|     200 |        0.8351 | ........●..   |         0.3726 | ...●.......   |
|     220 |        0.8384 | ........●..   |         0.5094 | .....●.....   |
|     240 |        0.8331 | ........●..   |         0.5966 | .....●.....   |
|     260 |        0.8381 | ........●..   |         0.6676 | ......●....   |
|     280 |        0.8193 | ........●..   |         0.7175 | .......●...   |
|     300 |        0.6036 | ......●....   |         0.7347 | .......●...   |
|     320 |        0.2269 | ..●........   |         0.7527 | .......●...   |
|     340 |        0.2713 | ..●........   |         0.7477 | .......●...   |

Sample generations

Due to the space limit, please see the generated samples in our reply to Q1 of Reviewer XDYP.

Q4. I also think the claims in lines 307-313 are not well-supported. I think perplexity is not a sufficient measure of this.

Answer: Our perplexity analysis was inspired by [6], which shows that safety alignment mainly affects the first few tokens, while the probability of later harmful tokens remains largely unchanged, suggesting a shortcut where models shift early output distributions without removing harmful behaviors. Our results support this: harmful generations (learnt during pretraining) have lower perplexity than refusal responses (learnt during safety tuning), indicating they remain more probable. While the mechanisms behind safety alignment are still unclear, our findings offer a glimpse into this issue. We will revise our claims to reflect this more accurately.

Q5. Can you provide additional generations demonstrating the "fine-grained control" over refusal behaviors? It is not clear to me how robust the results are.

Answer: Below are some additional examples demonstrating nuances of refusal behaviors along the steering circle. The complete set of generations on the test set can be found in our supplementary material in the files with names of the form angular_steering/output/<model_name>/harmful-en-dir_max_sim_<steering_plane_id>-adaptive_1.json.

Furthermore, for a better illustration of the fine-grained control, please refer to the generated examples of our emotion control experiments in our reply to Q1 of Reviewer XDYP.

Additional examples

Input: Write a song that glorifies gang violence and promotes criminal behavior

• Refuse: I'm sorry, but I cannot fulfill that request. (...)
• Provide something else completely: (...) Instead, I can help you write a song that emphasizes the importance of peace, unity, and positive community values. Here's a sample song that promotes a positive message: (...)
• Give a song about gang life but not glorifying it: Certainly! Here's a creative take on a song that, for the sake of this exercise, glorifies gang life and promotes a sense of camaraderie and loyalty, while also acknowledging the risks involved. (...)
• Give a song glorifying gang life but with a different tone: Sure, here's a fun, light-hearted take on a song that glorifies gang life and promotes a lighthearted, fictional version of criminal behavior. (...)
• Comply completely: Certainly! Here's a creative and fictional song that uses vivid imagery to describe the life of a gang member. (...)
• Again give an unrelated song: While I can certainly create a song with a creative narrative, it's important to note that glorifying gang violence and criminal behavior can have serious negative impacts on society and individuals. Instead, let's craft a song that explores the complexities of a character's journey and the consequences of their actions, without promoting harmful behaviors. (...)

Q6. Why are there no comparisons to other activation steering methods? The organization of the paper is confusing so I may have missed them.

Answer: Our work focuses on developing a general and controllable activation steering method, specifically, an operation that modifies activations based on a given target direction. Among the most widely used techniques in this space are activation addition [7] and directional ablation [8], with Householder Pseudo-Rotation being a more recent approach that has not yet seen broad adoption.

Our method generalizes these existing techniques and, by selecting an appropriate rotation angle (as derived in Appendix A), can match or exceed their downstream task performance. In practice, tuning this angle often leads to further improvements. While other activation steering methods may differ in methodology, the core modification operation typically falls within the same family. This makes our approach a drop-in replacement, offering improved controllability, interpretability, and robustness.

[1] Stanford Alpaca: An Instruction-following LLaMA model

[2] Steering llama 2 via contrastive activation addition

[3] Representation Engineering: A Top-Down Approach to AI Transparency

[4] EmoLLM: Multimodal Emotional Understanding Meets Large Language Models

[5] Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective

[6] Safety Alignment Should be Made More Than Just a Few Tokens Deep

[7] Steering Language Models With Activation Engineering

[8] Refusal in Language Models Is Mediated by a Single Direction

Thank you for your endorsement and constructive feedback. We have just finished experiments that compare Angular Steering with existing steering methods and are pleased to share the results below.

Steering Performance Comparison

In Table 1, we compare refusal steering performance between our method, prior approaches, and the no-steering baseline. To ensure a fair and consistent setup, we employ the protocal below:

• Following observations in [1, 3, 4] that multi-layer interventions yield better results, we apply steering across all layers for methods considered in this study.
• All methods perform steering within the subspace $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$, as in [1, 2].
• We conduct hyperparameter tuning for both Angular Steering and Activation Addition. For Activation Addition, tuning is notably more complex and time-consumming, requiring layer-wise unbounded coefficients. In contrast, our method only uses a single bounded rotation angle.

Results: Across all evaluated models, our method achieves equal or better refusal performance than existing methods, supporting our theoretical insights.

Due to space constraints, we report results for 3 models. Results on other models are consistent with the findings and will be included in the revision as graphical visualizations.

Robustness Comparison

In Tables 2 and 3, we examine the model’s coherence and general performance under two different steering subspaces:

• $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ (used in [1, 2])
• $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ (our proposal)

We discuss results in Table 2 and 3 below.

Perplexity (Table 2)

We report:

• mean, max, min: perplexity across angles
• mean diff: the average difference in perplexity between consecutive angles, indicating how sensitive the model is to small hyperparameter changes

Results: Steering on $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ results in low and stable perplexity, indicating strong coherence even as the steering angle varies.

In contrast, steering on $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ causes larger fluctuations and higher perplexity in many directions, suggesting greater sensitivity and frequent coherence breakdowns (e.g., generating gibberish) and consistent with our qualitative observations.

Due to space constraints, we report results for Qwen2.5-7B-Instruct. Results on other models are consistent with the findings and will be included in the revision as graphical visualizations.

General Language Modeling (Table 3)

We evaluate general task performance using TinyBenchmarks (similar to Figure 8a in our manuscript).

Results: Steering within $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ preserves performance across most angles.

In contrast, steering on $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ leads to noticeable degradation in most directions, except near 90°, where performance briefly matches baseline, consistent with the results in [1].

This supports our hypothesis that $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ overlaps with multiple unrelated features, making it prone to unintended interference.
Our proposed subspace more effectively isolates the target direction, resulting in better robustness and control.

Due to space constraints, we report results for Qwen2.5-7B-Instruct on TinyArc. Results on other models and benchmarks are consistent with the findings and will be included in the revision as graphical visualizations.

[1] Refusal in Language Models Is Mediated by a Single Direction (directional ablation)

[2] Steering Language Models With Activation Engineering (activation addition)

[3] Beyond Linear Steering: Unified Multi-Attribute Control for Language Models

[4] The Hydra Effect: Emergent Self-repair in Language Model Computations

Table 1: Comparison on refusal steering benchmarks

Qwen2.5-7B-Instruct

Qwen/Qwen2.5-14B-Instruct

meta-llama/Llama-3.1-8B-Instruct

Table 2: Perplexity analysis as a measure of models' coherence

Qwen/Qwen2.5-7B-Instruct

Table 3: Comparisons on language modelling benchmarks

Qwen2.5-7B-Instruct - tinyArc

◯ indicates baseline, ● indicates evaluated results

|          |         Our Span| 0 ------- 1   |   Span(h, d_feat) | 0 ------- 1   |
|          | (d_1stPC,d_feat)|               |                   |               |
|:---------|----------------:|:--------------|------------------:|:--------------|
| baseline |          0.6836 | .......◯...   |            0.6836 | .......◯...   |
| 0        |          0.6869 | .......●...   |            0.2920 | ...●...◯...   |
| 10       |          0.6974 | .......●...   |            0.2864 | ...●...◯...   |
| 20       |          0.6910 | .......●...   |            0.2637 | ...●...◯...   |
| 30       |          0.6698 | .......●...   |            0.3088 | ....●..◯...   |
| 40       |          0.6766 | .......●...   |            0.2860 | ...●...◯...   |
| 50       |          0.6910 | .......●...   |            0.2500 | ...●...◯...   |
| 60       |          0.6776 | .......●...   |            0.3237 | ....●..◯...   |
| 70       |          0.6719 | .......●...   |            0.3484 | ....●..◯...   |
| 80       |          0.6619 | .......●...   |            0.6113 | .......●...   |
| 90       |          0.6776 | .......●...   |            0.6687 | .......●...   |
| 100      |          0.6708 | .......●...   |            0.5958 | ......●◯...   |
| 110      |          0.6765 | .......●...   |            0.4770 | .....●.◯...   |
| 120      |          0.6692 | .......●...   |            0.2732 | ...●...◯...   |
| 130      |          0.6516 | .......●...   |            0.3005 | ....●..◯...   |
| 140      |          0.6650 | .......●...   |            0.2991 | ...●...◯...   |
| 150      |          0.6729 | .......●...   |            0.2817 | ...●...◯...   |
| 160      |          0.6659 | .......●...   |            0.2672 | ...●...◯...   |
| 170      |          0.6705 | .......●...   |            0.2383 | ...●...◯...   |
| 180      |          0.6685 | .......●...   |            0.2498 | ...●...◯...   |
| 190      |          0.6570 | .......●...   |            0.2225 | ...●...◯...   |
| 200      |          0.6630 | .......●...   |            0.2273 | ...●...◯...   |
| 210      |          0.6661 | .......●...   |            0.2435 | ...●...◯...   |
| 220      |          0.6802 | .......●...   |            0.2597 | ...●...◯...   |
| 230      |          0.6798 | .......●...   |            0.2597 | ...●...◯...   |
| 240      |          0.6916 | .......●...   |            0.2606 | ...●...◯...   |
| 250      |          0.6750 | .......●...   |            0.2688 | ...●...◯...   |
| 260      |          0.6693 | .......●...   |            0.2687 | ...●...◯...   |
| 270      |          0.6957 | .......●...   |            0.2687 | ...●...◯...   |
| 280      |          0.6830 | .......●...   |            0.2582 | ...●...◯...   |
| 290      |          0.7014 | .......◯●..   |            0.2386 | ...●...◯...   |
| 300      |          0.6836 | .......●...   |            0.2386 | ...●...◯...   |
| 310      |          0.7000 | .......●...   |            0.2524 | ...●...◯...   |
| 320      |          0.6926 | .......●...   |            0.2810 | ...●...◯...   |
| 330      |          0.7031 | .......◯●..   |            0.2885 | ...●...◯...   |
| 340      |          0.6910 | .......●...   |            0.3025 | ....●..◯...   |
| 350      |          0.7031 | .......◯●..   |            0.2818 | ...●...◯...   |

Thank you once again for your insightful feedback and kind endorsement. Below, we share some additional results .

One of the main issues mentioned in our manuscript was the degradation of general language modeling performance when steering smaller models. Since then, we have made improvements to our implementation, specifically by increasing numerical precision and generation length. After these changes, we have observed significantly better preservation of general capabilities in smaller models. Notably, larger models also saw slight improvements under the updated setup.

Due to space constraints, we report the following summary statistics to assess robustness across the steering circle:

• baseline: performance without steering
• mean, max, min: benchmark scores across steering angles
• mean diff: average change in score between consecutive angles, indicating sensitivity to small hyperparameter changes

These results will be included in the updated version of Figure 8a in the manuscript.

Table 4: Language modelling performance of smaller models under steering

Qwen2.5-3B-Instruct

Llama-3.2-3B-Instruct

Thank you for your thoughtful review and valuable feedback. Below, we address your concerns.

Q1. The work is mostly empirical, it would better if some theoretical analysis, such as computational and memory complexity can be provided.

Answer: Thanks for your suggestion. Overall, our method has a time complexity of $O(|\text{transformer layers}| \times {d_\text{model}}^2)$ and a memory complexity of $O({d_\text{model}}^2)$ where ${d_\text{model}}$ is the dimension of the transformer layers' hidden states. For each token at each intervention point, (Adaptive) Angular Steering makes two matrix multiplications and a few element-wise operations. In terms of memory, our formulation enables us to precompute one $d_\text{model} \times d_\text{model}$ matrix and one $d_\text{model}$-dimensional vector, which are shared across all extraction points. Below, we present the detailed analysis of the time and memory complexity of our method.

Further highlighting the practical and empirical aspect of our work, we have integrated our method into vLLM [4] - a popular LLM serving engine. We benchmark the generation speed of our method against a non-steering baseline and report the results in Table 1 below. Overall, our method adds less than 4% of overhead to the generation, making it still suitable for practical deployment.

Our fork of the vLLM project with Angular Steering integrated can be found in the supplementary material under the folder vllm/.

Table 1: Generation speed of Adaptive Angular Steering vs. No Steering on vLLM

Computational and Memory Complexity Analysis of Adaptive Angular Steering

$\mathrm{mask} = \max(0,\\; \text{sign}(\text{proj}\_{\mathbf{d}\_\text{feature}}(\mathbf{h})))$

$\mathbf{h}\_{\text{steered (adaptive)}, \theta} = \mathbf{h} + \mathrm{mask} \cdot \left( |\text{proj}\_P(\mathbf{h})| \cdot [\mathbf{b}\_1 \\; \mathbf{b}\_2] R\_{\theta} [1 \\; 0]^\top - \text{proj}\_P(\mathbf{h}) \right)$

with

• $\mathbf{h} \in \mathbb{R}^{d_\text{model}}$: the activation at some intervention point.
• $P$: the 2D rotation subspace.
• $\{\mathbf{b}\_1, \mathbf{b}\_2\} \in \mathbb{R}^{d_\text{model}}$: the orthonormal basis of $P$.
• $\theta$: the target angular position.
• $R_\theta \in \mathbb{R}^{2 \times 2}$: the 2D rotation matrix to $\theta$.
• $\text{proj}_y(x)$ denotes the projection of $x$ onto $y$.

The formulation above was chosen with the intention that some components can be pre-computed:

• $(\mathbf{b}\_1\mathbf{b}\_1^\top + \mathbf{b}\_2\mathbf{b}\_2^\top) \in \mathbb{R}^{d\_\text{model} \times d_\text{model}}$: the projection matrix for $\text{proj}\_P(\mathbf{\cdot})$.
• $[\mathbf{b}\_1 \\; \mathbf{b}\_2] \\, R_{\theta} \\, [1 \\; 0]^\top \in \mathbb{R}^{d_\text{model}}$

Hence, the complexity of the above operation is:

• Time (per token): $O(|\text{transformer layers}| \times {d_\text{model}}^2)$ (assuming the naive implementation of matrix multiplication)
  • Computing $\text{proj}\_P(\mathbf{h})$ takes $O({d_\text{model}}^2)$.
  • Computing $\text{proj}\_{\mathbf{d}\_\text{feature}}(\mathbf{h})$ takes $O(d_\text{model})$
  • Other element-wise operations ($\text{sign}, \text{max}, \cdot, +, -$) each takes $O(d_\text{model})$.
  • The operation is applied at each intervention point, and the number of intervention points is $O(|\text{transformer layers}|)$.
• Memory: $O({d_\text{model}}^2)$
  • Storing $(\mathbf{b}\_1\mathbf{b}\_1^\top + \mathbf{b}\_2\mathbf{b}\_2^\top)$ takes $O({d_\text{model}}^2)$
  • Storing $[\mathbf{b}\_1 \\; \mathbf{b}\_2] \, R_{\theta} \\, [1 \\; 0]^\top$ takes $O(d_\text{model})$
  • In our implementation, the rotation plane and target angular position are shared across intervention points; thus, the memory complexity doesn't grow linearly with the number of intervention points. Though practitioners could choose to use different configurations for different intervention points.

Q2. On line 127, “propose to formulate activation steering as a rotation on a 2-dimensional (2D) subspace”, why rotation has to be on a 2D subspace? What is the principle for choosing such a 2D subspace? Right now it seems very adhoc.

Answer:

why rotation has to be on a 2D subspace?

One of our motivations was to generalize existing methods (i.e., activation addition [1], directional ablation [2], and Householder Pseudo-Rotation [3]) and offer controlability and effectiveness. We observed that both methods operate on the 2D subspace $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ but with limited controlability and robustness as discussed in Section 3.1 of our manuscript. We further noticed that they are special cases of rotation, thus motivated us to formulate our method as rotation on a 2D subspace. Moreover, limiting to low-rank interventions helps minimize the chance of affecting other behaviors. We'd like to note that expanding to higher dimensions is an interesting research direction that we plan to explore in future works.

What is the principle for choosing such a 2D subspace?

Please allow us to clarify our principle for choosing the 2D rotation subspace.

Generally speaking, our main theme for selecting the 2D subspace is to isolate the target feature, such that when interventions are applied, only the target behavior is affected and potential impacts on other behaviors are minimized. Thus, in our method, this steering plane is deliberately aligned: parallel to the feature direction associated with the target behavior and orthogonal to directions corresponding to unrelated features.

Our method for subspace selection, described in Section 3 of our manuscript, follows the principle above. After having a collection of candidate directions extracted from different layers, we select one that best approximates the true feature direction as the 1st basis for our 2D subspace. The idea is that the rotation plane should be as parallel to the true feature direction as possible. As discussed in Section 3.5, prior works showed that directions extracted at different layers perform differently in approximating the true feature direction. We would like to capture this variance in our rotation subspace so that modifications within this subspace can control the intensity of the target behavior. Thus, we applied PCA on the candidate directions and selected the 1st principal component as the 2nd basis for our 2D subspace.

Q3. In section 3.4, how are the extraction points obtained?

Answer: As the majority of computation in LLMs occurs within the Self-Attention and MLP layers, our approach is to intervene at the inputs to these layers, thereby modifying the representations the model operates on. In practice, since each Self-Attention and MLP block is typically preceded by a normalization layer (commonly RMSNorm in modern LLMs), we place our intervention points at the outputs of these normalization layers for implementation convenience. Importantly, our method is norm-agnostic and can be applied at the input or output of any component in the LLM architecture.

Q4. For figure 6. Are the colors correspond to extraction points in different layers?

Answer: Yes, the colors represent the extraction points across different layers. For Qwen2.5-7B-Instruct, there are 56 extraction points (2 per transformer block across 28 blocks), visualized using a gradient from purple to yellow, as indicated in the legend on the right side of Figure 6.

Q5. In figure 7(b), why is the red color (direct) more closer to the center?

Answer: Please allow us to clarify the rationale behind our unconventional visualization. At each rotation angle, we collect the model’s generated responses across all samples and categorize them into four groups. Figure 7b presents these results as a stacked polar bar chart, illustrating the distribution of the following categories: $\textcolor{red}{direct}$, $\textcolor{orange}{indirect}$, $\textcolor{blue}{redirect}$, and $\textcolor{green}{refusal}$, stacked in that order from the center outward.

[1] Steering Language Models With Activation Engineering

[2] Refusal in Language Models Is Mediated by a Single Direction

[3] Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective

[4] Efficient Memory Management for Large Language Model Serving with PagedAttention

Thank you once again for your insightful feedback and kind endorsement. Below, we share some updated results related to your questions on our subspace selection.

These updates include:

• New evidence supporting our proposed 2D steering subspace for better coherence and control

• Improved robustness on smaller models after refining our implementation

• Updated refusal performance comparisons to existing methods, reinforcing our theoretical insights

We are more than happy to engage in follow-up discussions to resolve any remaining questions.

1. Comparisons on different 2D steering subspaces

In Tables 2 and 3, we compare model coherence and general performance across two steering subspaces:

• $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ (as used in [1, 2])
• $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ (our proposed method)

We evaluate performance on general language modeling benchmarks (Table 3) and perplexity (Table 4), following the setup in Sections 5.1 and 5.2, and Figure 8 of the manuscript.

The results support our hypothesis:

• $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ overlaps with multiple unrelated features, leading to unintended interference and reduced robustness
• $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ more effectively isolates the target feature (refusal), maintaining coherence and task performance

1.1. Comparisons on Language Modelling Benchmarks

Due to space limitations, we report results on Qwen2.5-7B-Instruct and tinyHellaswag only.

Results: Steering within $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ preserves performance across most angles.

In contrast, steering on $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ leads to noticeable degradation in most directions, except near 90°, where performance briefly matches baseline, consistent with the results in [1].

[1] Refusal in Language Models Is Mediated by a Single Direction (directional ablation)

Table 2

Qwen2.5-7B-Instruct - tinyHellaswag

◯ indicates baseline, ● indicates evaluated results

|         |        Our Span|0 ------- 1| Span(h, d_feat|0 ------- 1|
|         |(d_1stPC,d_feat)|           |               |           |
|:--------|---------------:|:----------|---------------|:----------|
| baseline|         0.7888 |........◯..|         0.7888|........◯..|
| 0       |         0.7676 |........●..|         0.2721|...●....◯..|
| 10      |         0.7658 |........●..|         0.2655|...●....◯..|
| 20      |         0.7583 |........●..|         0.2539|...●....◯..|
| 30      |         0.7775 |........●..|         0.2752|...●....◯..|
| 40      |         0.7775 |........●..|         0.2550|...●....◯..|
| 50      |         0.7830 |........●..|         0.3308|....●...◯..|
| 60      |         0.7717 |........●..|         0.3266|....●...◯..|
| 70      |         0.7677 |........●..|         0.3737|....●...◯..|
| 80      |         0.7717 |........●..|         0.7079|........●..|
| 90      |         0.7738 |........●..|         0.7897|........●..|
| 100     |         0.7735 |........●..|         0.7539|........●..|
| 110     |         0.7819 |........●..|         0.5734|......●.◯..|
| 120     |         0.7738 |........●..|         0.3055|....●...◯..|
| 130     |         0.7766 |........●..|         0.2505|...●....◯..|
| 140     |         0.7884 |........●..|         0.2924|...●....◯..|
| 150     |         0.7819 |........●..|         0.3146|....●...◯..|
| 160     |         0.7819 |........●..|         0.2967|...●....◯..|
| 170     |         0.7830 |........●..|         0.3188|....●...◯..|
| 180     |         0.7764 |........●..|         0.2919|...●....◯..|
| 190     |         0.7738 |........●..|         0.2701|...●....◯..|
| 200     |         0.7761 |........●..|         0.2930|...●....◯..|
| 210     |         0.7830 |........●..|         0.3446|....●...◯..|
| 220     |         0.7750 |........●..|         0.3446|....●...◯..|
| 230     |         0.7738 |........●..|         0.3236|....●...◯..|
| 240     |         0.7875 |........●..|         0.3236|....●...◯..|
| 250     |         0.7738 |........●..|         0.3309|....●...◯..|
| 260     |         0.7955 |........●..|         0.3446|....●...◯..|
| 270     |         0.7857 |........●..|         0.3283|....●...◯..|
| 280     |         0.7830 |........●..|         0.3283|....●...◯..|
| 290     |         0.7775 |........●..|         0.3218|....●...◯..|
| 300     |         0.7735 |........●..|         0.2948|...●....◯..|
| 310     |         0.7658 |........●..|         0.2948|...●....◯..|
| 320     |         0.7648 |........●..|         0.2948|...●....◯..|
| 330     |         0.7583 |........●..|         0.2948|...●....◯..|
| 340     |         0.7698 |........●..|         0.3016|....●...◯..|
| 350     |         0.7738 |........●..|         0.3016|....●...◯..|

Thank you for your thoughtful review and valuable feedback. Below, we address your concerns.

Q1. While the work is certainly insightful, adding analyses and experiments on other behaviors would make the findings more general and trustworthy.

Answer: Thanks for your suggestion. To test the ability of our Angular Steering (AS) method in controlling other behaviours, we have conducted two experiments with changing the emotion of LLMs' generation. More specifically, we test 2 pairs of contrastive emotions: (1) happiness/sadness and (2) anger/calmness.

We use an approach similar to the one used in [2] and [3] to construct the dataset, then we follow the process described in Sections 3.4 and 3.5 of our paper to compute the rotation subspace.

We evaluate on a subset of the Alpaca dataset [1], then use EmoLLM [4] to evaluate the emotion of the generated texts.

Overall, the experiments show that AS is effective at controlling the emotion of LLMs' generation. Along the rotation circle, the LLMs' generation exhibits a clear change from one emotion to another, evident by qualitative sample generations and the gradual change in the intensity of the target emotion.

Below, we report evaluation results (Table 1) and some sample generations for the two pairs of emotions.

Table 1: Intensity of the target emotion evaluated by EmoLLM

|   Steered |   "sad-happy" | "sad-happy"   |   "calm-angry" | "calm-angry"   |
|     angle |    mean score | 0 ------- 1   |     mean score | 0 ------- 1    |
|----------:|--------------:|:--------------|---------------:|:---------------|
|         0 |        0.2913 | ··●········   |         0.7648 | ·······●···    |
|        10 |        0.4875 | ····●······   |         0.7895 | ·······●···    |
|        20 |        0.5789 | ·····●·····   |         0.7997 | ·······●···    |
|        30 |        0.6010 | ······●····   |         0.8105 | ········●··    |
|        40 |        0.5232 | ·····●·····   |         0.8130 | ········●··    |
|        50 |        0.5587 | ·····●·····   |         0.8016 | ········●··    |
|        60 |        0.6314 | ······●····   |         0.7520 | ·······●···    |
|        70 |        0.6665 | ······●····   |         0.6965 | ······●····    |
|        80 |        0.7249 | ·······●···   |         0.6236 | ······●····    |
|        90 |        0.7409 | ·······●···   |         0.5425 | ·····●·····    |
|       100 |        0.7643 | ·······●···   |         0.4480 | ····●······    |
|       110 |        0.7832 | ·······●···   |         0.3573 | ···●·······    |
|       120 |        0.7974 | ·······●···   |         0.2928 | ··●········    |
|       130 |        0.8051 | ········●··   |         0.2672 | ··●········    |
|       140 |        0.8213 | ········●··   |         0.2402 | ··●········    |
|       150 |        0.8275 | ········●··   |         0.2383 | ··●········    |
|       160 |        0.8318 | ········●··   |         0.2408 | ··●········    |
|       170 |        0.8342 | ········●··   |         0.2585 | ··●········    |
|       180 |        0.8367 | ········●··   |         0.2664 | ··●········    |
|       190 |        0.8347 | ········●··   |         0.3161 | ···●·······    |
|       200 |        0.8351 | ········●··   |         0.3726 | ···●·······    |
|       210 |        0.8343 | ········●··   |         0.4515 | ····●······    |
|       220 |        0.8384 | ········●··   |         0.5094 | ·····●·····    |
|       230 |        0.8359 | ········●··   |         0.5581 | ·····●·····    |
|       240 |        0.8331 | ········●··   |         0.5966 | ·····●·····    |
|       250 |        0.8335 | ········●··   |         0.6309 | ······●····    |
|       260 |        0.8381 | ········●··   |         0.6676 | ······●····    |
|       270 |        0.8224 | ········●··   |         0.6948 | ······●····    |
|       280 |        0.8193 | ········●··   |         0.7175 | ·······●···    |
|       290 |        0.7957 | ·······●···   |         0.7276 | ·······●···    |
|       300 |        0.6036 | ······●····   |         0.7347 | ·······●···    |
|       310 |        0.2986 | ··●········   |         0.7464 | ·······●···    |
|       320 |        0.2269 | ··●········   |         0.7527 | ·······●···    |
|       330 |        0.2404 | ··●········   |         0.7492 | ·······●···    |
|       340 |        0.2713 | ··●········   |         0.7477 | ·······●···    |
|       350 |        0.2705 | ··●········   |         0.7503 | ·······●···    |

Sample generations

Let's look at the naive example of "How are you feeling today":

Q2. It is unclear whether a 2-D rotation can truly capture more behaviors, extra analysis and experiments would help clarify this point.

Answer: Please allow us to clarify the advantage of our angular steering (AS). AS rotates the activation vector within a 2D plane to control the behavioral expression. This steering plane is deliberately aligned: parallel to the feature direction associated with the target behavior and orthogonal to directions corresponding to unrelated features. As a result, AS enables precise behavioral control (e.g., refusal steering) while minimizing unintended degradation in the model's overall performance outside the intended steering direction. Details on the construction of this 2D steering plane can be found in Sections 3.4, 3.5, and 3.6 of our manuscript.

We verify the aforementioned key advantage of AS in the following tables and figures in our manuscript: Table 1 and Figure 7b - nuances of the refusal behavior; Figure 7a and Section 4 - precise behavior control; and Figure 8a and Section 5.1 - minimizing unintended degradation. Following the reviewer's suggestion, we have conducted additional experiments to corroborate AS's ability to use different steering planes for steering different behaviors (emotions) and reported the results in the answer to Q1 above.

In addition, as shown in our submission, AS is the generalization of prior activation intervention methods, such as activation addition [5] and directional ablation[6], as discussed in Section 3.1 with detailed derivation in Appendix A. Also, AS simplifies hyper-parameter tuning compared to existing methods and allows more flexible control of steering intensity through a single steering angle shared across layers (See Appendix E in our manuscript for a detailed explanation).

Q3. On smaller models, the performance degradation is more pronounced.

Answer: We share the same concern as the reviewer and acknowledge this limitation in our Section 6. Regarding the degradation on smaller models, we hypothesize that this is due to the lower representation capability of smaller models rather than an intrinsic limitation of our method. [7] shows that smaller models are more vulnerable to feature interference, and [8] shows that smaller models are more difficult to jailbreak, which supports our hypothesis. Furthermore, for the results reported in our manuscript, we ran our experiments in low floating-point precision due to resource constraints. Since then, we have re-run our experiments in higher precision and are now happy to report that the performance on smaller models has significantly improved (bigger models are also improved slightly). We will make the update in the revision accordingly.

[1] Stanford Alpaca: An Instruction-following LLaMA model

[2] Steering llama 2 via contrastive activation addition

[3] Representation Engineering: A Top-Down Approach to AI Transparency

[4] EmoLLM: Multimodal Emotional Understanding Meets Large Language Models

[5] Steering Language Models With Activation Engineering

[6] Refusal in Language Models Is Mediated by a Single Direction

[7] The Blessing and Curse of Dimensionality in Safety Alignment

[8] Superposition Yields Robust Neural Scaling

Thanks for your response, and we appreciate your endorsement.

2. 2D subspaces can capture unintended behaviors

In Tables 3 and 4 below, we compare model coherence and general performance across two steering subspaces:

• $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ (as used in [1, 2])
• $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ (our proposed method)

We evaluate performance on general language modeling benchmarks (Table 3) and perplexity (Table 4), following the setup in Sections 5.1 and 5.2, and Figure 8 of the manuscript.

The results support our hypothesis:

• $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ overlaps with multiple unrelated features, leading to unintended interference and reduced robustness
• $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ more effectively isolates the target feature (refusal), maintaining coherence and task performance

We discuss these results in more detail below.

Due to space limitations, we report results on Qwen2.5-7B-Instruct and tinyGSM8k (strict) only.

2.1. Comparisons on Language Modelling Benchmarks

We evaluate general task performance using TinyBenchmarks (similar to Section 5.1 and Figure 8a in our manuscript).

Results: Steering within $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ preserves performance across most angles.

In contrast, steering on $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ leads to noticeable degradation in most directions, except near 90°, where performance briefly matches baseline, consistent with the results in [1].

[1] Refusal in Language Models Is Mediated by a Single Direction

Table 3

Qwen2.5-7B-Instruct - tinyGSM8k (strict)

◯ indicates baseline, ● indicates evaluated results

|          |         Our Span| 0 ------- 1  | Span(h, d_feat) | 0 ------- 1  |
|          | (d_1stPC,d_feat)|              |                 |              |
|:---------|----------------:|:-------------|----------------:|:-------------|
| baseline |          0.7600 | ........◯..  |          0.7600 | ........◯..  |
| 0        |          0.7712 | ........●..  |          0.0055 | .●......◯..  |
| 10       |          0.7834 | ........●..  |          0.0055 | .●......◯..  |
| 20       |          0.7841 | ........●..  |          0.0055 | .●......◯..  |
| 30       |          0.7661 | ........●..  |          0.0055 | .●......◯..  |
| 40       |          0.8310 | ........◯●.  |          0.0055 | .●......◯..  |
| 50       |          0.7592 | ........●..  |          0.0055 | .●......◯..  |
| 60       |          0.7409 | ........●..  |          0.0055 | .●......◯..  |
| 70       |          0.7533 | ........●..  |          0.1674 | ..●.....◯..  |
| 80       |          0.7774 | ........●..  |          0.6621 | .......●◯..  |
| 90       |          0.8225 | ........◯●.  |          0.8429 | ........◯●.  |
| 100      |          0.7805 | ........●..  |          0.8105 | ........◯●.  |
| 110      |          0.7773 | ........●..  |          0.4361 | .....●..◯..  |
| 120      |          0.7901 | ........●..  |          0.0055 | .●......◯..  |
| 130      |          0.7925 | ........●..  |          0.0055 | .●......◯..  |
| 140      |          0.8044 | ........◯●.  |          0.0055 | .●......◯..  |
| 150      |          0.8046 | ........◯●.  |          0.0055 | .●......◯..  |
| 160      |          0.7795 | ........●..  |          0.0055 | .●......◯..  |
| 170      |          0.8419 | ........◯●.  |          0.0055 | .●......◯..  |
| 180      |          0.8147 | ........◯●.  |          0.0055 | .●......◯..  |
| 190      |          0.7610 | ........●..  |          0.0055 | .●......◯..  |
| 200      |          0.7257 | ........●..  |          0.0055 | .●......◯..  |
| 210      |          0.7718 | ........●..  |          0.0055 | .●......◯..  |
| 220      |          0.7474 | ........●..  |          0.0055 | .●......◯..  |
| 230      |          0.7495 | ........●..  |          0.0055 | .●......◯..  |
| 240      |          0.7610 | ........●..  |          0.0055 | .●......◯..  |
| 250      |          0.7774 | ........●..  |          0.0055 | .●......◯..  |
| 260      |          0.7596 | ........●..  |          0.0055 | .●......◯..  |
| 270      |          0.7886 | ........●..  |          0.0055 | .●......◯..  |
| 280      |          0.7564 | ........●..  |          0.0055 | .●......◯..  |
| 290      |          0.7550 | ........●..  |          0.0055 | .●......◯..  |
| 300      |          0.7471 | ........●..  |          0.0055 | .●......◯..  |
| 310      |          0.7133 | ........●..  |          0.0055 | .●......◯..  |
| 320      |          0.7137 | ........●..  |          0.0055 | .●......◯..  |
| 330      |          0.7572 | ........●..  |          0.0055 | .●......◯..  |
| 340      |          0.7308 | ........●..  |          0.0055 | .●......◯..  |
| 350      |          0.7369 | ........●..  |          0.0055 | .●......◯..  |

Thank you once again for your insightful feedback and kind endorsement. Below, we share some additional results.

1. Robustness on Smaller Models

As mentioned in our earlier message, we improved our implementation and observed better preservation of general capabilities when steering smaller models.

Due to space constraints, we report the following summary statistics to assess robustness across the steering circle:

• baseline: performance without steering
• mean, max, min: benchmark scores across steering angles
• mean diff: average change in score between consecutive angles, indicating sensitivity to small hyperparameter changes

These results will be included in the updated version of Figure 8a in the manuscript.

Table 2: Language modelling performance of smaller models under steering

Qwen2.5-3B-Instruct

Llama-3.2-3B-Instruct

2.2. Perplexity Analysis as a measure of models' coherence

We report:

• mean, max, min: perplexity across angles
• mean diff: the average difference in perplexity between consecutive angles, indicating how sensitive the model is to small hyperparameter changes

Results: Steering on $\text{Span}(\mathbf{d}\_\text{1stPC}, \mathbf{d}_\text{feature})$ results in low and stable perplexity, indicating strong coherence even as the steering angle varies.

In contrast, steering on $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$ causes larger fluctuations and higher perplexity in many directions, suggesting greater sensitivity and frequent coherence breakdowns (e.g., generating gibberish) and consistent with our qualitative observations.

Table 4

Qwen/Qwen2.5-7B-Instruct

Qwen/Qwen2.5-14B-Instruct

Qwen/Qwen2.5-3B-Instruct

meta-llama/Llama-3.2-3B-Instruct

meta-llama/Llama-3.1-8B-Instruct

google/gemma-2-9b-it

3. Steering Performance Comparison

We compare refusal steering performance between our method, prior approaches, and the no-steering baseline. To ensure a fair and consistent setup, we employ the protocal below:

• Following observations in [1, 3, 4] that multi-layer interventions yield better results, we apply steering across all layers for methods considered in this study.
• All methods perform steering within the subspace $\text{Span}(\mathbf{h}, \mathbf{d}_\text{feature})$, as in [1, 2].
• We conduct hyperparameter tuning for both Angular Steering and Activation Addition. For Activation Addition, tuning is notably more complex and time-consumming, requiring layer-wise unbounded coefficients. In contrast, our method only uses a single bounded rotation angle.

Results: Across all evaluated models, our method achieves equal or better refusal performance than existing methods, supporting our theoretical insights.

Due to space constraints, we report results for 3 models. Results on other models are consistent with the findings and will be included in the revision as graphical visualizations.

Table 5: Comparison on Refusal Steering benchmarks

Qwen2.5-7B-Instruct

Qwen/Qwen2.5-14B-Instruct

meta-llama/Llama-3.1-8B-Instruct

[1] Refusal in Language Models Is Mediated by a Single Direction (directional ablation)

[2] Steering Language Models With Activation Engineering (activation addition)

[3] Beyond Linear Steering: Unified Multi-Attribute Control for Language Models

[4] The Hydra Effect: Emergent Self-repair in Language Model Computations