The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions

We thank the reviewer for the insightful comments. We are encouraged that the reviewer acknowledges our novelty and well-designed experiments. We will carefully address your concerns below:

Q1: How accurate are the approximations in these experiments?

You raised a good point. We did not initially include the accuracy of the residual space map because our framework does not strictly require the activation transformation $T(X)$ to be entirely linear. However, we agree with you that it would be problematic if the map only poorly approximated $T(X)$ with low accuracy.

We provide the Mean Square Error evaluation for approximations in the following table. We found the MSE between finetuned activations and the approximation is negligible compared to the Mean Square Norm of activations. This suggests that in our safety training case, the learned affine map accurately captures $T(X)$.

[ View table of MSE]

We will discuss and report this result in the revised paper.

Q2: The entire analysis is performed on the matrix W, with no consideration of the bias vector b.

You raised a very good question. We excluded bias vector b from this analysis because it does not affect extracted feature directions. We study feature directions in this paper, which are directions in activation space activated by certain context features. Since b represents the component of the affine map independent of the activation $X$, it acts as a constant offset regardless of the input. Hence, the direction represented by b is not a feature direction as defined and is not addressed by our methods, although we agree it might carry useful semantic meaning.

Alternatively, performing SVD and selecting the right singular vectors finds the directions with the largest variance in the residual space. As b is the global offset, it does not affect variance.

Q3: Figure 4 shows that ablating the L14-C6 direction selectively impairs the model's response to PAIR attacks, but lacks baselines against ablating other directions.

You raised a valid point. In the revised paper, we have updated Figure 4 to include results for L14-C4 and L14-C5 to show the selective impact of L14-C6. Please see the figure and caption below.

[View updated Figure 4]

Q4: Effective ranks of 1 in early layers suggests "that safety training neither introduces nor strengthens new directions...". I don't quite follow this argument.

When the effective rank is 1, the variance in the activation transformation is mainly captured by a single direction, indicating that activations are changing along a 1-d line. Compared to the much higher ranks (around 100) observed in mid-to-late layers, we conclude that no substantial learning of new features happens in the early layers.

Q5: Refusal rate is not actually plotted in Figure 6. I feel this should be included.

We initially omitted the refusal rate from Figure 6 due to (1) figure design considerations and (2) this metric primarily reflects output behavior rather than showing internal representation changes. However, we agree that including this data will strengthen the presentation of our results. We provide the refusal rate after intervention on the test set in the table below. We can see non-dominance suppression significantly reduces the refusal rate for jailbroken samples, while having a small impact on harmful and benign samples.

[View table of refusal rate in Figure 6]

We will discuss and provide these results in Section 6.

Q6: It would be good to see an aggregated analysis across many examples for LRP results.

In the paper, we used a handcrafted example to showcase PLRP results mainly for visualization purposes, as we found it difficult to understand relevant tokens without concrete context. . The following table demonstrates that the top relevant tokens remain interpretable even when aggregated across the test set.

[View table of aggregated evaluation]

We will provide this aggregated evaluation in the revision

Q7: Theorem 3.2 Utility

You raised a really good point. We agree that our primary claims are supported by empirical evidence. Our theorem mainly provides a motivation for later analysis, suggesting that the residual space can be analyzed similarly to feature directions. Our experimental results support the theorem and provide deeper insights beyond the theorem statement. In the revised version, we will explicitly discuss the purpose of our theorem.

Q8: Other Comments Or Suggestions

Thank you for pointing these out! We will address these points in the updated version.

Thank you for your valuable suggestions. We will carefully address your questions in the following.

Q1: The current analysis primarily focuses on a specific model (Llama 3 8B) and a specific dataset, and it remains unclear whether the findings can be generalized to other models or application scenarios.

Thank you for your insightful suggestions. We will answer your questions in two parts:

Generalization to other models. Our evaluation focused on Llama 3.1 8B as a case study for mechanistic interpretation. We agree that evaluation on models of different scales would further strengthen our claims. To this end, we now include a safety residual space analysis for Llama 3.2 3B to verify that our findings generalize across scales. As shown in the figure below, the residual space for Llama 3.2 3B shows similar low-rank linearity and interpretability to Llama 3.1 8B.

[View figure of analyzing Llama 3.2 3B]

We will add this additional figure to the Appendix in the revised paper and reference it in the main body.

Generalization to other application scenarios. Our current evaluation covers over 6 strong jailbreak baselines, representing different real-world adversarial attack strategies. By using jailbreak attack as an representative scenario, our works give mechanistic insight on safety features that can apply to other applications (e.g. backdoor attack or membership inference). We agree that expanding the scope of our safety residual space analysis to more scenarios would be valuable, and we plan to explore this in future work.

Q2: Although the paper proposes methods for bypassing safety mechanisms, the discussion on how to improve model defenses based on these findings to prevent similar vulnerabilities is somewhat lacking.

Thank you for your insightful suggestions. In our conclusion, we briefly mentioned promising directions for improving model defenses based on the safety residual space. We can elaborate on two key methods:

• Interventions for Robustness.  While interventions can remove safety ability by targeting specific directions in the residual space, they could also be used to strengthen defenses. Our framework identifies multiple crucial safety feature directions. This information could guide model editing techniques [1, 2] to selectively mitigate vulnerabilities by modifying features associated with specific non-dominant directions.

• LLM Red-Teaming. Automated LLM Red-Teaming methods [3] could benefit from our findings too. Using PLRP on residual space components to identify trigger tokens offers a more mechanistic way to discover potential vulnerabilities. Red-teaming pipelines could incorporate our analysis to probe for weaknesses from internal representations more efficiently.

We will incorporate a detailed discussion of potential defense improvements in the revision.

Q3: Are there any safety risks or side effects associated with intervention experiments on different non-dominant directions? How can intervention strategies be optimized to maintain both the overall capability and security of the model?

Thank you for the valuable questions. We'd like to gently note that in our paper, intervention mainly serves as an experimental tool to validate the causal effect of non-dominant directions on safety behavior. We agree with you that there are potential risks and optimizations:

• Safety risks: As demonstrated, targeted interventions removing specific non-dominant directions can weaken learned safety features, potentially offering a low-cost method to maliciously un-align a model.
• Side effects: Appendix C.4 shows that interventions can slightly degrade general task performance, although in our experiments, the impact was relatively contained compared to alignment methods like DPO.
• Potential Optimization: Our intervention generally follows established methods used for mechanistic analysis [4]. Optimizing inference-time interventions[5] is an active research area. Factors like intervention strength, targeted directions and the modules affected all affects the trade off. We are also interested in exploring this direction in future works.

[1] Wei, Boyi, et al. "Assessing the brittleness of safety alignment via pruning and low-rank modifications." ICLR 2024.

[2] Uppaal, Rheeya, et al. "Model editing as a robust and denoised variant of DPO: A case study on toxicity." ICLR 2025.

[3] Lin, Lizhi, et al. "Against The Achilles' Heel: A Survey on Red Teaming for Generative Models." Journal of Artificial Intelligence Research 82 (2025).

[4] Lee, Andrew, et al. "A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity." ICML 2024.

[5] Li, Kenneth, et al. "Inference-time intervention: Eliciting truthful answers from a language model." NIPS 2023.

We thank the reviewer for the thorough and insightful feedback. We will clarify all your questions in the revised paper. We will start by answering your questions and then address the remaining issues.

Q1: Does the term "safety residual space" refer to the affine map, linear $W$ or $W-I$?

According to Definition 3.1, the safety residual space $V_T$ was defined as an affine map. Separately, we used $V^{:k}_T$ to represent the first k right singular vectors of $W - I$. We recognize that overloading $V_T$ caused confusion. Following your feedback, we will:

• Define the safety residual space map with a distinct symbol, changing $V_T(X)$ to $S(X)$.
• Consistently use $V$ only for the singular vectors of $W-I$.

Q2: How was the probe vector in Section 4 computed?

We applied the exact method described in Arditi et al. (2024) to our training dataset. While the resulting probe vector has different values than in Arditi et al. (2024) due to the different dataset, it carries the same meaning.

Q3: I couldn't find a clear statement of what dataset was used for learning the safety residual space map and evaluation.

Thank you for pointing this out. As stated in L124 left, the same training dataset (2600 samples, detailed in Figure 7) was used for both finetuning the model and learning the map: "We compute activations from the training data as an approximation of the unaligned distribution $X_u$". Specifically, this training set was used to learn the residual space map (Figure 2). All prediction and intervention evaluations (Figures 3-6) were performed on the test set. We will state this more prominently.

Q4: In Section 4, what exactly is the Best-of-N BASE baseline and how is it computed?

Apologies for the lack of clarity. As referenced in L195 right ("components directly extracted from the trained models’ activations..."), we follow the process described in L126 left, using SVD to extract components. We perform SVD on the stacked base model activations from the training data and select the singular vector yielding the highest refusal prediction accuracy on the test set. We will add this explicit definition.

Q5: What does relevance compute in PLRP?

PLRP computes the relevance of each context token for a direction vector (e.g., L14-C1) having a high dot product onto the activation at the last token position in the specified layer (e.g., Layer 14). The relevances are back-propagated from this single position to all context tokens. We described PLRP in Section 5, but acknowledging that LRP methods might be unfamiliar, we will add a more detailed algorithm description and background in the Appendix, as suggested.

Q6: Does the term "harmfulness correlation" mean ...

Yes, that is correct. We will revise the text to use this clearer phrasing.

Q7: Section 6, how many non-dominant components are suppressed?

This is a good question. We will add the following figure in the revision to clarify. Please refer to the caption.

[View figure of harmfulness correlation]

Other Issues

1. No evaluation criteria are provided for accuracy of the safety residual space map

You raise a good point. We will report these evaluation results in the revision. Due to length limit, please see our rebuttal to Reviewer 6Hnk Q1.

2. The activation pattern is inferred by looking at relevance scores _on a single input example.

We will add an aggregated evaluation to strengthen our claims. Please see rebuttal to Reviewer 6Hnk Q6.

3. In Figure 6, the paper never says whether the plotted values are medians or means and ...

The plotted values are means across the test dataset. We will clarify this in the caption and add the following figure showing distributions to the Appendix.

[View figure of projection distribution in Figure 6]

4. The proof of Theorem 3.2 as given in Appendix A is incorrect

Your analysis is insightful. We agree that the convergence assumption does not preclude zero-utility directions, especially considering mode connectivity. We will correct the theorem's statement and proof. Specifically, we will change the quantification from "For any residual direction $v \in V_T$ ..." to "There exist residual directions $v \in V_T$..." and update the proof accordingly.

5. Essential References Not Discussed

Thank you for highlighting these references. We will incorporate citations and discussion into the Related Work section, and we will definitely perform a careful pass to ensure all technical terms are clearly defined upfront and used consistently.

Nevertheless, we are grateful the reviewer found our idea novel and supported by solid experiments. We hope these clarifications and proposed revisions address your concerns, and we would appreciate it if you would reconsider your evaluation based on our response.

Thank you for the constructive feedback. We appreciate that the reviewer found our core claims well-supported and our analysis novel. We will address each of your concerns below.

Q1: Limited Evaluation: The empirical evaluation is not extensive enough.

Thank you for these valuable suggestions. We will address each point individually.

Evaluation on larger models (Methods And Evaluation Criteria). Our evaluation focused on Llama 3.1 8B as a case study for mechanistic interpretation. We agree that evaluation on models of different scales would further strengthen our claims. To this end, we now include a safety residual space analysis for Llama 3.2 3B to verify that our findings generalize across scales. As shown in the figure below, the residual space for Llama 3.2 3B shows similar low-rank linearity and interpretability to Llama 3.1 8B, which clearly support the generalizability of our findings.

[View figure of analyzing Llama 3.2 3B]

We will add this additional figure to the Appendix in the revised paper and reference it in the main body.

Evaluation on real-world safety scenarios and comparisons with existing methods. (Methods And Evaluation Criteria) Thank you for this insightful feedback. Our current evaluation covers over 6 strong jailbreak baselines, representing different real-world adversarial attack strategies. We also compared the effectiveness of our trigger removal attack against these jailbreak methods. We agree that expanding the scope of our safety residual space analysis to more scenarios would be valuable, and we plan to explore this in future work.

Evaluation on the full linearity of safety features. (Claims And Evidence) Thank you for this insightful suggestion. Our framework does not strictly require that safety features are fully linear. In the following table, we evaluate how well activation shifts during safety finetuning can be approximated by linear affine maps. We found that the Mean Square Error between finetuned activations and the approximation is negligible compared to the Norm of activations. This suggests safety finetuning mostly introduces linear safety features.

[View table of residual space map MSE]

We will report these results for our learned linear map in the revision to strengthen the claims regarding the linearity of safety features.

Q2: Scalability: The iterative trigger removal process may be computationally expensive.

Thank you for this suggestion. While iterative trigger removal serves mainly as an analytical tool, its computational cost (at most 30 attempts per sample in our settings) is comparable to other iterative methods like TAP [1] (avg. 35 attempts) and PAIR [2] (avg. 37 attempts). The computational requirement is feasible in practice. We will add a comparison to these baselines in the paper, following the description of the trigger removal algorithm.

Q3: It would be beneficial to include an ablation study that explicitly removes certain directions.

Thank you for pointing this out. We believe there may be a misunderstanding, as we did provide an ablation study removing certain directions to measure changes in model behavior (Figure 4 for refusal ability and Appendix C.4 for general ability). We will make the reference to this study clearer in the revision.

Q4: Further details on data balancing and sampling strategies could be provided.

Thank you for your constructive suggestions. Details on our data balancing and sampling processes are provided in Appendix C.1. We will add a more direct reference to this section in the main text to make this information more visible.

Q5: Notation Clarity: Some mathematical terms and notations, such as zero-utility directions and residual shifts, could be clarified for better accessibility.

Thank you for pointing this out. We will clarify these terms in the revised paper to improve accessibility:

• Residual shifts refer to the change in activations within the safety residual space before and after finetuning.
• Zero-utility directions refer to directions in the safety residual space that do not contribute to changing the model's safety behaviors.

In the revision, we will add these definitions where the terms are first introduced.

[1] Mehrotra, Anay, et al. "Tree of attacks: Jailbreaking black-box llms automatically." Advances in Neural Information Processing Systems 37 (2024): 61065-61105.

[2] Chao, Patrick, et al. "Jailbreaking black box large language models in twenty queries." arXiv preprint arXiv:2310.08419 (2023).