LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning

We are grateful to the reviewer for recognizing the applicability of our method, as well as the results and ablation studies. Below, we provide detailed responses to each of your questions.

According to the align formula of this paper: delta = align - base. This method has an strong assumption: access to both aligned and unaligned model weights, which seems to be increasingly difficult to hold in the face of increasingly large and black-box LLM model deployments.

We thank the reviewer for raising this concern. We would like to emphasize that our assumption applies to scenarios where the person performing the safety alignment is also interested in ensuring its robustness. For example, the vendor of the aligned models. In such cases, LoX can be effectively used to augment the alignment.

Adding some newer and more capable baseline models—such as DeepSeek-R1-7b/14b[1]—could improve the robustness and generalizability of the evaluation.

We appreciate the reviewer’s insightful suggestion. In response, we have aligned DeepSeek-R1-Distill-Qwen-1.5B using the same methodology described in our paper. The resulting model already demonstrates strong robustness, with ASR values of 0.02 on GSM8K and Alpaca, and 0.01 under the Identity Shifting Attack. Given this high level of robustness, additional robustness is unnecessary in this case, as essentially no safety degradation is observed. We would like to emphasize that our primary contribution lies in addressing safety degradation when it does arise.

Theoretical justification for why low-rank extrapolation improves safety is limited and mainly empirical. The entire approach is based on a core assumption that critical security-related information resides in low-rank subspaces of the model, and that fine-tuning erodes these specific subspaces. While the paper provides empirical evidence for this assumption, the theorical explanation behind the evidence seems to be lacking. Can some theoretical explanations be given for some smaller models?

We thank the reviewer for this suggestion. We agree that this could be a limitation for an empirical research. But we believe our empirical contributions are sound and novel as reviewers agree. We will leave this for future work.

We are grateful to the reviewer for acknowledging the quality of our writing, as well as the simplicity, efficacy, and thorough analysis of our technique. Below, we provide detailed responses to each of your questions.

A potential risk is that bad actors might exploit the technique to reduce model alignment. Specifically, since the alpha parameter can be set to a negative value, it could be used to amplify safety degradation.

In the Ethics section, please clarify how the risk of misuse (e.g., setting alpha to negative values) can be mitigated.

We thank the reviewer for raising this concern. We argue that, for our method to work, it requires both an aligned (safe) and unaligned (unsafe) checkpoints. If the individual can apply LoX, it implies that they have access to the unsafe model. Therefore, the individual would have no motivation to diminish the alignment. We will update the ethics section to address this discussion.

Regarding the statement: “As k increases, changes in Rft/Ralign become less significant, highlighting the importance of the top-ranks on the safety robustness” — could you elaborate on why the top ranks are particularly important here?

In this experiment, we increase the samples in fine-tuning (marker size), which results in more safety degradation (y-axis). We hypothesize that the safety degradation is a result of the change of the ratio R_ft / R_align​ after fine-tuning. Therefore, we test if the safety degradation is correlated with the ratio R_ft / R_align​. In the figure, the correlation is more statistically significant when the rank k is smaller (i.e., top-rank subspace are measured by the ratio). In contrast, with larger rank k, the ratio is barely changed by the safety degradation. In other words, the top-rank subspace (the smaller rank) plays a more important role, impacting safety during fine-tuning, than the full-rank subspace.

Figure 2 is somewhat difficult to read due to small marker sizes. Could you adjust the graph to improve readability?

We appreciate the feedback regarding the readability of Figure 2 and will improve the marker sizes in the next revision of the paper.

Nit: Line 241 — typo: “Iddentity Shifting” → should be “Identity Shifting”.

We thank the reviewer for pointing out the typo and will correct it in the next revision.

We thank the reviewer for recognizing the applicability of our method and the studies conducted in our work. Please find our point-by-point responses to the questions below.

I would appreciate a more thorough discussion on why adding the full rank makes training unstable. What are the performance difference?

We appreciate your request for more discussion. But to our best knowledge, we did not claim that adding full-rank will make training unstable. Instead, we would like to clarify that adding full-rank components degrades the model prior to the fine-tuning attack; we do not claim any effect on fine-tuning dynamics. As stated in line 352, we hypothesize that this degradation is due to excessive noise in the bottom ranks. The impact of adding full-rank components compared to low-rank ones is illustrated in Figure 8 (model output examples) and Table 7 (utility metric comparison) in Appendix E. We will revise the main text to explicitly reference these results.

The safety landscape in section 5.3 is interesting. Do you have a hypothesis why adding this low-rank approximation of delta help models move to a flatter region?

Prior work [1] has shown that the safety landscape exhibits a basin shape, and that fine-tuning can drag the models that are close to the border out of the basin. We conjecture that alignment stops close to the border of the basin (due to gradient flattening), and that extrapolating in the safety direction (i. e. adding delta to the model) pushes the model deeper into the basin. The low-rank approximation performs better because it avoids extrapolating the noise present in the bottom ranks, allowing us to move “further” into the safe region without degrading the model’s outputs.

I am curious about the model after applying LoX and before training. What are the behavior changes?

As previously mentioned in the answer to the first question, Figure 8 and Table 7 in Appendix E show the effects of applying LoX prior to the fine-tuning attack. We also provide additional results comparing utility metrics for the base model (baseline), LoX, and the full-rank variant (ExPO), all evaluated with an extrapolation factor of α = 1.25.

The results show a degradation across all five metrics when comparing both LoX and ExPO to the base model, with LoX mitigating this degradation relative to ExPO. This indicates that using only top-ranks is capable of preserving more the original performance of the model, when compared to full-rank.

References: [1] Peng, S., Chen, P.Y., Hull, M., & Chau, D. (2024). Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models. In Advances in Neural Information Processing Systems (pp. 95692–95715). Curran Associates, Inc..

We thank the reviewer for recognizing the clarity of our writing, the effectiveness of our method, and the novelty of studying safety preservation in language model training.

“ fine-tuning can unintentionally alter low-rank subspaces of model weights related to safety of well-aligned LLMs, and finally result in diminished safety”. This is a pretty strong claim & hypothesis, it would be great if there are some theoretical analysis that can be conducted as evidence. For example, why finetuning only degradate the safety knowledge, how do we split the knowledge space of LLM?

First, we would like to clarify that we didn’t claim that fine-tuning affects only safety, as catastrophic forgetting is a well known problem in the ML community. If the reviewer could point to the part in the text that makes them understand that, we would be happy to edit it.

Regarding the theoretical analysis, we thank the reviewer for the suggestion. While we do not yet have strong theoretical insights, we believe our empirical contributions—specifically, the new insights on the safety subspace and our method for robustifying safety-aligned models—are both sound and novel, as acknowledged by the reviewers. We plan to explore the theoretical aspects in future work.

Lack some reference about the adversarial robustness of LMs, like [1, 2, 3]. Would be good to discuss and compare with those methods.

We thank the reviewer for sharing this related work. We provide a discussion comparing our work with theirs below. This discussion will be integrated in our final revision.

RIFT [1]: RIFT addresses the problem of catastrophic forgetting during adversarial fine-tuning by encouraging pre-trained language models to retain their robust, generic linguistic features. RIFT could be categorized as a fine-tuning stage defense, as it changes fine-tuning dynamics to retain information from the base model. Therefore, RIFT is not suitable for the scenario where a harmful attacker (i. e. a person with intention of jailbreaking the model through fine-tuning) has control over the fine-tuning process. On the other hand, LoX can be applied after alignment stage, being suitable for such scenario.

Textgrad [2]: TextGrad adresses the lack of a principled, gradient-based method for generating fluent adversarial examples in NLP, enabling more effective robustness evaluation and adversarial training. Their work focuses on classification tasks, and not text generation like ours.

RoAST [3]: ROAST tackles the lack of unified robustness in fine-tuned language models by combining adversarial perturbations with selective parameter updates, improving resilience across multiple robustness dimensions. Similar to RIFT, RoAST can be categorized as a fine-tuning stage defense, and therefore is also not suitable for the scenario described.