Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety

Q1.Virus and some other missing relevant work

Thanks for the suggestions. We have added all the work to our manuscript.

Q2.1 Additional experiment on GSM8k Q2.2 Utility downgrade raises concern on fine-tuning

Thanks for the great question.

Fine-tuning on GSM8K We fine-tune LLaMA-2-7B-Chat on 100 samples selected via random, BA, and our method. The train-test split follows (He et al. 2024). Test accuracy is the percentage of correct answers, and HS is measured on the Hex-Phi dataset with GPT-4o as the judge.

As seen, our method successfully unaligns the model, and fine-tuning slightly improves test accuracy, confirming its downstream benefits.

Why utility downgrades: Unlike GSM8K, where utility is measured by test accuracy, Table 1 uses MT-Bench to evaluate general generation quality. A possible reason is that after fine-tuning, response lengths tend to be shorter, possibly to mimic the Dolly-style concise responses. Since MT-Bench relies on model-based evaluation, potential verbosity bias in the judge model[1] may contribute to the utility downgrade.

Utility downgrade is also observed in Table 8 of [3]: Even fine-tuning on the full benign dataset leads to a slight drop. While scores slightly decrease, the generations are of similar high quality from human inspections.

Q3.Logistical flow for the attack method design

Lines 147-161 outline our intuition: for an aligned LLM, clean and safe samples lie comfortably within the safety distribution, while harmful samples become “outlier” samples that lie outside the safety scope. We hypothesize that certain outlier samples within benign datasets may have a high potential to push LLM parameters toward harmful zones during fine-tuning. The following experiments are used to validate this intuition (Fig. 3&4).

Q4.What will happen to a dataset where most samples are short samples

The short samples will be detected.

We'd like to discuss a potential misunderstanding of the outlier samples. Our outlier detection is to find some outliers that are away from the model's alignment space, but not to detect relative outliers in the given dataset (see Reviewer RTza Q6). Thus, these short samples are selected, potentially because of their weird length compared to the training dataset of the aligned LLMs. Thus, regardless of the fine-tuning dataset's length distribution, the vanilla Self-Inf score always favors shorter samples. Therefore, the finding is not contradictory.

Additionally, while short samples are effective, the revised Self-Inf-N, which normalizes length bias, is more useful in capturing 'harmful benign samples.' We welcome further discussion!

Q5. Advantages of our method to BA.

• A novel outlier-based approach to breaking safety alignment.
• A significantly more efficient method, requiring only one-third of BA's runtime.
• No reliance on an anchor dataset.
• More practical applications, e.g, continuous learning.

Additionally, our primary goal is not to surpass BA in harmfulness. An average score >3 is sufficient to cause substantial safety degradation. Furthermore, since GPT-4o models serve as the judges, score variations might also exist.

Q6.The method seems to be a direct application of (Pruthi et al., 2020)

Thanks. Please refer to Reviewer RTza Q2.

Q7.Naming issues(Malicious Fine-tuning)

Thanks. We have revised our manuscript.

Q8.Lisa: Beavertails vs. RepNoise

Yes, we use original Beavertails. The table below presents the Lisa defense results with repnoise_beavertail (same settings as Figure 8(a)), where refusal responses are used. The results show that the new Beavertail dataset effectively suppresses LLM harmfulness during fine-tuning.

This experiment made us reconsider our earlier claim that the bi-state optimization strategy has limited effectiveness in suppressing the harmfulness of benign samples. We have now included this experiment and revised our arguments accordingly in the updated manuscript.

Q9.Different observations in Lisa's Table 15.

As discussed in Q8, Lisa’s performance improves significantly with repnoise_beavertail. We've updated our statement to reflect these findings, which partially align with Table 15 of Lisa's paper.

However, key questions remain: Why does the alignment dataset influence Lisa’s performance? And how can we design better defenses against benign fine-tuning attacks? These might be interesting for future readers.

Q10.Why benign fine-tuning attack fails in Lisa's testbed.

Lisa seems to fine-tune directly with BA-selected Alpaca samples, but the sample scores were computed using a LLaMA-Chat model in full fine-tuning mode. While BA's paper shows some transferability, a more direct approach would be replicating the scoring process with a LLaMA-NonChat model in LoRA mode.

[1]https://arxiv.org/pdf/2310.10076

Q1.The difference between our paper and [1]

[1] raises an important observation: even fine-tuning on benign samples can lead to a certain degree of safety degradation. However, despite this initial investigation, two key questions remain unexplored:

• The increase in harmfulness is quite limited and occurs only in certain categories (see Figure 1 in [1]).
• Why does safety degradation happen? [1] attributes this phenomenon to "catastrophic forgetting" on the utility-oriented dataset but does not delve into deeper investigations.

To address these questions, we take a step further by investigating whether a specific subset of the benign dataset contributes disproportionately to safety degradation. We approach this from a novel outlier detection perspective. We believe that the insights gained from our outlier detection analysis (as outlined in Q2) will enhance the community’s understanding of LLM vulnerabilities.

Q2.Technical improvement does not sufficiently advance the understanding of LLM safety vulnerabilities.

Technical improvement. While Self-Inf is a well-established method [2,3], we are the first to adapt it for benign fine-tuning attacks. Importantly, our proposed Self-Inf-N introduces non-trivial refinements(mitigating length bias), which lead to significantly stronger attacks on LLMs (see Figure 4).

The core objective of our paper is to uncover vulnerabilities in LLMs during fine-tuning. We believe our contributions offer meaningful insights for the community:

• Short samples can severely degrade safety alignment (Figure 3; Section 3.3).
• Self-Inf-N, a refined outlier detection method, improves attack effectiveness (Figure 4).
• Strong empirical results show that Self-Inf-N misaligns LLMs across model sizes and architectures, and remains effective in continuous learning and data poisoning scenarios (Figure 4, Table 1, Figure 5).

Q3.The main experiments are biased toward Llama-2 models

Thanks for the great question. We also evaluate our method with other more advanced methods like Qwen2-7b-instruct. The following table shows the results.

It is shown that, our method still successfully leads the fine-tuned Qwen model to a safety degradation problem. We have added the experiments to our updated manuscript.

Q4.Lack experiments with closed-source models.

Thank you for the great question. Our method requires gradient access, which is not feasible for closed-source models. However, we evaluate the transferability of samples selected using LLaMA-2-7B-Chat to GPT-3.5 and GPT-4. The results show limited impact, likely due to (1) large architectural differences between the two model series and (2) (potential) stronger safety mechanisms in closed models. We believe this is a valuable direction for future research.

Q5.How does the proposed method impact deployed systems?

Escaping existing safety moderations. Firstly, our method can successfully escape the moderation tools (Sec 4.6), while the harmful samples would be easily detected.

Benign Fine-tuning breaks safety Our attack method is effective for the popular fine-tuning-as-a-service setting for the deployed LLM: when the user uploads the selected "seemingly" benign samples, the safety of the fine-tuned LLM would be broken.

Q6.Comparisons with other outlier detection methods.

Results with DBScan We conducted a simple experiment using DBSCAN, applying it to the last-layer hidden representations to identify 100 samples whose embeddings differ most from the overall distribution. The results on the Dolly dataset are shown in the table below. As observed, DBSCAN demonstrates significantly lower effectiveness compared to our method.

Why do density-based and DL approaches fail? We believe the key difference lies in the objective: rather than identifying relative outliers within the dataset, our goal is to find samples that most strongly influence the deviation of the aligned model’s parameters. In other words, we intend to find samples that are most "shocked" (influential) to the aligned model $f_{\theta}$. Density-based and DL approaches, however, are not suitable for our setting as they mainly focus on finding relative outlier samples in the given dataset.

Why use (Pruthi et al., 2020)? We chose the method from Pruthi et al. (2020) due to its proven effectiveness and ease of use in LLMs, as supported by recent work in instruction tuning data selection [2] and outlier analysis [3]. Additionally, compared to other gradient-based influence estimation methods (e.g., Data-inf), it offers significantly better memory efficiency.

[1]Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety

[2]Less: Selecting influential data for targeted instruction tuning

[3]Outlier gradient analysis: Efficiently improving deep learning model performance via hessian-free influence functions.

Q1.Why using API-based detection tools?

Thank you for the insightful question. We totally agree that the suggested evaluation model can further enhance the robustness of our analysis, hence we have already included it in our updated results.

Each cell shows two values: the number of detected safe samples and unsafe samples, respectively. Our method clearly outperforms the harmful dataset.

We also respectfully argue that API-based and model-based detection tools are fundamentally similar, as API-based tools also rely on underlying models trained on large, crowd-labeled datasets for general-purpose safety detection (similar to how ChatGPT operates). We chose API-based tools due to their widespread adoption in both academic research[6] and commercial applications[1].

Q2.Why not considering other safety benchmarks, e.g., BBQ, ToxicChat, and XSTest.

We fully agree that including additional benchmarks leads to a more comprehensive evaluation. Hence we have incorporated both the BBQ and XSTest datasets to further evaluate our method on LLaMA-2-7B-Chat. For XSTest, we followed the original paper's setup and used GPT-4 to classify the model’s responses to unsafe queries into three categories: full_compliance, full_refusal, and partial_refusal.

For BBQ, we computed the bias scores on both ambiguous and disambiguated subsets (higher scores indicate more bias):

As seen, LLM fined-tuned with our method exhibits a more compliance to the harmful query prompts in XSTest, and a more biased outputs in BBQ, compared to BA and un-finetuned version.

The limitations of ToxiChat, BBQ, and XSTest

• ToxiChat primarily evaluates toxicity detection tools (e.g., OpenAI Moderation API), which falls outside the scope of our work.
• BBQ focuses on bias in models' generations, which is just one dimension of safety.
• XSTest is indeed a good benchmark, which contains ten safety categories, , but mainly used to study over-refusal behaviors. Compared to HEx-PHI, it has fewer unsafe prompts (200) and covers safety categories less comprehensively.

Why we use Hex-PHI? Hex-PHI is a well-established benchmark for evaluating LLM's safety. It is grounded in the 'exhaustive lists of prohibited use cases found in Meta’s Llama-2 usage policy and OpenAI’s usage policy.'[2], covering a diverse range of safety categories such as illegal activity, privacy violations, and child exploitation. It has also been adopted in several recent works [3,4,5], making it a strong choice for evaluating LLM safety in our context.

Q3.Prompt for the judge model

As noted in Appendix A, 'The prompt provided to the judge model is identical to that used in [2]'. Please refer to [2] for more details.

Q4.Corroboration of the judge model's effectiveness

Thank you for the thoughtful question. Following your suggestion, we further evaluate the generated 330 responses on Hex-PhI using additional guard models: LlamaGuard, GraniteGuard, and WildGuard.

These results lead to the same conclusion that our method leads to significantly more unsafe responses. However, through manual inspection, we identified two limitations of these tools: (1) High false negative rate – many harmful outputs are incorrectly labeled as safe. (2) Limited granularity – guard models are trained on predefined categories and may not detect nuanced harms (e.g., economic harm in HEx-PHI is hard to be detected). Therefore, we believe LLM-as-a-judge remains a more reliable evaluation method for our setting.

Using LLM-as-a-judge is a widely accepted practice for safety evaluation, as adopted by several well-established works [2,3]. (Reviewer UXrg and RTza also noted, GPT-4 as a judge for assessing harmfulness is a standard metric).

Q5.GPT-4 vs. GPT-4o

Thanks for your detailed review. That was indeed a typo, we used GPT-4o in our experiments.

[1]https://medium.com/jigsaw/reducing-toxicity-in-large-language-models-with-perspective-api-c31c39b7a4d7

[2]Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024)

[3]Safety alignment should be made more than just a few tokens deep. (ICLR 2025)

[4]Artprompt: Ascii art-based jailbreak attacks against aligned llms. (ACL 2024)

[5]SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding. (ACL 2024)

[6]Bias and fairness in large language models: A survey

Q1. The paper discusses practical scenarios like continuous learning and data poisoning, but the experiments are somewhat limited in scope.

Thank you for the insightful question. The continuous learning and data poisoning settings represent our preliminary exploration into how the proposed method might realistically threaten LLM alignment in practice. In particular, we incorporated practical considerations such as a real-world medical downstream task (Asclepius) and experiments with low poisoning ratios (see Section 4.4.2). While we acknowledge that our current work cannot exhaustively cover all practical scenarios, we hope these initial investigations can serve as a foundation for future work in this important area.

Q2. The authors suggest that benign samples could avoid toxicity detection which is trivial as the data are benign samples but the toxicity detection tools are designed for harmful training data. Is there any design suggestion in mitigating safety issue during fine-tuning?

Thank you for raising this important point.

Any Mitigation Strategies During Fine-tuning? We have explored additional mitigation strategies in Section 4.6, including both data augmentation and fine-tuning-stage defense methods. Specifically, we evaluate whether incorporating extra alignment samples can mitigate harmfulness, and we also test advanced fine-tuning defenses such as LISA [3]. Empirical results show that augmenting the fine-tuning dataset with safety samples from the Bianchi dataset would effectively suppress the harmfulness.

Why do we incorporate toxicity detection? We agree that it is expected for benign samples to bypass toxicity detectors, and we include this analysis primarily to highlight the stealthiness of such attacks in practice.

Q3. Although using a different approach, there are some concurrent works that could be discussed in a future revision:

Thanks for these suggestions. We have included the suggested papers in our updated manuscript!

Q4. This paper did not discuss any corresponding defense mechanism in addressing the potential threats.

In Section 4.6, we identify a promising mitigation strategy: augmenting the fine-tuning dataset with a subset of safety-aligned data from Bianchi et al., which appears to reduce the harmfulness of LLM outputs. We also discuss the limitations of existing defense methods and emphasize the need for more advanced techniques specifically designed to address benign-sample-based alignment attacks. We hope this work can help motivate further research in this critical direction.

[1] Hsiung et al. "Your Task May Vary: A Systematic Understanding of Alignment and Safety Degradation when Fine-tuning LLMs" (2024)

[2] Mu et al. "Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring" (NAACL 2025)

[3] Huang T, Hu S, Ilhan F, et al. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack[J]. Advances in Neural Information Processing Systems, 2024, 37: 104521-104555.