Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning

Thank you for your thoughtful feedback on our work. We appreciate your recognition that this is a forward-looking work in a new domain that will gain importance. We are also grateful for your acknowledgment that our experiments are well-designed and comprehensive, and that our design of VAA based on GDRO is a good fit and the first application to HFT scenarios. We are pleased that you recognize our method works well and achieves SOTA results. We will address your comment below.

Response to the question: I wonder if you didn't poison your data yourself, and had to determine which subsets were harmful, how would that be realized?

Thank you for your thoughtful question, and we apologize for the lack of clarity in the original manuscript.

At stage 1, we simulate harmful fine-tuning (HFT) using proxy data. The goal of this stage is to partition the alignment data into vulnerable and invulnerable subsets. This method is generally applicable and independent of the actual downstream poisoning process.

The alignment data can be partitioned in this way; it is not grounded in theoretical assumptions, but rather in empirical observations of large-scale heterogeneous alignment data. Specifically, we observed that certain examples are more likely to be forgotten during HFT, and we refer to these as vulnerable samples. Our experiments (Figure 2) show that different poison data—even with 0% poison—result in common forgetting. This enables us to use proxy data to simulate HFT. Our goal is to estimate, in a data-driven manner, which portions of the alignment dataset are more susceptible or robust to such fine-tuning, even in the absence of explicit data poisoning.

To operationalize this, we begin with an aligned model that performs well on its training set. We then simulate HFT by fine-tuning the model on a proxy dataset (Alpaca) augmented with 10% randomly sampled harmful data. During this process, we evaluate the model's predictions on the alignment training set over T iterations and compute how many times each example is forgotten, denoted as ForgotNum (Equation 1).

Examples with ForgotNum = 0 (i.e., never forgotten across T iterations) are assigned to the invulnerable group, while all others are assigned to the vulnerable group. This empirical partition serves as a prior for estimating data vulnerability, which our algorithm leverages in subsequent stages.

---

Thank you once again for your professional and inspiring feedbacks. We hope that our responses could effectively address your comments, and we look forward to any further feedback you may have.

Thank you for your insightful and positive feedback. We appreciate your recognition of our adaptation of the Group DRO framework into a two-player game between a hard sampler and the LLM as both interesting and technically sound. We also value your acknowledgment that our extensive experimental results generally support the method's effectiveness. Below, we address your comments with additional analysis and new experimental results.

Response to W1: Experiments on More LLMs

Thank you for the suggestion. We extended our experiments to include VAA and all baselines on Qwen2.5-7B, with results shown in Table 1.

While performance varies across models, VAA consistently outperforms all baselines. Notably, as HFT epochs increase, the performance advantage of our method becomes more pronounced, further demonstrating its generalization ability. We will include these results in our revision.

Table 1. Experiments on Qwen2.5-7B.

Response to W2: Analysis of Convergence Trends

Thank you for your insightful suggestion! Indeed, in convex optimization scenarios, adversarial training reaches an equilibrium solution (saddle point) at convergence. However, in large-scale, non-convex LLM training, models often employ "few-pass training" due to computational constraints, stopping after several epochs—far from full convergence. This highlights a practical gap between LLM training and classical optimization theory.

Despite this, we observe a clear convergence trend in our adversarial sampling setup: the weights assigned to vulnerable groups increase over time, while those of less vulnerable groups decrease. This dynamic evolves in response to group-wise loss, as shown in Table 2, and indicates that the model progressively balances performance across groups.

Table 2. Evolution of Group Weights and Loss During Training (Vulnerable: Invulnerable)

We will include this discussion and analysis in our revision. We appreciate your thoughtful feedback on this important point.

Response to W3: Discussion of Computional Overhead

Thank you for raising this point. We measure efficiency by the number of backpropagation steps (the dominant training cost). Below is a comparison:

VAA saves an average of 0.5×BP computation compared to the fastest baseline (Vaccine) by employing a curriculum learning strategy that gradually warms up the perturbation probability from 0% to 100%.

In practice, SFT for a 7B model finishes in under an hour, making the added cost acceptable given the substantial safety gains.

We will include this discussion in the revision.

Theoretical Support

Due to space constraints, please refer to our response to Reviewer g5ub.

Response to Dataset Style Mixing

Thank you for bringing this up. We followed the experimental settings from previous works (Vaccine, Booster, etc.), which indeed introduces distribution variations. Given the complexity of real-world user behavior, we consider three possible scenarios:

1. Standard users: Real-world data naturally contains mixed, heterogeneous content
2. Malicious users: Deliberately injecting harmful data to undermine alignment
3. Standard users: Harmful content naturally occurring within homogeneous datasets

Our setup addresses the first two scenarios. We acknowledge the importance of the third scenario and plan to construct homogeneous test benchmark for HFT in future work.

We appreciate your thoughtful feedback and will add this discussion in our revision.

---

Thank you once again for your insightful comments, which has undoubtedly strengthened our work. We hope that our responses could effectively address your concerns, and we look forward to any further feedback you may have.

Thank you for your professional and encouraging feedback. We are pleased that you recognize our work as the first to reveal uneven vulnerability under HFT. We also appreciate your recognition of VAA as a novel method that significantly reduces harmful scores while preserving performance, and your acknowledgment that our claims are well supported by convincing evidence.

Below, we address your points with supporting analyses and new results.

Response to W1: Theoretical Support

To explain the principles of VAA, we present a variance-based perspective.

Variance Decomposition under HFT. For any distribution $P \in \mathcal{Q}$ and parameter perturbation $\delta$ induced by HFT, the variance decomposes according to the Law of Total Variance:

Var_P[ℓ(θ* + δ)] = E[Var_G_i[ℓ(θ* + δ)]] + Var[E_G_i[ℓ(θ* + δ)]]

The first term captures within-group variance, and the second term captures between-group variance.

Discussion:

• The robust loss (group-wise perturbation module), reduces within-group variance by simulating perturbations during alignment, thereby improving robustness at the group level.
• The GDRO (adv sampler module) reduces between-group variance by improving the performance of the most vulnerable group, leading to more balanced robustness across subpopulations.

Together, these components enable VAA to address both parameter sensitivity within groups and imbalanced vulnerability across groups under HFT.

Response to W2: Experiments on More LLMs

Thank you for the suggestion. We expanded our experiments to include Qwen2.5-7B, with results shown in Tables 1.

While HFT sensitivity varies across models, VAA consistently outperforms all baselines. Notably, as HFT epochs increase, the performance advantage of our method over baselines becomes more pronounced.

Table 1. Experiments on Qwen2.5-7B.

Clarification on Figure 2

We apologize for any confusion in our initial representation and provide the following clarifications:

Q1: “The meaning of ‘Common’ is unclear.”

In Fig 2, “Common” refers to the intersection of forgotten (or unforgotten) examples across different poisoning ratios. We define:

Common = $\frac{|A_1 \cap A_2 \cap A_3|}{N}$, (1)

CommonRatio = $\frac{|A_1 \cap A_2 \cap A_3|}{\min(|A_1|, |A_2|, |A_3|)}$. (2)

Here, $A_i$ is the forgotten (or unforgotten) sets under a setting; $N$ is the dataset size. In the figure, non-shaded bars represent forgetting in a setting ($|A_i|/N$), while shaded bars represent overlap across different settings (Eq. 1).

Q2: “SST2 (p=10%) looks identical to p=20% in Fig 2(a) but not (b). Is this a mistake?”

There is no mistake. In Fig 2(a), the shaded regions indicate examples forgotten under all settings (Eq. 1), which are identical across poison ratios. The non-shaded heights differ slightly, with 20% showing more forgetting than 10%.

Compared with Fig 2(b), non-shaded heights for SST2 (p=10%) remain the same (Ai/N), but shaded regions differ as they reflect intersection of forgotten examples across different settings

Q3: “...claims significant forgetting overlap, but Fig 2 does not show this.”

This is reflected in Figure 2(a), where the shaded portion occupies most of the 0% bar—indicating a high CommonRatio. This suggests that examples forgotten under the clean setting are also frequently forgotten under poisoned settings.

We will make revisions for clarity.

Explanation of Stage 1

Thank you for your question. In Stage 1, we estimate which portions of alignment examples are more vulnerable to HFT. The process works as follows:

1. We begin with an aligned model and simulate HFT using Alpaca data mixed with 10% harmful content.
2. During the HFT process, we record T predictions and compute ForgotNum—the number of times each example is forgotten (using Eq. (1) in manu).
3. Examples with ForgotNum = 0 (never forget) are deemed invulnerable; others are considered vulnerable.

This estimation provides prior knowledge about data vulnerability that our algorithm uses in later stages. We will clarify this process in our revision.

---

Thank you again for your constructive feedback, which has undoubtedly strengthened our work. We hope our responses could effectively address your concerns and welcome any further suggestions.

Thank you for the professional and constructive feedback! We are delighted that you recognized our adaptation of GDRO to address a new and important problem, and appreciated the various interesting observations in our work. We're glad you found our experimental setup largely acceptable and our results demonstrating VAA's improvement over existing methods particularly encouraging.

We address your suggestions below with additional analysis and results.

Response to W1: Deeper Analysis of the Method

We address the suggestions in three parts: design rationale, variance-based explanation and training stability (curriculum learning).

1. Design Intuitions

VAA integrates two components—robust loss and group DRO (GDRO)—each addressing a distinct challenge posed by HFT:

1. Robust Loss (Parameter Perturbation): HFT degrades performance via parameter shifts from θ to θ′. To model this, we use a surrogate objective, min_θ L(θ') = min_θ L(θ + ε), to proactively simulates these shifts during alignment, helping LLMs to defense against HFT.

2. GDRO (Adversarial Sampler): Our analysis (Claim 1) shows ~30% of alignment data is vulnerable to forgetting under HFT, while ~70% is relatively robust. This data imbalance leads to "gradient starvation [1]" under ERM, where gradients from the smaller group are dominated by those from the larger group, resulting in "imbalanced vulnerability". GDRO mitigates this by upweighting underperforming groups, encourage balanced learning between groups.

Together, these modules incorporate two key priors into the alignment process: parameter sensitivity and imbalanced vulnerability across the dataset.

2. Variance-Based Explanation

To provide deeper insight, we present a variance-based perspective.

For any distribution $P \in \mathcal{Q}$ and parameter perturbation $\delta$ induced by HFT, the variance decomposes according to the Law of Total Variance:

Var_P[ℓ(θ* + δ)] = E[Var_G_i[ℓ(θ* + δ)]] + Var[E_G_i[ℓ(θ* + δ)]]

The first term captures within-group variance; the second captures between-group variance.

• The robust loss reduces within-group variance by simulating perturbations.

• The GDRO reduces between-group variance by improving the performance of the vulnerable group, leading to more balanced robustness.

3. Training Stability and Curriculum Learning

We analyze training stability in both modules:

LLM Module: Training stability of LLM is primarily affected by parameter perturbation. In Table 6 in manu, we analyze the effect of perturbation strength on performance. Now we further analyze its impact on training instability.

With small perturbations, training is stable but defense is limited. Larger perturbations improve defense but destabilize training. Curriculum learning (CL) addresses this trade-off. As shown in Table 1, CL enables stable training under larger perturbations, providing better defense.

Table 1. Training Stability Under Large Perturbation

Adversarial Sampler: The sampler's training stability depends primarily on learning rate. As analyzed in Table 7, we selected an appropriate rate that enables smooth updates. No significant stability issues were observed in practice.

Clarification of Terminology

We'll consistently use "vulnerable group" throughout.

Response to W2: Experiments

1. Reporting Variance

In Tab 2, we conducted additional runs of SFT and VAA under three random seeds. VAA consistently outperforms SFT with low standard deviations, indicating significance.

Table 2. Performance (mean ± std)

2. Analysis of Baseline Failures

RepNoise & Booster: Both implement explicit unlearning for harmful inputs (RepNoise corrupts internal representations, Booster prevents gradient descent), creating objective conflicts that impair generation abilities on tasks, compromising both harm mitigation and task performance.

Vaccine: While applying robust learning, Vaccine overlooks data imbalance, leading to "gradient starvation" where vulnerable groups receive insufficient updates, limiting its effectiveness.

---

Thank you for your constructive feedback and detailed suggestions, which has significantly strengthened our work. We hope our responses address your concerns and welcome any further feedback.

---

Reference

[1] Gradient Starvation: A Learning Proclivity in Neural Networks. NIPS 2021