Improving Fisher Information Estimation and Efficiency for LoRA-based LLM Unlearning

Dear Reviewer ZX4F,

Thank you very much for your thoughtful comments and for taking the time to review our work. We truly appreciate your feedback and are glad that the additional experiments addressed your concerns.

As suggested, we will incorporate the isolation ablation studies into the final version. Your input has been very helpful in improving the quality and clarity of the paper.

Best regards,
Authors

Thanks for the detailed response and additional experiments. I have raised my rating and kindly request that authors include the isolation experiments in the paper for future versions.

$5$ Approximation Assumptions in Theorem 1

We sincerely thank the reviewer for highlighting important issues regarding the approximation assumptions in our theoretical analysis. We fully acknowledge and appreciate these insightful comments. To address them, we have revised and expanded our proof, explicitly stated four assumptions (A.1–A.4), and provided clear empirical validation.

For clarity and readability, here we highlight the key improvements. The full revised proof with detailed validations is provided in the following supplementary document (Full Proof). Please note that reviewing this supplementary proof is not necessary to understand our main response; it is included simply to provide additional details for completeness.

(1) LoRA Initialization (A.1)

Purely zero-initialized LoRA parameters ($A=0,B=0$) yield zero gradients, making meaningful importance estimation impossible. Empirically, we observe that overly small initialization (e.g., std=0.01) led to unstable gradients, whereas overly large values (std ≥ 0.40) violated Assumption A.1 by introducing excessive noise. Hence, we treat LoRA initialization as a hyperparameter tuned equally across all methods within 15 validation trials, confirming effective and stable initialization values are easily identifiable (e.g., std=0.05).

(2) Empirical Validation and Discussion of Approximation Assumptions

We fully agree with the reviewer regarding the necessity of empirically validating our theoretical assumptions. Therefore, we explicitly revisit and empirically validate each assumption as follows:

• Assumption A.1 (Negligible Magnitude of Initial Matrices): We empirically confirm that initial matrices $B\_0, A\_0$ and their associated terms ($B\_0A\_0, B\_0\\Delta A, \\Delta B A\_0$) are consistently at least 4 orders of magnitude smaller compared to the learned update term $\\Delta B\\Delta A$. This empirically justifies our approximation.

• Assumption A.2 (Independence between $\\Delta B$, $\\Delta A$): We empirically confirm that the resulting gradients follow approximately Gaussian distributions. We also measure the element-wise covariance between $\\Delta B$ and $\\Delta A$ across 500 gradient samples, separately for the forget and retain sets. The covariances are sharply concentrated near zero, supporting our approximation that the two matrices are effectively independent.

• Assumption A.3 (Independence among Distinct Elements within Each Matrix): Empirical measurements similarly confirm that off-diagonal covariance terms among distinct elements are negligible. Since each element follows an approximately Gaussian distribution, this further supports the assumption of statistical independence, validating the practical reasonability of this approximation.

• Assumption A.4 (Negligible Expectation Values): We find that the squared expectations are small but not strictly negligible. Still, we assume them to be negligible to significantly reduce memory and computation. This leads to a 100× memory and 40× speed improvement, with minimal impact on performance. Future work may revisit this assumption for further gains.

We hope this thoroughly addresses Reviewer 5shu's insightful comments. We greatly appreciate the reviewer's constructive feedback and will incorporate all improvements in our final manuscript.

$2$ Clarification on Equation 5

“The right-hand side of Equation 5 doesn't depend on $\\Delta W$.”

We appreciate the reviewer pointing out this issue. It was indeed a typo. We have corrected Equation 5 as follows to clearly reflect the dependence on $\\Delta W$:

$\mathrm{Var}\_{\mathcal{D}}[\Delta W] :=\mathbb{E}\_\mathcal{D} \left[ \left( \frac{\partial}{\partial W} \log p\_{W}(\mathcal{D}) \right)^2 \right] - \left( \mathbb{E}\_\mathcal{D} \left[ \frac{\partial}{\partial W} \log p\_{W}(\mathcal{D}) \right] \right)^2.$

$3$ Validity of the Expectation as an Importance Score

"Another question is that when $D\_f$ is different from $D$, the expectation of the score function is no longer zero, can we only take it as an importance score?"

To investigate this, we conduct additional experiments on two alternative importance measures:

• ExpILA: Uses the magnitude of the expected score function.
• AbsILA: Uses the expected magnitude of the score function.

For brevity, we present results only for the NPO setting (Phi-1.5B and Llama2-7B), but similar trends are consistently observed in the GD and IHL settings as well.

Key observations from the above experiments:

• ExpILA and AbsILA generally achieve comparable or better performance compared to FILA, particularly when the forget set size is small (Forget 1%).
• As the forget set size grows (Forget 5% and Forget 10%), VILA consistently demonstrates better performance than ExpILA and AbsILA.
• Importantly, these results empirically confirm our claim: when the distribution of the forget set differs from that of the entire data, the expectation values of the score function become different from zero. Thus, correcting the original Fisher Information estimation (as done in VILA) becomes essential for robust and accurate importance estimation.

We sincerely thank Reviewer 5shu for raising this insightful point. We will incorporate these empirical results and their analyses into our final manuscript to further clarify our method’s validity and effectiveness.

$4$ Clarification of Notations and Contributions

Reviewer 5shu pointed out a potential confusion regarding the notation $\\mathrm{Var}\_{\\mathcal{D}}$ used in both Equation 5 (variance of gradients for the full model parameters) and lines 417–419 (variance approximation under the LoRA-based efficiency assumption). This feedback enabled us to significantly improve not only our notation clarity but also the overall structure of our manuscript. We sincerely appreciate this valuable input.

Clarification: In our manuscript, the operator $\\mathrm{Var}\_{\\mathcal{D}}$ consistently denotes the empirical variance computed over dataset $\\mathcal{D}$. The difference between Equation 5 and lines 417–419 lies solely in their arguments (score functions):

• Equation 5: Variance computed for gradients w.r.t. original full-model parameters $W$.
• Lines 417–419: Variance computed specifically for gradients w.r.t. LoRA adaptor parameters $A, B$.

We will explicitly clarify in lines 417–419 that the variance calculation is performed specifically on the LoRA adaptor parameters' score functions.

Cause and resolution: We believe the underlying reason for the confusion was not merely due to insufficient notation clarification, but rather because our manuscript did not explicitly emphasize the distinct contributions of our proposed method:

• FI Correction: We correct FILA's incorrect assumption of a zero-expectation in Fisher Information estimation.
• Efficiency Improvement: We efficiently approximate Fisher Information using only LoRA adaptor parameters, rather than the full model.

These two contributions are developed separately and then combined into our final proposed method, VILA. However, our original manuscript did not clearly present this process. We believe that if this had been clearly explained, the confusion regarding notation would have been greatly reduced. Therefore, we will restructure our manuscript to present these two contributions individually and then explain how they are integrated into VILA.

We again thank Reviewer 5shu for this insightful comment, which has significantly improved the clarity of our manuscript.

$2$ Micro-level Impact Analysis

"What is the micro level impact of these design choices?"

We thank Reviewer WfNG for this insightful comment, which prompts us to conduct additional ablation experiments clearly isolating each proposed solution component:

• FI Correction (Eq. 5): Correcting the Fisher Information estimation.
• LoRA Approximation (Eq. 9): Approximating the full-model Fisher Information using LoRA adapter parameters to enhance computational efficiency.

We provide detailed ablation results using the Phi-1.5B model below:

Key observations from these experiments are:

• FI Correction alone consistently yields the highest unlearning performance across most settings, confirming that correcting Fisher Information significantly improves unlearning effectiveness.
• LoRA Approximation alone provides efficiency gains but achieves limited or no improvement over the FILA baseline.
• Our proposed method, VILA (combining both), achieves performance close to the FI Correction alone scenario while substantially reducing computational overhead. Thus, VILA effectively balances high unlearning performance with computational efficiency.

$3$ Additional Experiments and Insights

"One way to compensate would have been to dig down the analysis section to uncover some exciting learnings. That part is missing, making it another paper giving better numbers without deeper insights."

We fully acknowledge the concern and have conducted six additional experiments to investigate the micro-level impacts of our design choices. Below, we briefly summarize key insights from these analyses. More detailed experimental results and further analyses are available in the supplementary appendix (see Section B-G) at the following link: $Additional Experimental Results$ . Please note that reviewing these supplementary results is not necessary to understand our main response; it is included simply to provide additional details.

• In-domain Unlearning (Expectation ≈ 0): When the distribution of the forget set closely matches that of the entire dataset, the performance gains from FI correction diminish. This explicitly indicates that FI correction is particularly effective for scenarios with significant distribution mismatch. Notably, most unlearning scenarios inherently involve such distributional mismatches, underscoring the broad applicability and necessity of our FI correction.

• Expectation as an Importance Score: Using the raw expectation or its absolute value (ExpILA/AbsILA) provides reasonable performance for small forget sets but deteriorates as the forget set size increases. Importantly, these experiments empirically confirm that the expectation values of the FI score function become non-zero under realistic distribution mismatches. Thus, treating expectation values as zero, as done by existing approaches, is invalid. The expectation itself provides meaningful signals for parameter importance, clearly validating our corrected FI estimation approach (VILA).

• LoRA Initialization Sensitivity (Sigma Ablation): Our ablation study on LoRA parameter initialization shows that extremely small or large initialization standard deviations harm FI estimation accuracy and performance.

• Selective Application of Importance Map (25% layers): Interestingly, applying the importance map selectively to only the top 25% of layers achieves better unlearning performance than applying it to all layers uniformly. This demonstrates that our importance estimation is able to identify specific layers crucial for effective unlearning.

• Training Trajectories Analysis: Through detailed analysis of the unlearning trajectories, we demonstrate that VILA consistently maintains stable and superior performance across the entire training process. This indicates that VILA improves not only final performance but also robustness and stability during the unlearning procedure.

• Empirical Validation of Independence Assumptions (LoRA Parameters): Our theoretical analysis assumes independence both between LoRA update matrices ($\\Delta B$, $\\Delta A$; Assumption A.2) and among distinct elements within each matrix (Assumption A.3). Empirical covariance calculations across all parameters confirm that covariance terms, both between and within matrices, are consistently negligible (close to zero). These empirical observations support the validity of our independence assumptions.

Taken together, these micro-level insights clearly demonstrate that the superior performance and efficiency of our method stems from precise design decisions.

Remarks

We sincerely appreciate Reviewer WfNG's insightful comments and critiques. They have greatly assisted us in clarifying and strengthening our work. We will explicitly incorporate these insights into our revised manuscript to highlight both theoretical significance and empirical contributions clearly.

Thanks for your response. I think the reported results should be considered to be included in a revision. I'll keep my score unchanged.