Restoring Pruned Large Language Models via Lost Component Compensation

Dear Reviewer aFYT,

Thank you for reviewing our paper. The concerns you raised mainly focus on the novelty, clarification of our findings, and experimental details. We have carefully addressed all of these points, and we are glad that your suggestions have helped us improve the paper. Our responses [R] to your raised questions [Q] and weaknesses [W] are as follows.

[W1] Novelty clarification.

[R1] Thank you for your question. We respectfully disagree and would like to emphasize that our proposed RestoreLCC is conceptually and technically distinct from R1, R2, and their combination (R1+R2). We clarify the novelty of our method from both theoretical and empirical perspectives as follows.

(1) Unique Insight 1 – Focusing on a few discriminative heads (ours) rather than all modules (R1, R2 and other studies). As shown in Section 3 (Findings 1 and 2), we reveal that reintroducing lost components to selected pruned attention heads can significantly restore model performance. In other words, not all modules are useful. However, previous approaches that recover all attention heads and all FFNs result in lower efficiency and often limited recovery performance.

(2) Unique Insight 2 – Recovering discriminative components instead of principal components. Another key difference from R1, R2, and similar work is that RestoreLCC focuses on discriminative components/information rather than principal components. Previous methods emphasize recovering principal components or minimizing global losses such as MSE of hidden states. In contrast, our study (Findings 3 and Figure 3) shows that discriminative components are more effective than principal components for recovery. For example, Figure 9 shows that some minor components (indices > 20), even with extremely small eigenvalues, retain high magnitudes after tuning and contribute strongly to recovery. These components may not capture principal information, but they are highly discriminative. This phenomenon parallels the difference between PCA (capturing principal variance) and LDA (capturing discriminative directions) in classical machine learning theory. This insight also explains why we train coefficients for all components rather than only the top‑k principal components like previous studies.

(3) Methodological novelty – Leveraging the above insights and address key challenges.While these findings reveal promising opportunities, they also raise challenges. For example, how to identify discriminative heads? How to find discriminative components (possibly among minor components) and determine their coefficients? To tackle these issues, we design RestoreLCC, which integrates a contrastive probing module and a lost component compensation module. Although some terms may sound similar to those in previous work, the underlying mechanisms and method operations are fundamentally different and novel.

(4) Significant improvements over previous studies. We compared RestoreLCC with FLAP (R1) and EoRA (R2). As shown in the below table (same setup as lines 279–281 and Table 1), RestoreLCC outperforms EoRA by 2.95% on Wanda-pruned models and by 3.79% on SparseGPT-pruned models.

[W2] More details of probing classifiers.

[R2] Thank you for the question. We apologize for not providing enough details about the probing classifiers. The procedure is described in Section 4.1, and we will add the following clarification for the classifiers:

Specifically, for each classifier, we use the formulation: $y = \sigma(Wm + b)$, where $m$ is either $m^+$ or $m^-$, $W$ is a weight matrix (input dimension equal to the size of $m$, output dimension = 1), $b$ is a bias term, and $\sigma$ is the sigmoid function. The output $y \in [0,1]$ indicates contradiction or entailment. We use a cross‑entropy loss optimized with Adam (lr = 1e‑2). The dataset is split into training and validation sets with a 7:3 ratio, and models are trained for 100 epochs based on empirical experience. We report probing accuracy on the validation set and rank the importance of attention heads according to their probing accuracy.

[W3] Ablation study: number and effect of samples.

[R3] We apologize for not providing enough details.

For all experiments, we use the same sample sizes: general recovery – 1,000 Alpaca samples for probing and the full 52K for tuning; task‑specific recovery – 100 samples for both probing and tuning. The number of contrastive pairs equals the number of probing samples.

Sample size effect: for general recovery, 1,000 samples are sufficient (accuracy ≈ 58.8); for task‑specific recovery on BoolQ, 200 samples can be sufficient (accuracy≈ 76).

[W4] Details for baselines.

[R4] We apologize for not providing these details. For LoRA and DoRA, we use the same settings: α = 16 and rank = 8. Regarding the applied modules, we try two configurations: (1) [v_proj, o_proj], which tunes only the head output matrices; and (2)[q_proj, k_proj, v_proj, o_proj], which tunes all head matrices. The best results are reported in the paper.

For LoFiT, we experiment with 10%, 20%, and 30% of the heads and report the best results.

The training samples are same for all methods. The batch size is set to 8. The learning rate is 1e-4. The max sequence length is 512. The number of training epochs is 2 for global recovery and 5 for task-specific recovery. All experiments are conducted on a single H100 GPU.

[W5] CLarification on K.

[R5] We apologize for the oversight. The setting of $K$ is clarified as follows, and we will double‑check similar details to ensure correctness.

In our experiments, we set $K = 1$ (not 10) to obtain the reported results. We recommend using $K \leq 3$ because performance is consistently better in this range than with $K > 3$. Moreover, after the top 3 components, eigenvalues become very small, so additional components have negligible impact on the final results while increasing computation. Therefore, using $K \leq 3$ provides both better performance and efficiency.

[W6] Clarification on Finding 3.

[R6] Thank you for the question. We clarify Finding 3 as follows.

Unlike previous studies that assume principal components are always useful for recovery, we present a different insight: discriminative information can also reside in minor components rather than principal components. Principal components may capture high-variance but task‑irrelevant information, while minor components may encode task‑specific signals. This distinction is analogous to PCA vs. LDA in classical machine learning.

To verify this, we manually inspected the token distributions of each component (Eq. 6). For example, in BoolQ, the top tokens of some principal components (e.g., 1st or 2nd) may include stopwords like a, the, or unrelated words like walk. In contrast, some minor components have answer tokens like yes and no, which are highly relevant to the task. Such task‑specific minor components are more discriminative and recover performance more effectively. We further validated this hypothesis by manually selecting attention heads and comparing recovery using principal vs. discriminative components. Figure 3 illustrates this phenomenon. Figure 9 also validates this: the trained magnitudes of minor components can become larger than those of principal components due to their discriminative power.

Scaling factor: Eigenvalues of principal components can be more than 1000× larger than those of minor components. To ensure fair comparison, we apply a scaling factor (1000) when adding minor components so their contributions are comparable.

In summary, the key finding is that task‑specific discriminative information may reside in minor components, not just in principal components. This observation directly motivates RestoreLCC, whose goal is to leverage these discriminative components for effective recovery.

[W7] Clarification on Eq. 4 and Eq.8.

[R7] Thank you for the question. We would like to clarify that there is no error in either equation. In Eq. 4, the top‑k components are used only for probing (Section 4.1), whereas in Eq. 8 all components are used for recovery tuning.

This is because, in the first step, we use only the top‑k components to evaluate the discriminative power of each head, as minor components have extremely small eigenvalues and barely affect probing results. However, as highlighted in Finding 3, during recovery for the selected heads, important discriminative information may also lie in minor components rather than only in the top‑k ones.

Therefore, our method considers all components and learns a magnitude for each component (total of $d_h$) so that even minor but discriminative components can receive a large weight for recovery. Figure 9 also confirms this behavior. Therefore, in Eq. 8 it should indeed be $d_h$.

Thank you once again for reviewing our paper. We will incorporate the clarifications and discussions into the revised version. Please let us know if you have any further feedback, and we look forward to your reply.

Dear Reviewer aFYT,

Thank you very much for taking the time to review our rebuttal and for re-evaluating our paper. We are pleased to hear that most of your concerns, such as those related to the novelty and experimental aspects, have been satisfactorily addressed. We will thoughtfully incorporate all relevant points from the discussion into our revised manuscript.

Best regards,

Authors of Paper 7969

Dear Reviewer oP8x,

Thank you for taking the time to review our paper and for recognizing it as well‑motivated, novel, theoretically grounded, and successful. The concerns you raised mainly focus on comparisons with different head‑selection methods, EoRA, extension to quantized models, and clarifications on pruning. We have carefully addressed all of your concerns, and we are glad to see that, with your suggestions, the quality of the paper has been greatly improved. Our responses [R] to the raised weaknesses [W] and questions [Q] are summarized as follows.

[W1] Clarify pruning-related terms

[A1] Thank you for pointing this out. We apologize for not explaining these terms clearly. Below we clarify the terminology:

• LLM pruning: Current LLM pruning methods focus on pruning all the weight matrices (including attention matrices $Q, K, V, O$ and MLP block matrices) rather than the activations of each Transformer block. For example, Wanda removes 50% of the weights in each matrix of both the attention and MLP modules (setting them to zero), leading to an overall sparsity of 50%. Other methods follow a similar manner. Therefore, in our paper, all weight matrices in the LLM are pruned, and we focus on restoring them by compensating through attention heads. (The reasons for not using MLPs/FFNs are explained in our response R10 below.)

• Activation: An activation refers to the output of a specific module.

• Pruned activation: It is the activation produced by a pruned module. While the activations have the same dimension as those from the original model, they may be less informative because the underlying weight matrices are pruned. Note that the activations themselves are not pruned; only the weight matrices are. We acknowledge that the term “pruned activation” may be confusing, and will revise this terminology for clarity.

• Sparsity: It refers to the proportion of weight parameters that are pruned. The sparsity settings for our experiments can be found in Section 5. For Wanda and SparseGPT, each weight matrix is pruned with the same ratio (e.g., 50% parameters removed), resulting in an overall sparsity of 50%. For SlimGPT, a dynamic ratio is applied to different layers (e.g., 12.5% in shallow layers and 30% in deeper layers, giving an overall sparsity of 20%) based on a specific ratio-assignment formula.

• Performance: Based on your suggestions, we will explicitly use terms such as quality, execution speed, or efficiency in different contexts when discussing performance.

As the report results in Tables 1 - 3, our proposed method can be applied to all three different pruning paradigms.

[W2] Overhead details in Appex. C.

[R2] We apologize the for the missed details. For LoRA and DoRA, we use the same settings: α = 16 and rank = 8. Regarding the applied modules, we try two configurations: (1) [v_proj, o_proj], which tunes only the head output matrices; and (2)[q_proj, k_proj, v_proj, o_proj]`, which tunes all head matrices. The best results are reported in the paper.

For LoFiT, we experiment with 10% - 30% of the heads and report the best results.

The batch size is set to 8. The max sequence length is 512. All the experiments are conducted on a single H100 GPU.

The overhead is reported using the best hyperparameter settings, with the base model being LLaMA‑7B pruned by Wanda at 50% sparsity. The inference speed is measured as the ratio of the total testing time for general recovery (lines 279–281) of the restored model to that of the original pruned model.

[W3] Compare with other metric-selected heads.

[R3] Thank you for your suggestion. We have conducted additional experiments with two alternative head-selection strategies: (1) MSE-selected heads: selecting heads with the smallest MSE between the outputs of the dense and pruned models. (2) KL-selected heads: selecting heads with the smallest KL divergence. The results are as follows: our probing-based selection consistently identifies important heads and is more effective than other metric-based approaches.

[W4] line 347: top-10 should be top-5.

[R4] We apologize for the typo. We will correct “top‑10” to “top‑5” and carefully double‑check the paper for similar issues.

[W5] Discuss two references.

[R5] Thank you for you suggestions. We have carefully read both references and will add to our revised paper.

(1) (add to related work) Moore and Chaudhuri utilize activation noise to probe network structure and identify redundant neurons. (add to future work) They also discuss a novel way of leveraging activation noise for neuron identification. We believe activation noise could similarly be used to enhance contrastive probing, and we plan to explore this direction in future work.

(2) EoRA. (add to related work) EoRA provides a fine-tuning-free approach to recover pruned models by searching low-rank spaces in a task-specific eigenspace to minimize compression loss. (add to Table 1) We also compare our method with EoRA, and the results are summarized below (Detailed results can be found in R8 below). It can be observed that our method significantly outperforms EoRA.

[Q1] What weight tensors are pruned?

[R6] In our experiments, all weight matrices in both the attention and MLP blocks are pruned, following the exact pruning settings of the respective methods: Wanda, SparseGPT, and SlimGPT.

These three approaches are well‑established state‑of‑the‑art pruning methods, each representing a different pruning paradigm. The primary difference lies in the positional constraints of pruned parameters in a matrix. To demonstrate the generalizability of our method across different pruning strategies, we evaluate RestoreLCC on all three types of pruned models, as shown in Tables 1–3.

Our recovery operations focus on the attention modules in order to restore the performance of the entire model. We also discuss the possibility of recovering MLPs (FFNs) in our response [R10] below.

[Q2] Application to heavily quantized models.

[R7] Thank you for the question. Our method can also be applied to heavily quantized models. We conducted experiments with 4‑bit quantization on the pruned model (by Wanda) and report the recovery performance below. The experimental setup follows lines 282–285 and Table 3 in our paper. As shown, the proposed RestoreLCC successfully recovers the performance of heavily quantized LLMs.

[Q3] Comparison with EoRA.

[R8] Thank you for the question. We compare our method with EoRA, and the results are summarized in the table below. It can be observed that our method significantly outperforms EoRA. The experiment setup follows lines 279-281 and Table 1 in our paper.

[Q4] The used software and other overhead.

[R9] Thank you for your question. We report the details of recovering pruned LLaMA‑7B (1 H100 GPU, torch.bfloat16 precision, max_length=512, batch size=8, Alpaca Dataset).

The software we used is Python and torch. All the required packages and required versions can be found in our submitted materials: requirements.txt.

The computation time for probing is 8min23sec.

The 1-epoch training time for each method is as follows. For training time, the methods rank as follows: RestoreLCC < LoRA < DoRA < LoFiT. For GPU memory usage: LoRA < RestoreLCC = LoFiT < DoRA. Compared with the DoRA and LoFiT, RestoreLCC has the lowest overall overhead while providing superior performance.

[Q5] Limitations of addressing MLPs.

[R10] Thank you for raising this concern. Our goal is to restore pruned models in which all weight matrices in both the attention and MLP modules have been pruned. To achieve this, our recovery operations focus on the attention modules rather than the MLPs to restore the overall model performance. We provide both theoretical and empirical analyses to support this choice.

(1) Studies on mechanistic interpretability show that attention heads specialize in distinct functions for different tasks [1,2,3], while FFNs mainly store knowledge and map inputs to outputs [4,5]. The detailed discussion and references can be found in our response R1 to Reviewer Kk22.

(2) Empirical evidence. We would like to highlight that our method can be directly applied to FFNs. We select heads instead of FFNs for recovery also because of the empirical results. Compensating attention achieves 58.83, while compensating FFNs only reaches 57.00.

Thank you again for the insightful comments. We will incorporate the clarifications and discussions into the revised version and we look forward to your feedback.

Dear Reviewer oP8x,

Thank you very much for taking the time to carefully review our rebuttal responses. We truly appreciate your thoughtful feedback. We will ensure that this point, along with the other comments raised in the rebuttal, are clearly addressed and reflected in the revised manuscript.

Best regards,

Authors of Paper 7969

Dear Reviewer n2VC,

Thank you for taking the time to review our paper and for recognizing it as novel, with unique insights that are clearly and extensively justified. The concerns you raised mainly relate to extending our approach to other LLMs, scalability and the portability of the probing module. Following your suggestions, we have added experiments and additional analysis to address these points. We are glad to see that, with your feedback, the paper has been greatly improved. Our responses [R] to your mentioned weaknesses [W] and questions [Q] are summarized below.

[W1] Evaluate on more LLMs from different families.

[R1] Thank you for raising this question. To verify the generalizability of RestoreLCC on more recent LLMs, we evaluated it on the latest Qwen3 family and DeepSeek‑R1‑Qwen3‑8B, both released at the end of May 2025. Our results on Qwen3‑8B, Qwen3‑14B, and DS‑R1‑8B are as follows. Compared with other baseline methods, RestoreLCC consistently delivers significant improvements.

We would also like to highlight an interesting observation on the latest Qwen3 models: the recovered model performs even better than the original dense model. We believe this is because, despite the state‑of‑the‑art performance of Qwen3, the model still benefits substantially from additional tuning with the Alpaca instruction‑tuning dataset.

[W2] The contrastive probing module depends on data/model-specific classifier training, which could affect practical deployment and cross-task portability.

[R2] Thank you for pointing out this concern. We would like to highlight that (1) the overhead of training the classifiers on new models and datasets is negligible, (2) the probing classifier is very easy to implement for new datasets and new models, (3) cross‑task portability can be addressed using the Alpaca dataset when no data is available for a new task, and (4) the benefits of probing far outweigh its cost since it avoids tuning all heads. We agree this is an interesting direction and plan to explore task‑agnostic probing strategies in future work to further improve portability. We further clarify these four points as follows:

(1) Negligible overhead. For each attention head, we train only a simple linear layer as the probing classifier. This process is precomputed before the recovery tuning stage (Section 4.2 LCC). It can be completed even on a low‑end GPU (e.g., NVIDIA RTX 1080), takes only a few minutes, and requires very little GPU memory. Therefore, the overhead is negligible.

(2) Easy implementation. For new datasets and models, the only step needed is to obtain head‑wise activations from the new dataset and model. The code implementation is simple and we provide three different ways: (i) Use Python “hook” functions to capture the inputs/outputs of attention modules directly. (ii) Use the open‑source Pyvene package, which provides convenient APIs to extract activations. (iii) Use the open‑source TransformerLens package, which supports easy activation extraction even for the latest models (e.g., Qwen3). In all three cases, only a few lines of Python code are needed, after which the classifiers can be quickly trained.

(3) Cross‑task portability. When there is no dataset for a new task, the Alpaca instruction‑tuning dataset can be used in a general recovery setting (lines 279‑281). As shown in Table 1, this dataset works well for both language modeling and seven commonsense reasoning tasks, enabling recovery without task‑specific data. Therefore, in such cases, the Alpaca dataset can serve as a universal fallback for probing and tuning across different datasets. This substantially improves the portability of our method.

(4) Benefit outweights cost. Probing allows us to focus on informative heads: in practice, only 10%–30% of heads are restored. This greatly improves efficiency by avoiding unnecessary tuning of all heads. Hence, the efficiency gains from probing far outweigh its small cost.

[W3] A few sections (e.g., lines 38–59) in the main text are densely written, with overlapping figure references, which slightly affects clarity.

[R3] Thank you for pointing this out, and we apologize for the inconvenience caused. We will carefully double‑check and revise all such presentations to ensure clarity and correctness.

[Q1] Scope of Probing Generalization: Given that contrastive probing is data/model/pruning-specific, how scalable is RestoreLCC across multiple downstream tasks without retraining the probing module?

[R4] Thank you for your question. We would like to clarify that the probing module generalizes well across downstream tasks without requiring retraining. This is supported by the following two observations:

(1) Probing on a general dataset, then testing on diverse downstream tasks. Our general recovery setting (lines 279–281) follows this requirement. For the results in Table 1, attention heads are probed using the Alpaca instruction‑tuning dataset, while the evaluation tasks are very different, including a language modeling dataset and seven commonsense reasoning benchmarks. As shown in Table 1, heads probed on general instructions successfully recover performance on diverse tasks such as BoolQ and ARC.

(2) Probing on one specific downstream dataset, then testing on other tasks. To further verify the scalability, we use probing results obtained on the BoolQ dataset and apply them for recovery on other datasets including RTE, ARC‑e, and ARC‑c. The below table shows the results. Even without retraining, the pruned model’s performance is successfully recovered on these tasks, achieving better results than the pruned baseline. It is notable that the results of original pruned model are from Table 3 in our paper.

[Q2] Efficiency at Scale: While parameter overhead is small, does RestoreLCC scale gracefully with larger models (e.g., >30B)? Are there latency/throughput benchmarks in real-world scenarios?

[R5] Thank you for your question.

Firstly, RestoreLCC scales well to larger models. We conducted task‑specific recovery (lines 282–285 in our paper) on the BoolQ dataset using LLaMA‑70B. The results on the below table show that our proposed RestoreLCC continues to outperform the LoRA baseline.

Due to the significant time and GPU resources required for tuning a 70B model, we were only able to complete the current experiment before July 30, 2025. We will continue this work and include additional results in the revised version of the paper.

Secondly, regarding latency and throughput, we measured the inference speed on the BoolQ dataset after recovery. The delay is calculated as the ratio of total inference time with RestoreLCC to that of the original pruned model. This ratio is approximately 1.03, indicating only a ~3% slowdown, which can be ignored. Thus, RestoreLCC maintains nearly the same inference speed as the original pruned model while delivering significantly better performance.

[Q3] Concerns regarding the figure placement and labels.

[R6] Thank you for pointing out these concerns. We will carefully review all figures and make the necessary adjustments to their placement, labels, and formatting to ensure that they are clear and easy to read.

Thank you once again for reviewing our paper and for your insightful comments. We will incorporate all of these discussions into the revised version. Please let us know if you have any further questions, and we look forward to your feedback.

Dear Reviewer n2VC,

Thank you very much for taking the time to carefully review our rebuttal. We truly appreciate your comments, and we will incorporate all these discussions into our revised paper.

Best regards,

Authors of Paper 7969

Dear Reviewer Kk22,

Thank you very much for reviewing our paper and recognizing our method as effective. Your concerns mainly focus on the effect of RestoreLCC on FFNs, its overhead, and clarification of certain details. Following your suggestions, we have added further analysis and additional experiments on these points. We are pleased that, with your feedback, the paper has been significantly improved. Our responses (R) to your weaknesses (W) and questions (Q) are as follows.

[W1] Explain why RestoreLCC uses attention rather than FFN.

[R1] Thank you for this valuable comment. The output of each Transformer block consists of the attention module output (Eq.1 in our paper) and the FFN module output. In our study, we focus on the attention output matrix rather than the FFN output matrix for the following reasons:

(1) Studies on mechanistic interpretability show that attention heads specialize in distinct functions for different tasks, while FFNs mainly store knowledge and map inputs to outputs. For example, [1] finds that in arithmetic tasks, attention heads process key information and FFNs then produce the final answer. They also show that fine‑tuning only important heads outperforms tuning all parameters (including FFNs). Similarly, [2] demonstrates that different heads play distinct roles, and [3] also supports this observation. By contrast, FFNs store factual knowledge [4] and refine output logits [5]. Hence, compensating attention heads is more appropriate than compensating FFNs.

(2) Attention heads provide finer information, whereas FFNs produce high‑dimensional, aggregated outputs. In LLaMA-7B, each head outputs 128 dimensions (32 heads per layer), while an FFN outputs 4096. Smaller head outputs allow finer analysis. Moreover, choosing 32 heads can span multiple layers, while the same size in FFN covers only one layer.

(3) Empirical evidence. We applied RestoredLCC to FFNs and report the best results as follows (More results can be found in below reponse[R6].) Compensating attention achieves 58.83, while compensating FFNs only reaches 57.00.

[1] Interpreting and improving large language models in arithmetic calculation, ICML 2024

[2] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs, ICLR 2025

[3] Dressing up llm: Efficient stylized question-answering via style subspace editing. ICLR 2025

[4] Locating and Editing Factual Associations in GPT, NeurIPS 2022

[5] Transformer Feed-Forward Layers Are Key-Value Memories, EMNLP 2021.

[A2] Training overhead.

[R2] Thank you for the question. We would like to clarify that the overhead of our proposed RestoreLCC is smaller than advanced PEFT methods while achieving much better performance. While recovering pruned LLaMA‑7B (1 H100 GPU, torch.bfloat16 precision, 1 epoch, max_length=512, batch size=8, Alpaca Dataset),the empirical and theoretical evidence are:

(1) Empirical Analysis. The table below compares training time and GPU memory usage. For training time, the methods rank as follows: RestoreLCC < LoRA < DoRA < LoFiT. For GPU memory usage: LoRA < RestoreLCC = LoFiT < DoRA. Compared with the DoRA and LoFiT, RestoreLCC has the lowest overall overhead while providing superior performance.

(2) Theoretical analysis. For RestoreLCC, the main overhead comes from: (i) trainable parameters (the scaling factor β in Eq. 8 and one bias term per head), (ii) intermediate activations (components vᵢ in Eq. 8), and (iii) coding efficiency due to modifications in the attention head output.

(i) RestoreLCC introduces fewer trainable parameters than PEFT methods such as LoRA and DoRA (see Section 4.3 and Appen. C for details)

(ii) The overhead from intermediate activations (vᵢ) is extremely small and can be ignored. The activations are fixed (frozen) during training. Each attention head has 128 dimensions; by the property of SVD this corresponds to at most 128 components, i.e., 128 × 128 = 16,384 parameters per head. Even in the extreme case of using all 1,024 heads, the adds only 1,024 × 16,384 ≈ 16.8M parameters - only about 0.24% of a 7B model. Moreover, as shown in Section 3 and Figure 8, only 10–30% of heads are required for recovery, so the actual overhead is much smaller.

(3) We optimized our code implementation to avoid inefficient loops. Unlike LoFiT, which iterates over each head in thier code, our method uses only matrix multiplications to update all heads. This eliminates costly looping inside the Transformer. Consequently, RestoreLCC trains much faster than LoFiT; re‑implementing LoFiT with our approach similarly reduces its training time, confirming that the efficiency gain comes from our design.

In summary, RestoreLCC introduces smaller overhead than advanced PEFT methods.

The core code implementation is as follows:

# (B:batch_size, T: sequence_length 
# H:head number per layer, D: head_dimension)
out = attn_output.view(B, T, H, D)
# comp: [H, D, D]:decomposed fixed components for heads
# beta [H,D]: beta in Eq.8
# b [H,D]: bias in Eq.8
added_info = (
    torch.bmm(comp.transpose(1, 2),  # (H, D, D)
              beta.unsqueeze(2))  # (H, D, 1)
    .squeeze(2)
)
b = b.view(1, 1, H, D)
add = added_info.view(1, 1, H, D)
# modify every head in one go
attn_output = out  + add + b
attn_output = attn_output.view(B, T, H*D)

Notably, only a subset of attention heads use non‑zero values for comp (frozen, no gradients), beta (trainable), and b (trainable). GPUs will automatically optimize both the storage and the arithmetic for these zero‑valued vectors.

[W3] Overhead sensitivity to number of parameters.

[R3] We would like to state that our method is not sensitive to the number of trainable parameters.

(1) First, as in our response [R2] to your [W2], RestoreLCC introduces less overhead than other advanced PEFT methods.

(2) Empirical results. The number of trainable parameters in RestoreLCC depends on the proportion of attention heads selected for tuning. The table below reports GPU memory and training time for different ratios of heads. The results show: (i) The GPU memory consumption remains almost unchanged as trainable parameters increase and constantly lower than DoRA (71 GB). (ii) The training time is mostly smaller than DoRA and LoFiT.

(3) We would like to highlight that more parameters do not necessarily lead to better performance and thus do not introduce sensitivity. As shown in Section 3, not all attention heads are equally useful for recovery. Also, figure 8 demonstrates that using only 10%–30% of the most discriminative heads is sufficient to achieve the best performance. This is also why our method first contrastively probes important heads.

[Q1] Results explanation on Figure 1.

[R4] Thank you for raising this point. The improvement are for two reasons:

(1) Task‑specific information. The added components act as task‑specific vectors (often referred to as task function vectors [6] or task steering vectors [7]). Consistent with these prior studies, such vectors provide task‑aware guidance and significantly enhance the model’s performance over the original model.

(2) Removal of useless or harmful components.Some lost components in each attention head can even be harmful, as they may introduce noise that negatively affects the model’s output. By recovering only the useful components and excluding the harmful ones, the model’s performance improves.

[6] Function Vectors in Large Language Models, ICLR 2024

[7] Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors, ICLR 2025

[Q2] Comparison with full-parameter tuning.

[R5] We conduct experiments on LLaMA‑7B. Code implementation is from Wanda. The average results are as follows. On Wanda‑pruned models, RestoreLCC achieves slightly better performance than FT. On SparseGPT‑pruned models, RestoreLCC is slightly worse than FT. Overall, RestoreLCC achieves comparable to FT.

[Q3] Effects of using FFNs.

[R6] We apply RestoreLCC to FFNs and evaluated different numbers of FFN layers. The best result obtained was 57.00, which is lower than the 58.83 achieved when using attention heads.

[Q4] Details for PEFT implementations.

[R7] We apologize for not providing these details. For LoRA and DoRA, we use the same settings: α = 16 and rank = 8. Regarding the applied modules, we try two configurations: (1) ["v_proj", "o_proj"], which tunes only the head output matrices; and (2)["q_proj", "k_proj", "v_proj", "o_proj"]`, which tunes all head matrices. The best results are reported in the paper.

For LoFiT, we experiment with 10%, 20%, and 30% of the heads and report the best results.

The batch size is set to 8. The learning rate is 1e-4. The max sequence length is 512. The number of training epochs is 2 for global recovery and 5 for task-specific recovery. All experiments are conducted on a single H100 GPU.

Thank you again for taking the time to review our paper. We will incorporate these discussions into the revised version and look forward to your feedback.