LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions

We are grateful to the reviewer for their detailed feedback of our work, and are appreciative of their time, efforts, and consideration. Below we provide more discussion on the points raised.

• Partially data-driven analysis methods:

  We thank the reviewer for raising this important clarification. We agree that activation-based methods such as those proposed in [20–23] do typically require access to training (or validation) data to compute layer-wise activations, and therefore are not entirely “model-only” in the strictest sense.

  Our original phrasing in Lines 32–33 aimed to distinguish methods that rely primarily on internal model representations (e.g., weight norms, activations, attention maps) from those that explicitly quantify model–data interactions, such as Influence Functions. While activation-based techniques indeed use input data to probe activations, they do not trace the causal effect of individual training examples on model predictions by analyzing the loss. In contrast, Influence Functions estimate the impact on the loss of perturbing or removing training samples.

  We will revise our language in the introduction to clarify this distinction and cite other papers that are more absolutely model-centric. A more accurate phrasing would be:

  “Many approaches take a model-centric perspective, examining how the model’s learned weights effect predictions [1,2]. Other methods use activations, or attention patterns to deduce internal mechanisms [3,4,5,6]. In contrast, Influence Functions offer a more explicitly data-centric perspective by quantifying the effect of individual training points on model's loss landscape.”

  We appreciate the reviewer’s suggestion and will make this clarification in the final version.

• Clarifying Figure 1:

  Figure 1 is just a hypothetical illustration (not with real numbers) of our method against model only or heuristic based methods. We will add this clarification in the caption.

  To improve the clarity of our method we will aim to provide a concrete step-by-step walk-through, in the revised version of the paper, as described below:

  Concrete walk‑through for expert allocation (CommonQ dataset, 32‑layer LLM, T=160 experts, beta=3)

  Below we provide the exact numbers produced by our implementation. For brevity only the first 5 layers are shown; the remaining layers follow identically.

  Layer $i$	Raw IF score $v_i$	Inverted $ṽ_i$	Power‑scaled $v̂_i$	Fractional $f_i$	Final experts $s_i$
  0	-5.01×10¹¹	5.01×10¹¹	1.26×10³⁵	1.12	2
  1	-8.36×10¹¹	8.36×10¹¹	5.84×10³⁵	5.21	6
  2	-7.99×10¹¹	7.99×10¹¹	5.10×10³⁵	4.55	6
  3	-8.46×10¹¹	8.46×10¹¹	6.05×10³⁵	5.40	6
  4	-8.16×10¹¹	8.16×10¹¹	5.43×10³⁵	4.84	6
  …	…	…	…	…	…

  (1) Step1 (sign inversion). Invert the negative values to positive.

  (2) Step2 (power transform). Amplify disparities by raising to the power beta to get v̂.

  (3) Step3 (normalise to budget). Scale v̂ so that the fractional allocations ∑fi equal T−m=128 extra experts (m=32 layers).

  (4) Step4 (floor+redistribute). Set si=⌊fi⌋+1, guaranteeing ≥1 expert/layer, then distribute the remaining 24 units to the layers with the largest fractional parts.

  Finally, the complete allocation for CommonQ becomes:

  [2, 6, 6, 6, 6, 5, 7, 6, 8, 7, 6, 7, 6, 6, 6, 5, 6, 6, 5, 4, 5, 5, 4, 2, 4, 5, 3, 4, 2, 3, 4, 3]

  Interpretation. The middle layers receive the most experts, indicating they are comparatively under‑trained for CommonQ; the first and very deep layers receive lower number of experts, reflecting higher estimated quality.

  Similarly we will also provide an example for the layer-wise sparsity allocation setting. If the reviewer would like us to take any other steps to improve clarity, we are more than happy to do so.

• Reverse experiment:

  We thank the reviewer for this suggestion. We provide an ablation where we reversed our methodology by pruning the less sensitive layers more at a 70% sparsity budget on Mistral-7b-v0.1 and find that our design choice leads to improved performance over reversed allocation. We present the results below.

  Method	Magnitude	Wanda	SparseGPT	Average
  Our Method	33.439	32.497	41.143	35.693
  Reversed Allocation	33.101	32.015	38.450	34.522

  We will add this ablation to the revised version of our paper.

• LayerIF having algorithmic inconsistency:

  We would like to clarify the generality of our approach. Influence Functions (IFs), by design, are diagnostic tools intended to inform model developers about the sensitivity of model components to training data. LayerIF extends this diagnostic capability to estimate per-layer quality in a principled, data-driven manner.

  While it is true that expert allocation and pruning use distinct post-processing steps after the initial LayerIF score computation, this is by design: each downstream application naturally requires a different mapping from layer quality estimates to actionable decisions (e.g., allocating experts vs. assigning sparsity levels). However, the core LayerIF signal remains the same and is easily integrated across use case.

  Importantly, LayerIF is not specific to the two tasks studied in this paper. It is a general-purpose tool that can be applied in the future to any task where layer quality matters (e.g. layer-wise unlearning). Its ability to improve performance across diverse tasks and compete against methods that are tailor to those specific tasks, in our view, is a strength rather than a limitation.

• Clarification on numbers:

  All these numbers follow the relative change formula of (New-Old)/Old * 100. We provide the calculations below:

  • 1.61: IF (Top 25%) and MoLA-8642. (82.27-80.97)/80.97 * 100.
  • 3.54: Average of IF variants and Average of baselines. (81.96-79.175)/79.175 * 100. This is a typo the correct value is 3.52. We will change this.
  • 1.03: IF SparseGPT and OWL SparseGPT. (60.61-59.99)/59.99 * 100.
  • 0.90: IF SparseGPT and AlphaPruning SparseGPT. (60.61-60.07)/60.07 * 100.

  We hope this clears the confusion on how we arrived on these numbers.

• Improvement appears marginal:

  We appreciate the reviewer’s concern and would like to offer some clarification. While the absolute gains over the uniform sparsity baseline in the pruning experiments may appear numerically modest (under 0.5%), this level of improvement is consistent with recent state-of-the-art work that we compare against OWL (ICML 2024) and AlphaPruning (NeurIPS 2024).

  Moreover, our method is not designed exclusively for pruning. Model pruning was included as one illustrative downstream task to demonstrate the applicability of LayerIF, not as the core objective of the paper. LayerIF is a general-purpose diagnostic tool, and its effectiveness across multiple settings speaks to its broader utility beyond the pruning context.

  Thus, we believe that providing a unified, data-driven method that performs competitively across such diverse applications is a notable contribution.

We thank the reviewer for their comments and we hope that we provided sufficient clarifications. We hope to discuss further in the discussion period in case any others remain. Thank you.

References:

[1]. Yin, Lu, et al. "Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity." arXiv preprint arXiv:2310.05175 (2023).

[2]. Lu, Haiquan, et al. "Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models." Advances in neural information processing systems 37 (2024): 9117-9152.

[3]. Serrano, Sofia, and Noah A. Smith. "Is attention interpretable?." arXiv preprint arXiv:1906.03731 (2019).

[4]. Abnar, Samira, and Willem Zuidema. "Quantifying attention flow in transformers." arXiv preprint arXiv:2005.00928 (2020).

[5]. Wiegreffe, Sarah, and Yuval Pinter. "Attention is not not explanation." arXiv preprint arXiv:1908.04626 (2019).

[6]. Hao, Yaru, et al. "Self-attention attribution: Interpreting information interactions inside transformer." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 14. 2021.

The authors thank the reviewer for engaging thoughtfully with our rebuttal. Below, we address the specific points raised:

• Figure 1: We acknowledge the reviewer’s concerns and will remove all hypothetical numbers from the figure in the revised version. This change is straightforward to implement.

• Relative Improvement: We apologize for the confusion. Our use of relative improvement metrics was guided by conventions in the area, where such reporting is common—for example, see Section 4.2 in [1] and Figure 30 in [2]. That said, we are happy to include the corresponding absolute values in the text to improve clarity.

• Comparison to OWL: We also observed that OWL underperforms relative to the baselines in certain settings; however, we note that OWL was the second-best performing configuration at three out of the four sparsity levels reported in Figure 3. This trend is even more pronounced in our Gemma‑7B results (included in the supplementary and reproduced below). We attribute these differences to variations in model architecture, though a deeper investigation is warranted and could be pursued in future work.

We thank the reviewer for engaging, hopefully our further clarifications warrant an increase in score.

References:

[1]. Abhineet Agarwal, Yan Shuo Tan, Omer Ronen, Chandan Singh, Bin Yu Proceedings of the 39th International Conference on Machine Learning, PMLR 162:111-135, 2022.

[2]. Grosse, Roger, et al. "Studying large language model generalization with influence functions." arXiv preprint arXiv:2308.03296 (2023).

Thank the authors for the follow-up responses. Some of my concerns are addressed. Just on point: in your response, the improvement compared to AlphaPruning seems margnial. However, based on your your detailed and prompt reponses in the whole rebuttal period, I will increase my scores accordingly.

We are grateful to the reviewer for their detailed feedback of our work, and are appreciative of their time, efforts, and consideration. Below we provide more discussion on the points raised.

• Inconsistency in formulation:

  Thank you for pointing out this typo, we appreciate it. We sincerely apologize for the missing (leading) '-' sign in the formulation, and will easily make this correction in the paper revision. However, after adding the - sign, the rest of the paper can remain unchanged and no other major changes are required.

  We would also like to mention that the typo in the formulation does not change the validity of our experimental design or the results of our experiments, and we apologize for the inconsistency once again.

• Lack of Justification for Score Aggregation:

  We thank the reviewer for raising this point. Our choice to aggregate only positive influence values was guided primarily by empirical evidence rather than theoretical assumptions. To validate this design choice we conducted this ablation in Section 4.1 where the Top 25% variant was the highest performing. Additionally, we perform another ablation in the rebuttal period where we compare keeping all of the samples against keeping only the positive samples on the Gemma-7b model.

  Dataset	IF (+ve)	IF (ALL)
  MRPC	84.75	82.49
  CoLA	86.96	85.14
  Common_Q	83.46	82.06
  Openbook_Q	88.20	86.20
  Text_Science_Q	93.66	94.56
  Average	87.81	86.09

  Here, we see that IF (+ve) was significantly better performing on average and individually more performant on 4 out of the 5 datasets.

  We acknowledge that this does not preclude the possibility of more nuanced aggregation schemes. Future work could explore adaptive thresholds or weighting strategies that combine both helpful and harmful samples in a principled way. We will expand our discussion to acknowledge this broader design space in the revised version of this paper.

• Lack of Clarity in MoE Allocation Method:

  We apologize for the lack of clarity. The $v_i$ is indeed corresponding the the individual layer scores in $S^{(l)}$. Since our implementation followed the DataInf convention as described above, the layerwise values in $S^{(l)}$ were all negative, hence the inversion and then power transformation did not lead to complex values. We will revise this and make things consistent in the paper.

• Computational feasibility for DataInf and TracIn under varying datasets and model sizes:

  We thank the reviewer for raising the important point regarding the computational times of Influence Function methods. Below, we provide wall-clock timings for computing the influence function for a single layer across varying training and validation set sizes. We used 4× NVIDIA RTX 6000 Ada Generation GPUs to perform these experiments.

  Scalability to different Dataset Sizes	Method	Runtime (seconds)
  Mistral-7B-v0.1, 3000×500	DataInf	163,721
  	TracIn	0.0135
  Mistral-7B-v0.1, 300×50	DataInf	402.77
  	TracIn	3.314×10^-5
  Mistral-7B-v0.1, 30×5	DataInf	15.3

  As we can see the time increases non-linearly with larger datasets, but we intend to clarify that in our experiments, having approximately 300-500 training samples and 50-100 validation samples is typically sufficient to gauge how well an LLM layer generalizes to a task. Moreover, the current implementation of DataInf runs primarily on CPU; preliminary tests show that offloading key computations to the GPU yields up to a 2× speedup, indicating substantial room for further efficiency improvements.

  Similarly, we report the runtime for computing the influence function for 1 layer for different model sizes.

  Scalability to different Model Sizes	Method	Runtime (seconds)
  	TracIn	1.192×10^-6
  Qwen2.5‑3B, 300×50	DataInf	58.063
  	TracIn	2.789×10^-5
  Qwen2.5‑32B, 300×50	DataInf	2,533.46
  	TracIn	5.245×10^-6

  As expected, larger models incur higher influence computation times; however, these costs are negligible relative to the overall expense of pretraining large language models. For context, training LLaMA2‑70B required approximately 1.72 million GPU hours. By comparison, our method represents a minor overhead and can be integrated into the broader pretraining pipeline to diagnose underperforming layers and nudge the model toward more uniform training dynamics. This can support developers in identifying and addressing layer‑specific inefficiencies during model development.

  Furthermore, the peak GPU memory usage during influence computation for a dataset with 300 training and 50 validation samples was approximately 35,000 MiB for a 7B parameter model. This footprint is relatively modest given the model size and suggests that influence-based diagnostics can be feasibly incorporated into GPU-based training pipelines without exceeding typical memory constraints.

  Finally, for error analysis, we defer to the approximation error bounds provided in [1], as we directly employ their DataInf method to compute influence scores. To further validate the reliability of these scores, we compare them against alternative estimators, notably TracIn, and demonstrate its limitations as a proxy for layer quality in our setting (possibly owing to the Hessian being discarded).

• Minor Issues: Thank you for pointing these out, and apologies for missing these prior to submission. We shall fix these small errors and typos.

Thank you once again for all your efforts in helping to strengthen our work. We hope that our comments helped address your concerns and look forward to engaging with you more during the author-reviewer discussion phase regarding any others that remain.

References:

[1]. Kwon, Yongchan, et al. "Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models." ICLR 2024.

We are grateful to the reviewer for their detailed feedback of our work, and are appreciative of their time, efforts, and consideration. Below we provide more discussion on some of the points raised.

• Explicit Walk-through of the algorithm:

  We thank the reviewer for their suggestion and will improve the clarity of the methods (as well as design choices) as requested. We will aim to provide a concrete step-by-step walk-through of our method, in the revised version of the paper, as described below:

  Concrete walk‑through for expert allocation (CommonQ dataset, 32‑layer LLM, T=160 experts, beta=3)

  Below we provide the exact numbers produced by our implementation. For brevity only the first 5 layers are shown; the remaining layers follow identically.

  Layer $i$	Raw IF score $v_i$	Inverted $ṽ_i$	Power‑scaled $v̂_i$	Fractional $f_i$	Final experts $s_i$
  0	-5.01×10¹¹	5.01×10¹¹	1.26×10³⁵	1.12	2
  1	-8.36×10¹¹	8.36×10¹¹	5.84×10³⁵	5.21	6
  2	-7.99×10¹¹	7.99×10¹¹	5.10×10³⁵	4.55	6
  3	-8.46×10¹¹	8.46×10¹¹	6.05×10³⁵	5.40	6
  4	-8.16×10¹¹	8.16×10¹¹	5.43×10³⁵	4.84	6
  …	…	…	…	…	…

  (1) Step1 (sign inversion). Invert the negative values to positive.

  (2) Step2 (power transform). Amplify disparities by raising to the power beta to get v̂.

  (3) Step3 (normalise to budget). Scale v̂ so that the fractional allocations ∑fi equal T−m=128 extra experts (m=32 layers).

  (4) Step4 (floor+redistribute). Set si=⌊fi⌋+1, guaranteeing ≥1 expert/layer, then distribute the remaining 24 units to the layers with the largest fractional parts.

  Finally, the complete allocation for CommonQ becomes:

  [2, 6, 6, 6, 6, 5, 7, 6, 8, 7, 6, 7, 6, 6, 6, 5, 6, 6, 5, 4, 5, 5, 4, 2, 4, 5, 3, 4, 2, 3, 4, 3]

  Interpretation. The middle layers receive the most experts, indicating they are comparatively under‑trained for CommonQ; the first and very deep layers receive lower number of experts, reflecting higher estimated quality.

  Similarly we will also provide an example for the layer-wise sparsity allocation setting. If the reviewer would like us to take any other steps to improve clarity, we are more than happy to do so.

• Why are the raw scores noisy between adjacent layers?:

  We thank the Reviewer for this important question. Influence‑function estimates at adjacent layers are inevitably noisy: they are computed from stochastic Hessian approximations and mini‑batch gradients, and thus contain high‑frequency fluctuations that can misrepresent the true differences in layer quality. Grosse et al. [1] visualize this variability directly: the layer × token heatmap in Figure 4 (p.20) reveals sharp red/teal oscillations across adjacent layers, reflecting high-frequency fluctuations in estimated influence. The authors also note in Section 2.1 that influence values are extremely heavy-tailed, which complicates traditional statistical analysis and motivates the need for robust aggregation strategies.

  Hence, we apply a Savitzky–Golay smoother (window = 5, poly‑order = 2) to: a) Suppress spurious high‑frequency noise, while b) Preserving low‑frequency structure, the monotonic rise–fall pattern that truly indicates which layers of the LLM are under or over‑trained.

  We will add a sentence clarifying this rationale in Section 3.4 accordingly.

• Wall Clock times and Memory Consumption for DataInf and TracIn under varying datasets and model sizes:

  We thank the reviewer for raising the important point regarding the computational times of Influence Function methods. Below, we provide wall-clock timings for computing the influence function for a single layer across varying training and validation set sizes. We used 4× NVIDIA RTX 6000 Ada Generation GPUs to perform these experiments.

  Scalability to different Dataset Sizes	Method	Runtime (seconds)
  Mistral-7B-v0.1, 3000×500	DataInf	163,721
  	TracIn	0.0135
  Mistral-7B-v0.1, 300×50	DataInf	402.77
  	TracIn	3.314×10^-5
  Mistral-7B-v0.1, 30×5	DataInf	15.3

  As we can see the time increases non-linearly with larger datasets, but we intend to clarify that in our experiments, having approximately 300-500 training samples and 50-100 validation samples is typically sufficient to gauge how well an LLM layer generalizes to a task. Moreover, the current implementation of DataInf runs primarily on CPU; preliminary tests show that offloading key computations to the GPU yields up to a 2× speedup, indicating substantial room for further efficiency improvements.

  Similarly, we report the runtime for computing the influence function for 1 layer for different model sizes.

  Scalability to different Model Sizes	Method	Runtime (seconds)
  	TracIn	1.192×10^-6
  Qwen2.5‑3B, 300×50	DataInf	58.063
  	TracIn	2.789×10^-5
  Qwen2.5‑32B, 300×50	DataInf	2,533.46
  	TracIn	5.245×10^-6

  As expected, larger models incur higher influence computation times; however, these costs are negligible relative to the overall expense of pretraining large language models. For context, training LLaMA2‑70B required approximately 1.72 million GPU hours. By comparison, our method represents a minor overhead and can be integrated into the broader pretraining pipeline to diagnose underperforming layers and nudge the model toward more uniform training dynamics. This can support developers in identifying and addressing layer‑specific inefficiencies during model development.

  Finally, the peak GPU memory usage during influence computation for a dataset with 300 training and 50 validation samples was approximately 35,000 MiB for a 7B parameter model. This footprint is relatively modest given the model size and suggests that influence-based diagnostics can be feasibly incorporated into GPU-based training pipelines without exceeding typical memory constraints.

• Gauss-Newton Approximation's applicability:

  The main bottleneck with the Gauss-Newton Hessian is the time required for IF computation. While doing initial experiments, we considered utilizing the Guass-Newton Hessian instead of the DataInf approximation. However, even the time taken to compute the Gauss-Newton Hessian IF for 30 Train and 5 Validation samples was 2494.72 seconds for a 7B model whereas the equivalent experiment with the DataInf IF approximation took only 15.3 seconds. Hence, it made it unfeasable for our experiments, which required 300 Train and 50 Validation samples.

• Applicability VLA settings:

  Thank you. We appreciate the reviewer’s suggestion and agree that multimodal mixtures‑of‑experts (MoEs) and particularly VLAs represent an exciting next step. There is a conceptual fit as our influence‑based layer‑quality metric is modality‑agnostic: it only requires gradients and Hessian–vector products for the parameters under consideration. In principle, we can therefore:

  1. Compute per‑expert influence for vision, language, and cross‑modal experts separately;
  2. Compare their sensitivity profiles on joint vision–language objectives (e.g., image‑text contrastive loss, VQA loss);
  3. Adapt the routing policy according to these influence scores.

  We have actually begun conducting some preliminary research in this very exciting area as part of follow-up work, and will be releasing it in the near future for the community.

Thank you once again for all your efforts in helping to strengthen our work. We hope that our comments helped address your concerns and if so, would be extremely obliged if you could consider increasing your score.

References:

[1] Grosse, Roger, et al. "Studying large language model generalization with influence functions." arXiv preprint arXiv:2308.03296 (2023).

For the sake of completeness, we conducted an additional experiment to assess the effectiveness of the Gauss-Newton Hessian on the Text_Science_Q dataset using the Mistral-7B-v0.1 model. Due to time constraints, layer-wise influence scores were computed on a subset of 30 training and 5 validation samples. The resulting model achieved an accuracy of 72.57%, which is substantially lower than that of the DataInf-based methods.

We are grateful to the reviewer for their feedback of our work, and are appreciative of their time, effort, and consideration. Below we provide more discussion on the points raised.

• Technical Presentation/Clarity:

  We thank the reviewer for their suggestion and will improve the clarity of Section 3.3 and Section 3.4 as requested. We will aim to provide a concrete step-by-step walk-through of our method, in the revised version of the paper, as described below:

  Concrete walk‑through for expert allocation (CommonQ dataset, 32‑layer LLM, T=160 experts, beta=3)

  Below we provide the exact numbers produced by our implementation. For brevity only the first 5 layers are shown; the remaining layers follow identically.

  Layer $i$	Raw IF score $v_i$	Inverted $ṽ_i$	Power‑scaled $v̂_i$	Fractional $f_i$	Final experts $s_i$
  0	-5.01×10¹¹	5.01×10¹¹	1.26×10³⁵	1.12	2
  1	-8.36×10¹¹	8.36×10¹¹	5.84×10³⁵	5.21	6
  2	-7.99×10¹¹	7.99×10¹¹	5.10×10³⁵	4.55	6
  3	-8.46×10¹¹	8.46×10¹¹	6.05×10³⁵	5.40	6
  4	-8.16×10¹¹	8.16×10¹¹	5.43×10³⁵	4.84	6
  …	…	…	…	…	…

  (1) Step1 (sign inversion). Invert the negative values to positive.

  (2) Step2 (power transform). Amplify disparities by raising to the power beta to get v̂.

  (3) Step3 (normalise to budget). Scale v̂ so that the fractional allocations ∑fi equal T−m=128 extra experts (m=32 layers).

  (4) Step4 (floor+redistribute). Set si=⌊fi⌋+1, guaranteeing ≥1 expert/layer, then distribute the remaining 24 units to the layers with the largest fractional parts.

  Finally, the complete allocation for CommonQ becomes:

  [2, 6, 6, 6, 6, 5, 7, 6, 8, 7, 6, 7, 6, 6, 6, 5, 6, 6, 5, 4, 5, 5, 4, 2, 4, 5, 3, 4, 2, 3, 4, 3]

  Interpretation. The middle layers receive the most experts, indicating they are comparatively under‑trained for CommonQ; the first and very deep layers receive lower number of experts, reflecting higher estimated quality.

  Similarly we will also provide an example for the layer-wise sparsity allocation setting. If the reviewer would like us to take any other steps to improve clarity, we are more than happy to do so.

• The method description is inconsistent in Section 3.3:

  We appreciate the reviewer’s attention to this apparent inconsistency and apologize for the confusion. We intend to clarify that Section 3.3 describes the default pipeline used throughout the paper. Specifically, we aggregate only positive‑influence samples (+VE) and discard negative ones.

  In Section 4.1, we include two additional variants, ALL, which retains every sample, and TOP 25%, which selects the top quartile of positive IF scores by magnitude, as part of a controlled ablation study. These ablations were included at this stage because including them earlier would have introduced unnecessary complexity and we wanted to empirically test our design choices.

  These variants were included solely to assess robustness and justify our aggregation choice; they were not used in the main experimental pipeline, as reflected in Figures 2 and 3 and Table 2.

  Additionally, we perform another ablation in the rebuttal period where we compare keeping all of the samples against keeping only the positive samples on the Gemma-7b model.

  Dataset	IF (+ve)	IF (ALL)
  MRPC	84.75	82.49
  CoLA	86.96	85.14
  Common_Q	83.46	82.06
  Openbook_Q	88.20	86.20
  Text_Science_Q	93.66	94.56
  Average	87.81	86.09

        Here, we see that IF (+ve) was significantly better performing on average and individually more performant on 4 out of the 5 datasets.

• A principled guideline for choosing the variants or an automatic tuning strategy is missing:

  We thank the reviewer for highlighting this point. We would like to point that while Table 1 shows some variation across tasks as the reviewer mentioned e.g., +VE performs best on OpenbookQA, and TOP‑25% on MRPC and Text Science Q, the overall trend strongly favors the IF based methods, particularly TOP‑25%. Specifically, TOP‑25% achieves:

  • the highest macro-average performance (82.27%) across all tasks,
  • has the lowest standard deviation 1.84.

  This robustness makes TOP‑25% a strong default in practice. The use of a fixed, global aggregation threshold is also consistent with precedent in prior influence function research. For instance, in [1], as evident in Table 5 and Table 6, aggregate performance trends across tasks are generally more informative than per-dataset maxima. This is also evident in our results.

  However, we agree with the reviewer that dataset-specific optimal thresholding might yield further gains, and will definitely aim to study that exploration more in follow up work.

• Removing an important layer should increase loss significantly:

  We believe there may be a misunderstanding and would like to clarify our intended meaning. When we state in the Introduction that "IFs measure the sensitivity of model loss to individual training points and the loss of well-trained layers should show smaller changes," we are emphasizing that well-trained layers tend to be more stable i.e., they exhibit lower sensitivity to perturbations in the training data. This interpretation is further elaborated in the “Justification” paragraph of Section 3.2.

  We fully agree with the reviewer that removing important layers would significantly degrade model performance and increase loss. Our empirical results support this: the layers identified by our method as less sensitive (and thus likely better trained) receive fewer LoRA experts and are pruned more conservatively, leading to stronger downstream performance.

  To further emphasize this point, we conduct an ablation where we reversed our methodology by pruning the less sensitive layers more at a 70% sparsity budget on Mistral-7b-v0.1. We present the results below.

  Method	Magnitude	Wanda	SparseGPT	Average
  Our Method	33.439	32.497	41.143	35.693
  Reversed Allocation	33.101	32.015	38.450	34.522

        We will add this ablation to the revised version of our paper.

Thank you again for taking the time to review our paper. We hope that our responses alleviated the concerns raised, and would be happy to engage further during the discussion phase in case there are any others remaining.

References:

[1] Chhabra, Anshuman, et al. "Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models." ICML 2025.

Thank you for your response. I will consider increasing my score. (It seems that I can't temporarily modify the score at this stage; Once I modify, it would be my final score.)

Though the authors provide empirical evidence that TOP‑25% is strongest, why selecting the top quartile of positive IF scores is better than +VE and ALL still needs more justification. In principle, why does selecting partial positive IF help? And how can we determine an appropriate ratio?