Figure Modification

Comment (1): The first two subfigures in the introductory figure convey similar concepts and could potentially be merged to reduce redundancy and improve visual conciseness.

Response (1): We thank the reviewer for his helpful comment. In the revised version of the paper we will combine and merge the two sub-figures.

Analysis Distribution and Quantity of Skipped Blocks

Comment (2): It would be helpful to include an ablation study analyzing the distribution and quantity of skipped blocks. Specifically, showing how many layers are skipped at once and how this distribution affects performance would clarify how the redundancy lies.

Response (2): Below, we report experimental results for a fixed setting of eight pruned layers to illustrate performance variation similar to what we did in section A.10 of the appendix. The following table shows how average accuracy and cosine distance vary when fixing the number of pruned layers at 8.

Additionally, the following table details the relationship between different number of pruned layers, their distributions in the original network and their corresponding average accuracy:

We hope these results bring more clarity and successfully address the reviewer's concerns. We will add this table in the Appendix.

Additional Parameters

Comment (3): I am curious about the size of the added linear transformations used for error compensation. It would be useful to clarify whether the number of additional parameters introduced by these layers has a significant impact on overall performance or model size. A discussion or ablation on this aspect could enhance the understanding of the method’s trade-offs.

Response (3): We thank the reviewer for his comment. The outputs of the MLP in the $i$-th block and of the $(i+n+1)$-th block, both have shape $B \times L \times M$, where $B$ is the batch size, $L$ the sequence length, and $M$ the hidden dimension. The linear transformation that maps the MLP output of block $i$ to the output of block $i+n+1$ is therefore an $M \times M$ weight matrix. As shown in Section 2.2, this transformation can be fused into the MLP’s down‑projection layer, since they are two consecutive linear operations. Based on this observation our method introduces no additional parameters. In the revised version of the paper we will try to make this aspect clearer, so that to avoid any confusion to the readers.

Lack of Novelty

Comment (1): This method lacks novelty since fundamental motivation and pruning technique (cosine distance) are proposed in previous work UIDL.

Response (1): We thank the reviewer for his comment but we respectfully disagree with his assessment. The problem of model compression is a very important one with numerous practical applications and has gathered a wide research attention since several years ago. There are many different strategies that have been proposed including pruning methods (structured, unstructured, depth-pruning), and thus we don't believe that additional motivation for investigating this problem is needed. We agree with the reviewer that the cosine distance that we use for identifying the blocks to be removed was previously expored in UIDL and ShortGPT, but we never claimed it to be part of our contribution. Our main contribution is the replacement of a sequence of layers with a linear transformation and the computation of the specific form of this transformation using calibration data. More importantly, our approach achieves model compression without a significant drop in performance while requiring a fraction of the computational cost of competing strategies, as it eliminates the need for the expensive healing process present in prior methods. Furthermore, we have conducted a detailed analysis of linear operators that feature specific properties and assessed their impact on the approximation accuracy of the original model. Finally we have performed several comparisons against competitive state-of-the-art methods both in NLP and Vision tasks and we have shown that our approach compares favorably in all cases.

Higher Compression

Comment (2): The effectiveness is only on a low/moderate compression ratio 25%.

Response (2): The reviewer is correct that our main focus was on 25% compression ratio. The main reason for this is that with higher compression rates the compressed LLM models' performance typically drops below ~90% of the original performance. In such cases the resulting compressed model doesn't have much of practical value. Moreover, we believe that 25% is a significant pruning ratio, especially when we consider the modern LLMs who employ billions of parameters. In these cases the number of removed parameters is in the order of hundred of millions or even billions. Nevertheless, we agree with the reviewer that evaluating a wider range of compression ratios can still be insightful. In the revised version of the paper, we will include comparisons with UIDL without healing at compression ratios up to 50%. These results are also provided below, showing the superiority of our approach even for larger compression ratios.

In this Table we report the average performace accuracy for (Race, Winogrande, PIQA, BoolQ, OpenBookQA, and SciQ benchmarks), and the perplexity on wikitext. for Llama 3 8B instruct model.

CKA, CCA Ablations

Comment (3): Could the authors provide more ablation studies with other similarity measures like CKA, CCA?**

Response (3): We agree with the reviewer that the choice of distance metric is important, and we thank him for this valuable suggestion. In response, we have conducted additional experiments using CCA and CKA distances, as presented below. These are similar to those reported in Section A.10, where we record the model's average accuracy when we prune a fixed number of layers (eight in this case) but at diffferent locations of the original model, Llama 3 8B Instruct. The last three columns of the table contain the respective values for the cosine, CCA and CKA distances.

As we can see from the results, CCA and CKA show similar patterns to cosine distance. Their values are also high in the first layers while their minima are closer to the end of the model. However, cosine distance continues to yield superior performance: the highest accuracy is observed at the lowest cosine distance, while the accuracies corresponding to the lowest CKA and CCA distances are lower. These findings provide additional evidence that cosine distance is a reliable criterion for layer removal. We will include these results in the final version of the paper.

LTs VS LoRA

Comment (4): The linear transformation is just like a feature alignment method, does it have the same role as adapters? Could we replace the linear transformation matrix with PEFT methods like Adapter or LoRA?

Response (4): We thank the reviewer for raising this important point about the relationship between our linear transformation (LT) and parameter‑efficient fine‑tuning (PEFT) adapters. Whether one labels LT a “PEFT adapter” it ultimately depends on how one defines an adapter. In LLM Streamline, for example, pruned layers are replaced by a small MLP, what one would traditionally call an adapter, or even an entire Transformer block that is first initialized via an MSE loss and then fine‑tuned on next‑token prediction. Likewise, UIDL incorporates LoRA modules during its healing phase.

Our LT could, in theory, be viewed through the same lens: it is a feature‑alignment layer that sits where a removed block once was. However, it differs fundamentally in both initialization and deployment. We compute the LT in closed form or numerically on a small calibration dataset, without backpropagating through the entire model. Once LT is calibrated, it is merged into the previous layer’s down‑projection and introduces no extra parameters at inference. Although we do explore a subsequent healing step (see Section A.14). Perhaps most compellingly, even without any healing, our method outperforms the more complex adapters in LLM Streamline and the LoRA‑based modules in UIDL, as shown in Table 1 in the paper. We fully acknowledge that, under very aggressive compression ratios, a simple linear map may not possess sufficient expressive power to capture all dependencies. In such extreme settings, LoRA or deep non-linear adapters might achieve a better approximation. Yet those scenarios typically incur catastrophic quality drops and require prolonged healing, which runs counter to our goal of minimal overhead and predictable, one‑shot calibration.

We hope this clarifies the conceptual distinction and practical benefits of our LT approach compared to existing PEFT adapters. We would be happy to elaborate more on this if the reviewer requires more details.

Analysis of the Pruned Layers

Comment (5): Which layers/blocks are often pruned? Could the authors provide more analysis on this?

Response (5): In the table included in response (3) above we report accuracy and distance metrics for all combinations of eight removed layers.

In general, the most redundant layers lie in the middle and closer to the last blocks of the network, although the exact pattern varies by model. E.g., for Llama 3 8B, skipping layers 20–28 (1‑indexed) out of 32 yields the best results. We provide additional information in the following table:

It is also important to note that the optimal pruning range depends on both the architecture and the number of layers removed. Previous work has also observed that layers form natural groups [1], which we confirm using confusion matrices based on cosine similarity and CKA. While we cannot include these figures in the rebuttal due to the new NeurIPS policy, we will include them in the revised manuscript.

[1] Sun et al. "Transformer Layers as Painters" (2024)

Limited Compression Ratio

Comment (1) Limited compression ratio evaluation: The experiments primarily focus on a 25% compression ratio. While this is acknowledged in the limitations section, evaluating a wider range of ratios would provide a more complete picture of the method’s robustness.

Response (1) We appreciate the reviewer’s suggestions and we thank the reviewer for his constructive comments which will strengthen our paper. Compression Ratio Evaluation: While our main focus was on a 25% compression ratio (as models below ~90% original performance often cannot be successfully used in practice), we agree that evaluating a wider range of compression ratios can be insightful. We include here a comparison up to 50% and we will also include this in the final version of the paper.

More Pruning Baselines

Comment (2) The primary comparison is with UIDL. Including more pruning baselines would make the comparison more convincing.

Response (2) In our initial experiments targeting LLMs, we have compared against multiple state-of-the-art model compression strategies, including SVD-LLM, LaCo, LLM-Pruner, SliceGPT, and LLM-Streamline. Please refer to tables (1, 2) in the main paper.

For Vision Transformers the reviewer is correct as we indeed provide comparisons only against UIDL. The main reason for this is that almost all of the aforementioned methods have considered and provided comparisons exclusively on LLMs. In addition, our experiments on vision tasks are not by any means exhaustive, and their role is mainly to highlight that our strategy can also be successfuly deployed on vision models. Nevertheless, we agree with the reviewer that including more comparisons would be more convincing and will better highlight the benefits of our strategy. For this reason, in the updated version of our manuscript we will include additional comparisons on vision models with the LLM-streamline baseline. We include these new results below.

Missing Reference

Comment (3) The core idea, replacing layers with linear transformations, has been explored in prior work [1, 2], although not specifically in the pruning context. These works are missing from the related work discussion.

Response (3) We thank the reviewer for bringing to our attention the works of ([1, 2]), which we will refer to in the related work of the revised paper.

Calibration Data

Comment (4) The calibration dataset, a key component of the method, is only clearly explained in Section 3.3.1. Introducing this concept earlier would improve clarity and understanding

Response (4) We thank the reviewer for his very helpful comment. We agree that introducing the calibration dataset earlier would improve the clarity of the paper. For this reason, in the revised manuscript we will make explicit mention to the importance of the calibration data in Section 2 where we describe our method.

NonLinear Transform

Comment (5) Can the authors comment on using non-linear transformations instead of linear ones? Would this further improve performance?

Response (5)

• While non-linear transformations could potentially improve performance, they a) require significant fine-tuning ("healing") on large datasets and b) unlike our method, the parameters used to represent the non-linear transform cannot be directly fused with the model's existing parameters. Our key contribution is a training-free approach that achieves strong results at reasonable compression ratios (as shown in our main results tables), where we outperform methods like LLM-Streamline (that uses non-linear transformations) without requiring any healing phase.

• For applications where substantial training is feasible, non-linear transformations could in principle lead to further improvements, but such a strategy would represent a different approach from our training-free method. For this reason, we have decided to focus on a practical zero-training solution, which despite its simplicity and minimal requirements can maintain a very competitive performance.

Enviromental Impact

Comment (6) How is the “environmental impact” calculated?

Response (6) We calculated the environmental impact using the CodeCarbon library, which estimates energy consumption and carbon emissions during computation. We will add an explicit mention of this in the revised version of our manuscript.

Task Specific LTs

Comment (7) Is the learned linear transformation task-specific? That is, does each task/dataset require a new calibration and transformation? How does this compare with other pruning methods in terms of transferability?

Response (7) We thank the reviewer for raising a very interesting point. In the context of this paper we chose open source diverse data for calibration and we didn’t explore the task-specific recovery. However, the reviewer’s comment prompted us to conduct a preliminary investigation into whether, and to what extent, performance would improve if a task-specific calibration dataset were used. To explore this question, we conducted the following experiment:

1- Use the train split of the Sciq benchmark as our calibration data.

2- Evaluate the model on the test part

3- Use a subset of Orca open-source dataset as calibration data

4- Evaluate on the same benchmarks and compare the performance for the different calibration datasets.

The results are reported for Llama3 8B and we observe that indeed task-specific calibration data can further improve the performance of the compressed model. We plan to include these findings in the revised version of the paper.

Calibration Data Sensitivity

Comment (8) The choice of calibration dataset seems to have a large impact on performance. Do the authors have any insights on the reason of this sensitivity?

Response (8) In Table 3 of the main manuscript, we present an evaluation of model sensitivity to the calibration data. The results indicate a dependence of model performance on data type. For example, FineWeb, which consists of raw text, leads to a degradation in performance when used for instruction-tuned models, in contrast using instruction-finetuned data such as orca or mixed-domain corpora, including a multilingual composition of 66 languages, yields more stable and consistent performance across evaluated benchmarks. On the other hand, self generated data demonstrates superior performance in terms of language modeling efficacy, as measured by perplexity. However, this advantage can be accompanied by a decline in reasoning-oriented tasks likely related to the presence of noise, factual inaccuracies, or hallucinated content within the self-generated data.

In summary, our analysis reveals two principal observations regarding calibration data sensitivity: (1) instruction-tuned models achieve better performance when calibrated with instruction-aligned data; (2) while self-generated data can enhance language modeling capabilities, their potential contamination with spurious or incorrect content may adversely affect performance on downstream reasoning tasks. These insights are solely based on our experimental results. Further investigation of this topic is very interesting and we will consider it for our future work.

Justification for Removing Entire Blocks:

Comment (1): How is it that these entire complex blocks so redundant? ... Is it because some of the deeper layers not very useful? If so, why?

Response (1): The reviewer raises a valid and interesting question. While we don't have a definite answer, we hypothesize that large transformer models are often over-parameterized. This is supported by the fact that older models (like Llama 1 and 2 7B) are more compressible than newer ones (like Llama 3 8B), with the newer models being trained on significantly more data. Same findings extend to Qwen models (trained on 18T tokens), which are less compressible than Llama 3 (15T tokens). This suggests that better-trained models use their capacity more efficiently. Additionally, Huge models (e.g., Llama 3 70B) can be pruned more aggressively (up to 35%) with minimal performance loss.

Why Not Remove Redundant Blocks from the Start?

Comment(2): Can this not be done already from the outset? Can this prune and replace strategy be used during training also?

Response (2): While our findings in this work suggest that it is possible to achieve the same performance with smaller models, trying to train from the start such a small model that matches the performance of the original larger model can be very challenging. The main reason is that the objective function being minimized during training is highly non-convex and there is always the possibility of landing to a "bad" local minima. Prior work has demonstrated that over-parameterization can help during training, makes the overall loss more amenable to optimization and improves convergence (we refer to the work on ExpandNets [Guo, Shuxuan and Alvarez, Jose M. and Salzmann, Mathieu, ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks]). Therefore, while it might be possible to design more efficient architectures upfront, current training methods seem to benefit from more parameters.

Applicability to CNNs:

Comment (3): Why is this strategy well suited for transformer blocks. Why can a similar linear transformation not be used for other deep CNNs? What makes this so unique to transformer blocks, and can it be applied to CNNs? If not, why?

Response (3): The main reason we have focused on transformers is that they involve significanlty more parameters than CNNs, which casts pruning more impactful. While we don't believe that there is any particular reason why the same strategy couldn't be applied also to CNNs, their smaller size will most probably translate to less significant gains. However, exploring this direction could be an interesting future research direction. Furthermore, CNNs have different dimensions in between blocks due to the pooling operation that is typically used. In such case the learned linear transformation cannot be merged as in transformer architectures and will introduce additional parameters in the network. This will lead to smaller compression ratios given that CNNs rely on structured matrices that already use significantly less parameters than generic weight matrices.

Choice of Blocks

Comment (4): I would like the authors to speculate on the importance of choosing contiguous blocks versus optimizing the block sequence itself, Why do these blocks have to be contiguous? ... the linear layer is still used just after the last removed block.

Response (4): We thank the reviewer for his suggestion. In our experiments, we have explored both consecutive and non-consecutive block removal (Appendix Table 15). While contiguous blocks are often optimal (as cosine distance tends to favor them), the best grouping can vary by model. For example, Llama 3 performs better with 2 groups of 4 continuous blocks, while Mistral works best with 4 groups of 2 consecutive blocks. In practice, we have observed that even when we split into groups, in most cases the chosen groups tend to also be consecutive to each other. In addition, contiguous groups allow the merged linear transformations (LTs) to be combined into a single LT, simplifying the overall process. This is also the main reason that in the main paper we report only results with a single LT. Nevertheless, our library supports both strategies, with the greedy/non-contiguous search under the $\ell_2$ loss (LSTSQ method) being fast. We will add a discussion of this in the revised version of the paper.

Difficult Results Tables

Comment (5): what is the last column in Table 3? It just says %.

Response (5): We thank the reviewer for highlighting this lack of clarity. The "%" in Table 3 denotes the model’s performance relative to the original (uncompressed) model, where 100% is the baseline. Therefore, the values present in that column indicate the percentage of the original model's performance preserved after compression. We will revise the table headers and include a brief explanation in the caption to avoid any confusion to the readers.

Performance at high compression ratio

Comment (6): Clarification on performance at high compression ratio.

Response (6): We thank the reviewer for raising this important point. For reasonable compression ratios (e.g., 25% for 7B models), our results (Appendix Table 12) show that finetuning the compressed model can recover some additional performance. In fact, an interesting finding from these results is that either training the entire model or only the Linear Transformation (LT) will lead to almost the same performance, with the latter being significantly more computationally efficient than the former. The recovery of additional performance can be attributed to the fact that our initial optimization (using $\ell_2$/cosine loss) does not directly align with the LLM training objective. For even higher compression ratios, we agree with the reviewer's comment and we also recommend finetuning with LoRA or full-model training, as LTs alone may lack expressiveness. While our method provides a strong starting point, the optimal approach depends on the particular model and the available computing resources and budget. Below we attach a table with the results (confirming the superiority of our approach even for larger compression ratios). We will also include a corresponding figure that depicts these results in the final version of the paper.

Clarification on performance at high compression ratio

In this Table (1) we report the average performace accuracy for (Race, Winogrande, PIQA, BoolQ, OpenBookQA, and SciQ benchmarks), and the perplexity on wikitext. for Llama 3 8B instruct model.

Related Work

Comment (7): Perhaps the authors should consider describing works like [1] in their related work

Response (7): We thank the reviewer for bringing to our attention this work. The method in ([1]) is indeed relevant, as it shares similarities with SVD-LLM method, and we will expand our related work section to include it.

Clarification (1)

Comment (8): While I usually like a summary figure on page 1, I do expect it to be fully described. The figure 1 is placed even before the abstract and none of the abbreviations are introduced at this point; so the figure is misplaced. I would have preferred the method overview figure 2 in its place as it is quite self-contained

Response (8): We thank the reviewer for his suggestion. We agree with his assessment and we will follow his advice to move Figure 2 in the position of Figure 1.

Clarification (2)

Comment (9): reporting carbon emissions, make sure if it is CO2 or CO2 equivalent, as the latter is the more common measure

Response (9): We thank the reviewer for his comment. We used the publicly available tool codecarbon from github which reports emissions in CO₂ equivalent (CO₂eq), not just pure CO₂. We will make this point clear in the revised version of the manuscript.

Authors have satisfactorily addressed my concerns. I will increase my score from 4 to 5.