DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models

We want to thank the Reviewer BGu5 for your insightful and constructive comments. We will address each of your questions below. We use W1 to refer to weakness 1.

W1. Although the proposed method allows asymmetric dimensions in module weights across layers, it also requires the dimensions of hidden states from different layers maintain the same as that of the non-pruned model. This would restrict the use in scenarios where the dimension of the output hidden states is critically important (e.g., embedding for retrieval).

In our method, the model dimension (the dimensions of hidden states) remains unchanged after pruning, which is also the prerequisite for dimension-independent pruning. Consequently, our model can serve as a drop-in replacement for the original dense model in any downstream tasks, such as retrieval, as the feature map dimension will stay the same as the original model.

If this does not address your concern about the dimensions of hidden states and their impact on scenarios like embedding for retrieval, could you please provide more details or clarify your point further? We want to ensure we fully understand your concern and address it appropriately.

W2. The introduced indexing operation could be very time-consuming. Unexpectedly, the latency introduced by the indexing may outweight the latency reduced by pruning. And it is not discussed whether the pruned model is faster than the non-pruned one. I would recommend including such experiments in proper positions.

We present the acceleration results in Figure 5 (g) and Table 10 for the LLaMA-2 13B model, demonstrating that our model is faster than the non-pruned model. While we observe some speed improvements with our current model, we believe there is significant room for further enhancement, as the current PyTorch implementation of indexing operations is not efficient enough. Additionally, the theoretical FLOPs of indexing operations are very low. This is also discussed in the Limitation section of our paper. We believe the speed of our method can be further improved by incorporating a custom implementation of the matrix multiplication algorithm, where pruned columns and rows are skipped instead of using indexing operations. However, this approach primarily focuses on the engineering perspective and is beyond the scope of this work.

We want to thank the Reviewer AWnP for your insightful and constructive comments. We will address each of your questions below. We use W1 to refer to weakness 1 and Q1 to refer to question 1.

W1. Limited novelty beyond combining existing concepts

Our method is novel because it introduces a new pruning configuration by breaking structural dependencies, significantly expanding the potential search space for an optimal sub-network which is not presented in existing works. This increased search space makes most methods that rely on static metrics (such as weight norms) ineffective. Additionally, reinforcement learning or evolutionary search algorithms struggle in this context due to the difficulty of managing fine-grained search requirements, such as deciding whether to prune a single dimension. To address this, we employ gradient-based search as a highly efficient tool. From this perspective, our method is novel because we demonstrate that increasing the capacity (using latent hypernetworks) for gradient-based search is a highly effective approach for handling large and complex search spaces, such as DISP-LLM.

W2. Inconsistent performance gains across models/tasks

We believe it is reasonable that the performance gains for different models are different, as they are pre-trained with different data and computational costs. Models trained with higher costs are naturally harder to prune. For instance, consider the perplexity of OPT 6.7B and LLaMA-2 7B across different compression rates. It is possible to achieve lower perplexity than the dense model with OPT 6.7B (9.89 vs. 10.86), whereas it is impossible for LLaMA-2 7B (6.10 vs. 5.12), both of which are at a 20% compression rate. LLaMA-2 7B is often regarded as a better pre-trained model than OPT 6.7B, and it is also more challenging to prune. Thus, the performance of our method is naturally different for these two models.

We also think that performance gains across different tasks should vary since different tasks have different emphases. A single, small dataset like Wikitext or Alpaca cannot cover all perspectives, resulting in varied performance gains across different tasks.

If the above explanation does not address your concern about inconsistent performance gains across models/tasks, could you please provide more details? We would be happy to address any specific aspects you are referring to.

W3 & Q1: Insufficient ablation studies and sensitivity analysis for hypernetwork architecture and hyperparameters

Please see the first part of the general response.

W4 & Q2: Lack of detailed detailed computational cost comparisons.

We listed the detailed computational cost comparison below:

Computational Costs comparisons

The computational cost of our method is also listed in Table 5 in the Appendix. Compared to LLM Suregon, our method is 14.76 x to 27.39 x cheaper in terms of US dollars.

W5. Inadequate error analysis and failure case exploration

Please see the second part of the general response.

W6. Overstated claims based on limited experiments

We believe our experimental results are accurate and not overstated. Firstly, our method surpasses semi-structural pruning methods in terms of perplexity at the same compression rate. To the best of our knowledge, this is the first time that a structural pruning method outperforms semi-structural pruning methods in perplexity. Additionally, as mentioned in the contribution section, our method outperforms other baseline methods, with or without fine-tuning, without requiring updated model weights. This claim is also well reflected in the experimental section.

If there are specific claims or aspects of our experiments you find overstated or insufficiently supported, please let us know. We would appreciate more details so we can address your concerns thoroughly.

W7. Insufficient discussion of pruning ratio vs. performance trade-offs

Please see the third section of the general response.

Q3. How does it perform on fine-tuning tasks?

We further evaluate our method on the Vicuna 7B model, which is a model trained by fine-tuning the LLaMA 7B model. We list the results below:

Average zero-shot task performance with the Vicuna 7B model

The result above is consistent with pre-trained models, and our method can be seamlessly applied to models fine-tuning on different tasks.

Q4. Have you explored progressive pruning approaches?

For now, we have not explored the progressive pruning approach. Progressive pruning typically involves updating model weights through either full fine-tuning or LoRA fine-tuning, which would significantly increase the computational costs of our method. Given the current strong performance of our method, we aim to maintain its low cost without introducing an additional weight training process.

We want to thank the Reviewer 6CTn for your insightful and constructive comments. We will address each of your questions below. We use W1 to refer to weakness 1 and Q1 to refer to question 1.

W1. More comprehensive literature review about model compressing, like sparse mechanism.

Thanks for pointing this out. We will add more related works regarding sparse mechanisms, like ‘The Lottery Ticket Hypothesis’ and other works related to sparse mechanisms and model compression.

W2. Any case study for qualitative analysis to the pruned model? It is possible to conduct some interpretability analysis showing where the difference between sliceGPT and the proposed method along with a specific input example.

We provided two examples in Table 1 of the rebuttal pdf file. The examples are obtained based on removing 20% of the model weights with DISP-LLM and 20% slicing of SliceGPT (~10% compression rate). Both models are compressed from LLaMA-2 7B and they are not finetuned. From these two examples, we can see that SliceGPT only generates a small part of meaningful content and it then starts to repeat itself. On the other hand, our method tends to generate more relevant content, and the self-repeating behavior is much less obvious. These observations comply with the performance gap between DISP-LLM and SliceGPT.

Q1. When training with the pruning objectives all the layers are fine-tuned or only selected layers?

We do not fine-tune model weights. Throughout the whole training process, only the hypernetwork is trained and thus the total trainable parameters of our method are quite small which is usually around 2-3% of the original model weights. The hypernetwork is only responsible for controlling the selection matrices of different layers as shown in Figure 2.

Q2. Any insights from Figure 6, different model width as layer changes. If the observations aligned with existing research results?

From Figure 6, we can observe that the middle layers of the model are better kept. This may comply with some interpretability works. In [1], they argue that the middle layer of the model is likely to contain interesting, abstract features.

On the other hand, our method does not retain as much of the first and last layers as LLM Surgeon does. This is probably the result of removing structural dependency so that each layer has a larger flexibility.

We believe that future works are needed to fully understand the importance of different layers under our dimensional independent setting.

[1] Templeton, A. (2024). Scaling monosemanticity: Extracting interpretable features from Claude 3 sonnet. Anthropic.

We want to thank the Reviewer FhgQ for your insightful and constructive comments. We will address each of your questions below. We use W1 to refer to weakness 1 and Q1 to refer to question 1.

W1. Even though DISP-LLM does not update the original weights of LLMs, training is required.

In fact, the training of our method is very lightweight, as we pointed out in the response to Q3. Moreover, with a small training costs, our method largely outperforms previous pruning methods which suggests that our method is very cost-effective. In addition, freezing model weights give us potential opportunities to further reduce the costs. For example, we can quantize the original model weights to lower bits to further reduce its GPU memory costs. After pruning, we can restore the model to the original precision. If a method needs to train model weights during pruning, such as LLM Surgeon, it must implement the complex pipeline of quantization-aware training. In contrast, our method does not require this because we do not update the weights of the original model.

W2 & Q1 & Q2. How to decide $\lambda$ to 6 and the performance sensitivity to $\lambda$.

Please refer to the first section of the general response. Overall speaking, the hyperparameter selection of our method is not complex.

Q3. As you state in the paper, DISP-LLM is as efficient as LoRA. If so, can applying LoRA weights only to the kept dimensions be a baseline?

Indeed, the trainable parameters of our method are around 2-3% of the original model weights which is at a similar scale to LoRA. Though the training cost is similar, applying LoRA does not remove any model weights or reduce the model size. Thus, it may not be a useful baseline. If we apply LoRA weights to the pruned model, then it becomes recovery finetuning, which is definitely doable and we present some results here:

LoRA finetuning of the compressed LLaMA-2 7B model.

We follow similar settings of the SliceGPT and the model is fine-tuned on Alpaca. We can see that our method can be further boosted by using parameter-efficient finetuning techniques. Since the performance without fine-tuning is already good enough, we prefer not to involve this additional process in our method to save time and computational costs.

We want to thank Reivewer FhgQ for giving us the chance for further clarification. We list the comparison of our method, quantization, and QLoRA in the table below:

As a pruning method, DISP-LLM is orthogonal to Quantization or QLoRA methods as shown in the table, meaning it can be combined with quantization to further reduce model size. Firstly, we can combine our method and Quantization by applying DISP-LLM on top of a quantized model to reduce the training costs as stated in the response to W1. Additionally, once a model has been compressed using DISP-LLM, QLoRA can be employed to fine-tune the model with minimal resources. Besides the aforementioned contents, we believe that the search space in the dimensional independent setting offers new insights for designing a compact large language model that is not captured by QLoRA or quantization.