CLOVER: Cross-Layer Orthogonal Vectors Pruning

Q1: Benchmark the effectiveness of the proposed method.

A1: For the pruning experiments, we evaluated the effectiveness of CLOVER using the Wikitext2 eval-dataset with the following settings: batch size of 32, max sequence length of 1600, and tested on an A100-PCIE-40GB GPU with different pruning ratios for GPT-2-XL inference speed. The results are summarized in the table below:

The results clearly demonstrate that pruning the head dimension significantly accelerates model inference. Notably, CLOVER pruning 50% of the head dimensions outperforms vanilla pruning at 37.5% by generating 144 more tokens per second while maintaining a lower perplexity.

For the PEFT benchmarking experiment, please refer to A2 for Reviewer DDfo.

Q2: Conduct pruning on models other than GPT-2-XL and combine CLOVER with existing SOTA pruning techniques.

A2: Thank you for providing a list of some state-of-the-art pruning research. In Section 4.4, we also applied pruning to Whisper-large-v3 in addition to GPT-2-XL. Due to time constraints, we only compared CLOVER with SliceGPT during the rebuttal period. In the camera-ready version, we will cite them all and compare CLOVER with those methods. As mentioned in A3 for Reviewer jpx5, we expanded our experiments by using CLOVER pruning on Deepseek and OPT models. Early results indicate that CLOVER achieves a 5% pruning ratio improvement on these models, and we plan to refine these experiments further. Additionally, CLOVER supports pruning in models such as ViT and DiT, and we will continue to compare CLOVER with pruning methods in these domains.

Q3: L2-norm-based pruning typically utilizes the norm of either W_q or W_k individually, or their sum, rather than their product.

A3: You are correct that many L2-norm-based pruning methods use the norm of either W_q or W_k individually or their sum. However, there are cases where the product of both norms is used. For example:

1. When pruning the output of one layer affects the input of the next, considering both layers together may yield better results, and in such cases, the product of the two layers is used [1].

2. In the attention layer, the attention score is influenced by both Q and K, which is why their product is considered in pruning [2].

Q4: Does the experiment in Section 4.4 maintain the parallelizability of multi-head attention? Specifically, does it remove the same number of dimensions for each head?

A4: Yes, all pruning in this paper maintains the parallelizability of multi-head attention. In Section 4.1, the pruning ratio is consistent across all layers, with each attention head having the same dimension. In Section 4.4, to achieve higher pruning rates while maintaining a training-free approach, we allow for different pruning ratios for each layer while ensuring that the number of dimensions removed is the same for each attention head within the same layer. This design ensures that inference can proceed in parallel.

Q5: Figure annotations and typo corrections.

Thank you for pointing out these issues. In the camera-ready version, we will revise the figure captions for clarity and make necessary corrections to improve the overall expression.

Q5.1: In Figure 4, what values are being ranked in “Top,” “Next,” and “Bottom”?

A5.1: As explained in [Lines 375-377, left] and [Lines 362-364, right], after performing SVD on the model, According to the magnitude of the singular values, the singular vectors are denoted as: “Top-256,” “Next-256,” and “Bottom-256.” We found that only 10% of the feature components are projected onto the singular vectors corresponding to the Top-256 singular values.

Q5.2: In Figure 5, what do the horizontal and vertical axes represent?

A5.2: In Figure 5, the X-axis represents the index of the feature values of $\Delta W$, while the Y-axis represents the magnitude of these feature values.

We hope this clarifies your questions and concerns. We greatly appreciate your effort and look forward to your further feedback on our response.

References:

[1] Fluctuation-based Adaptive Structured Pruning for Large Language Models, AAAI 2024.

[2] Round and Round We Go! What makes Rotary Positional Encodings useful?, ICLR 2025.

Q1: Is the comparison in Section 4.5 fair, considering that LoRA is designed for low-rank?

A1: As noted in A2 for Reviewer DDfo, CLOVER and LoRA have an identical number of trainable parameters. CLOVER offers slight improvements in both training time and GPU memory consumption when compared to LoRA.The original paper also present comparisons with HiRA, a full-rank update method.

Q2.1: Are the experiments conducted with a single learning rate per method and a single seed?

A2.1: The results for LoRA and DoRA presented in Table 2 are directly taken from the original DoRA paper, where the hyperparameters are carefully tuned. Similarly, the results for HiRA are cited from its original publication. In contrast, we introduce new experimental results for PiSSA and CLOVER, both of which are optimized with the best learning rates and aligned hyperparameters. Specifically, PiSSA achieves optimal performance at a learning rate of 2e-5, while CLOVER performs best at 1e-4, as shown in the table below:

In the original paper, PiSSA and CLOVER were trained using the default seed (42). Following your suggestion, we tested multiple seeds (40, 41, and 42). We will conduct these experiments on additional models and include the updates in the camera-ready version.

Q2.2: Why was the learning rate of CLOVER† adjusted from 6e-4 to 6e-3?

A2.2: Both Vanilla Pruning and CLOVER employ full-parameter fine-tuning and same learning rate, while CLOVER† fine-tunes only singular values. Generally, PEFT converge more slowly than full-parameter methods. To speed up convergence, we increased the learning rate and removed weight decay for CLOVER†.

Q3: How does CLOVER compare to SOTA methods?

A3: Please refer to A3 for Reviewer jpx5.

Q4: What does the y-axis represent in Figure 2?

A4: After applying CLOVER, we multiply the square root of the singular values by each singular vector and calculate their L2 norms. This result is equivalent to directly using the singular values. Therefore, we uniform this representation for better comparison.

Q5: Section 4.4 presents a single example, which might seem like cherry-picking.

A5: The example is not cherry-picked; it is an official sample provided with the Whisper-large-v3 model [1]. Due to significant linear redundancies in the attention heads of the Whisper model, CLOVER demonstrates substantial effectiveness. We aimed to highlight this clearly in the paper. We plan to systematically evaluate the Whisper model and, as suggested, move this case study to supplementary materials.

Q6: What are the “series” and “parallel” baselines?

A6: “Series” and “parallel” are two standard baselines commonly used in parameter-efficient fine-tuning for NLP tasks. The series adapter [2] inserts a trainable module sequentially after the Attention and MLP layers, while the parallel adapter [3] integrates a trainable module parallel to the Attention and MLP layers.

Q7: Why is D compared with a single head in the experiments?

A7: The comparison is fair because we are comparing the reduced dimensions within each attention head, not the entire set of heads against a single head. This process reduces the dimension of each original attention head from $D \times d$ to $D \times r$, where $D$ is the hidden size, $d$ is the attention head dimension, and $r$ represents the attention head rank, with $r \leq d \ll D$.

Q8: What is CLOVER†?

A8: Please refer to A2 for Reviewer jpx5.

Q9: Are LoRA and DoRA properly tuned?

A9: Please refer to A2.1.

Q10: What does L262 mean by “Due to the non-linear…matrix”?

A10: Matrix merging and decomposition are linear operations, which require linearity to preserve equivariance during orthogonalization. However, RoPE encodings vary depending on token positions, introducing non-linearity. This prevents cross-layer merging and decomposition. To address this, we perform orthogonal decomposition within a single layer rather than across layers when fine-tuning Q-K Vectors with RoPE.

Q11: What does the x-axis represent in Figure 1d?

A11: It represents the pruning ratio.

Thank you for your thoughtful and constructive feedback, which has significantly improved our paper. Please feel free to reach out if you have any further questions or concerns.

[1] https://huggingface.co/openai/whisper-large-v3

[2] Parameter-Efficient Transfer Learning for NLP, ICML 2019.

[3] Towards a Unified View of Parameter-Efficient Transfer Learning, ICLR 2022.

Q1. Clarification of Domain

Q1.1: Whether CLOVER is a PEFT method.

A1.1: CLOVER is a straightforward yet effective re-initialization method that benefits both pruning and PEFT. Pruning and PEFT are closely interconnected, as both aim to achieve efficient training and inference under resource constraints. Typically, pruning requires a recovery process, and PEFT methods are commonly applied to minimize training overhead and enhance efficiency, particularly in data-scarce scenarios [1]. While many previous pruning methods primarily focus on the pruning phase and directly apply LoRA in the recovery phase [1, 2, 3, 4], CLOVER uniquely addresses both tasks simultaneously. It not only enhances pruning rates (comparing with vanilla pruning and SliceGPT, refer to A3 for reviewer jpx5) but also improves PEFT training effectiveness. We believe this dual approach provides valuable insights for efficient model deployment.

Q1.2: Whether CLOVER can only be used for LLMs.

A1.2: Indeed, our proposed CLOVER method targets on the attention component. But since attention mechanisms are widely used across many domains beyond LLMs (such as Vision Transformers (ViT) for image classification, detection, segmentation, DiT for image generation, and Automatic Speech Recognition, as well as Text-to-Speech), CLOVER has potential and value in a wide range of applications. Additionally, in Section 4.4, we provide experimental evidence using the speech recognition model Whisper, where CLOVER achieves parameter reductions of 56.01% in the Q-K pair and 36.82% in the V-O pair without the need of further fine-tuning. CLOVER’s flexibility allows it to integrate with other techniques, extending its applicability to other models.

Q2. GPU Memory Consumption and Runtime Comparison with LoRA

A2: We conducted experiments on the LLaMA-2-7B model using commonsense data, trained for 3 epochs with the following hyperparameters: model_max_length = 1024, per_device_train_batch_size = 2, gradient_accumulation_steps = 2, num_gpus = 4, executed on 4 × A800-80G GPUs, with seed = 42. The comparison of memory consumption and runtime is as follows:

the results demonstrate that CLOVER consumes less GPU memory and exhibits shorter training runtime compared to LoRA. We attribute this to CLOVER being applied between two layers, whereas LoRA operates in parallel with the main branch. This enables sequential computation, eliminating the need to retain the input features of the main branch.

Q3. Rigor in Demonstrating Full-Rank Adaptation’s Advantages and Information Redundancy

A3: In Section 4.5, we emphasize the importance of full-rank adaptation by demonstrating that only 6% of the orthogonal components from randomly selected commonsense samples project onto low-rank vectors, leading to inefficient data utilization. Full-rank adaptation resolves this issue by fully utilizing the data’s orthogonal projection components. Sections 4.6 and 4.7 visualize the matrices fine-tuned by various methods, showing that CLOVER’s effects closely resemble full-parameter fine-tuning, whereas LoRA, constrained to low-rank updates, introduces intrusive dimensions. Several studies [5, 6, 7, 8] have rigorously compared full-rank and low-rank adaptation, consistently concluding that full-rank updates outperform low-rank methods. Regarding the concern about information redundancy, we believe this issue is more closely related to the number of trainable parameters rather than the direction of adaptation itself. Since CLOVER maintains parameter efficiency, it mitigates redundancy effectively while offering greater flexibility and learning capacity.

Q4. Same as Q2

We hope that these responses address your concerns regarding CLOVER, and we look forward to further discussing any additional points you may have.

Reference:

[1] LLM-Pruner: On the Structural Pruning of Large Language Models, NeurIPS 2023.

[2] SlimGPT: Layer-wise Structured Pruning for Large Language Models, NeurIPS 2024.

[3] Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LargeLanguage Models, ICML2024.

[4] SliceGPT: Compress Large Language Models by Deleting Rows and Columns, ICLR24.

[5] HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models, ICLR 2025.

[6] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024.

[7] QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation, NeurIPS 2024.

[8] Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices, arxiv preprint.

We sincerely appreciate your thoughtful and constructive feedback. We have carefully addressed each of your concerns and will incorporate your valuable suggestions into the camera-ready version of the paper.

Q1: Clarify how the proposed method reduces memory load during inference.

A1: Autoregressive token generation in LLMs involves frequent access to the previous key-value (KV) cache, which can significantly hinder inference speed [1]. CLOVER addresses this issue by reducing the dimensionality of attention heads, thereby reducing the KV cache size and effectively lowering memory usage during inference. As noted in A1 to Reviewer AvA1, this reduction in KV cache size contributes to more efficient processing during inference.

Q2: Clarify the confusing descriptions of CLOVER and CLOVER†.

A2: Thank you for pointing out the confusion between CLOVER and CLOVER†. We acknowledge this naming conflict, especially in pruning experiments where both techniques are used together. To resolve this, we will clearly redefine the terms in the camera-ready version:

• CLOVER: After pruning the less important components based on singular value magnitudes, only the singular values are fine-tuned, which are then merged back into singular vectors (this corresponds to the original CLOVER†).

• CLOVER†: After pruning the less important components, they are directly merged into singular vectors, followed by full-parameter fine-tuning (this corresponds to the original CLOVER).

This adjustment will ensure clarity and consistency across both the methodology and experimental sections. In this context, CLOVER† serves as an ablation study, validating the effectiveness of the orthogonal initialization method in comparison to vanilla pruning.

Q3: Setting of Vanilla Pruning and comparison with SoTA methods.

Vanilla pruning differs from CLOVER primarily in that it does not perform orthogonalization. Instead, we directly prune the head dimension based on the vector norm. (Lines 182-183 right, 275-293 left). CLOVER is orthogonal to existing pruning methods and can be effectively combined with techniques like SliceGPT [2]. In response to your suggestion, we compared the combination of CLOVER and SliceGPT with SliceGPT alone on both OPT and Deepseek models, reporting the perplexity results on WikiText2, as shown in the table below:

The baseline represents the original model with no pruning. CLOVER achieves performance comparable to SliceGPT, despite pruning 30% of attention parameters, compared to SliceGPT’s 25%. This demonstrates the simplicity and effectiveness of CLOVER.

It is important to note that most current pruning techniques for Deepseek primarily focus on reducing the number of MoE experts [3] and token selection [4]. There is limited research on pruning attention heads, attention head dimensions, or latent dimensions. CLOVER facilitates pruning these components in Deepseek’s Multi-Layer Attention (MLA), significantly reducing the KV cache overhead. Additionally, CLOVER utilizes a small number of dominant singular values from the Query-Key matrices to dynamically identify important tokens for full computation, enhancing token selection accuracy, especially in long-context tasks when combined with methods like MoBA [4]. Moreover, by rotating Q-K and V-O pairs in attention heads, CLOVER minimizes outliers in weights and activations [5], which is advantageous for reducing quantization errors. We are excited to continue exploring CLOVER to optimize the efficient deployment of Deepseek models.

Q4: Address the inconsistency in the abstract between the submitted PDF and the ICML submission system.

A4: Thank you for bringing this to our attention. We will promptly update the abstract in the ICML submission system to ensure that it is consistent with the content in the submitted PDF.

We hope this addresses your comments and concerns effectively. We look forward to your feedback and the opportunity to further improve our work.

References:

[1] Keep the Cost Down: A Review on Methods to Optimize LLM’s KV Cache Consumption, COLM 2024.

[2] SliceGPT: Compress Large Language Models by Deleting Rows and Columns, ICLR 2024.

[3] MoE-Pruner: Pruning Mixture-of-Experts Large Language Models Using Hints from Its Router, arXiv preprint.

[4] MoBA: Mixture of Block Attention for Long-Context LLMs, arXiv preprint.

[5] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, arXiv preprint.