Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models

We greatly appreciate your thoughtful feedback and the opportunity to address your concerns.

Weakness 1: In section 4 ... PPL from ~5 to ~12.

Reply: We have conducted additional experiments to clarify the comparison.

It is important to note that ESPACE includes a fine-tuning stage with 200B tokens (as stated in Section 4.4 of the ESPACE paper), which significantly contributes to the <1 PPL increase at 50% compression. In contrast, the results we reported in Tables 2 and 3 of our paper are without any retraining. Fine-tuning results are presented in Table 4, where each pruned model is only retrained for ~128M tokens—orders of magnitude fewer than ESPACE.

To ensure a fair comparison, we reproduced the pruning step of ESPACE and compared it against SVD-LLM. ESPACE proposes six variants: MSE (Eq. 6), MSE-NORM (Eq. 7), GO-MSE (Eq. 8), GO-MSE-NORM (Eq. 8), NL-MSE (Eq. 9), and NL-MSE-NORM (Eq. 9). We exclude the NL-MSE variants from our comparison as they rely on backpropagation, which is infeasible on memory-constrained GPUs due to their high resource demands.

Perplexity on WikiText2 at 50% density using LLaMA2-7B:

SVD-LLM (W) indicates the standalone pruning output from SVD-LLM.

From the results:

• SVD-LLM performs slightly better than ESPACE’s best variant (GO-MSE-NORM) under comparable conditions (i.e., without fine-tuning).
• PIFA, M, and MPIFA (PIFA + M) consistently improve all ESPACE variants, demonstrating that our techniques are general-purpose and complementary, enhancing the performance of any low-rank pruning method.
• Based on these observations, SVD-LLM remains the strongest initialization among all tested options when no fine-tuning is used.

We have included this new comparison and discussion in the updated version of the manuscript.

Weakness 2: In equations (4) and (5) ... presented setup.

Reply: We agree that the quality of the initial low-rank decomposition, especially the choice of matrix $V$, can significantly influence the performance of the final reconstruction. A better-initialized matrix $V$ generally leads to a more accurate reconstructed matrix $U$, resulting in lower final PPL.

Currently, SVD-LLM remains the most effective low-rank pruning method in our setup, but this remains an open question. If future methods such as ESPACE can produce even better projections, they can naturally be integrated with MPIFA to achieve improved performance. Our framework is flexible and designed to enhance any low-rank pruning method, not tied to any single initialization strategy.

We have added this analysis and discussion to the updated version of the manuscript.

Weakness 3: Does it ... equation (8)?

Reply: We assume that by "EM" you are referring to an alternating approach that iteratively updates $U$ and $V$ using Equations (5) and (8) with more than 1 round, similar to alternating least squares (ALS). We are currently exploring this direction and will share the results as soon as they become available.

Weakness 4: The reliance on a mix ... golden uncompressed baseline?

Reply: In our preliminary findings, increasing $\lambda$ leads to overfitting on the calibration data, resulting in low perplexity on the calibration set but high perplexity on the full WikiText2 dataset.

To mitigate this overfitting, we explored several strategies:

1. Reducing $\lambda$, as mentioned in the right column of line 256, where the low-rank data flow acts as a form of regularization.
2. Increasing the size of the calibration dataset, which could help improve generalization.
3. Applying additional regularization techniques to control overfitting.

With sufficient calibration data, increasing $\lambda$ may become beneficial. We are currently running experiments to test this hypothesis and will report the results as soon as they are available.

Weakness 5: Can we have more evaluations on more downstream tasks such as LM eval harness and MMLU?

Reply: We have incorporated zero-shot evaluations on 8 downstream tasks from the SuperGLUE benchmark, using the lm-evaluation-harness framework as recommended.

The results, available at https://anonymous.4open.science/r/PIFA-68C3/zero_shot.png, show that MPIFA_NS outperforms other low-rank baselines on 6 out of 8 tasks, and reduces the average accuracy gap by 42.8% compared to the best-performing baseline. These evaluations have been added to the revised manuscript.

We are currently working on extending our evaluation to include MMLU and will provide updates as results become available.

Thank you for your detailed review and valuable insights. We are glad to address your concerns and provide clarifications.

Weakness 1: It is not very clear how QR or LU decomposition serve as the backbone of finding pivot rows would differ from each other.

Reply: The key idea behind PIFA is to select a set of linearly independent pivot rows that can be used to express the remaining non-pivot rows as linear combinations. Since a matrix of rank r contains multiple valid sets of r linearly independent rows, any such set can serve the purpose of PIFA.

QR decomposition with column pivoting and LU decomposition with row pivoting are variants of QR and LU respectively, designed to improve numerical stability compared with original QR or LU. Their pivoting mechanisms reorder the columns (or rows), making the leading r columns (or rows) guaranteed to be linearly independent. The corresponding permutation matrix—produced as a byproduct of either decomposition—can then be used to identify the indices of the pivot rows (or columns).

In summary, both QR and LU decomposition with pivoting can be used to extract a valid set of pivot indices, and they are mathematically lossless in the infinite precision setting.

To further clarify their practical differences under limited numerical precision, we conducted additional experiments comparing the residual error when using pivot rows to reconstruct non-pivot rows under both Float32 and Float16 precision:

These results are averaged over 10 random singular matrices. While both methods yield very low residual errors, QR with pivoting achieves lower numerical error than LU. This suggests that QR decomposition with pivoting is the preferred choice for identifying pivot rows in PIFA.

We appreciate the reviewer’s question and have included both this explanation and the supporting experiment in the updated manuscript.

Weakness 2: It would be much better to compare to structured pruning also as baselines.

Reply: Thank you for the suggestion. To address this, we have included a structured pruning baseline, LLM-Pruner, in the updated version of the manuscript.

Below is the perplexity comparison on WikiText2 using the LLaMA2-7B model across various parameter densities:

On average, MPIFA reduces the perplexity gap by 87.2% across these density levels compared to LLM-Pruner.

We also benchmarked inference speedup and memory usage relative to dense linear layers, using linear layers of different dimensions on an A6000 GPU:

Speedup over dense linear (higher is better):

Memory usage relative to dense (lower is better):

At the same density (55%), PIFA achieves similar speedup and memory efficiency as LLM-Pruner. When comparing MPIFA at 55% density to LLM-Pruner at 70% density, MPIFA consistently offers lower perplexity, faster inference, and reduced memory usage.

We have incorporated these comparisons into the revised manuscript.

Weakness 3: How about the memory footprint of PIFA and baselines.

Reply: Thank you for the question. We provide a detailed analysis of the memory footprint for both inference and compression.

Memory Footprint During Inference:

• As shown in Figure 5, PIFA consistently achieves lower memory usage compared to low-rank layers at the same rank. For example, at $r/d = 0.5$, PIFA losslessly compresses the memory of the low-rank layer by 24.2%.
• Table 5 demonstrates that PIFA at 55% density uses slightly less memory than a 2:4 semi-sparse layer.
• Table 6 shows that MPIFA_NS at 55% density reduces end-to-end memory usage by 42.8% on LLaMA2-7B and 43.8% on LLaMA2-13B.

Memory Footprint During Compression: We report peak memory usage during compression for each method:

For memory efficiency in method M:

1. Online calibration is used, where only the current sample is loaded to GPU. Other samples remain on CPU until needed.
2. Only the current pruning layer is loaded to GPU, while all other layers remain on CPU during processing.

We have included this analysis of memory footprint during both inference and compression in the updated version of the manuscript.

Thank you for your insightful comments and suggestions.

Weak1: The claim ... overstated.

Reply: We have removed the phrase “for the first time” from the abstract.

Weak2: Missing LLM Evaluation ...

Reply: We have expanded our evaluation to include both perplexity and downstream task accuracy on widely adopted benchmarks.

For perplexity, we report results on the C4 dataset, a large-scale and diverse corpus commonly used for evaluating LLMs. Across various compression densities, MPIFA consistently outperforms existing low-rank pruning baselines. Specifically, MPIFA reduces the perplexity gap by:

• 47.6% on LLaMA2-7B
• 34.5% on LLaMA2-13B
• 55.3% on LLaMA2-70B
• 62.6% on LLaMA3-8B

on average across all densities, compared to the best-performing low-rank pruning method.

To further assess real-world utility, we conducted zero-shot evaluations on the SuperGLUE benchmark, covering 8 downstream tasks using the lm-evaluation-harness framework. MPIFA_NS achieves the highest accuracy on 6 out of 8 tasks, and reduces the average accuracy gap to the dense model by 42.8%, compared to the strongest low-rank baseline.

The full evaluation results are available here:

• C4 PPL results: https://anonymous.4open.science/r/PIFA-68C3/c4_ppl.png
• SuperGLUE zero-shot results: https://anonymous.4open.science/r/PIFA-68C3/zero_shot.png

We appreciate the reviewer’s suggestion and have incorporated these evaluations into the revised manuscript.

Weak3: Compression Resource Comparison ... practicality.

Reply: We have added a detailed comparison of the compression time and peak memory usage during compression across different methods.

Compression Time (on A6000 GPU):

Peak Memory Usage During Compression:

Notes:

• All compression times are measured on a single A6000 GPU. On an A100 GPU, the compression time is approximately half.
• For the "M" method, we report only the reconstruction time, excluding the time taken by the low-rank pruning step.

As for memory efficiency in method M:

1. Online calibration is used, where only the current sample is loaded to GPU. Other samples remain on CPU until needed.
2. Only the current pruning layer is loaded to GPU, while all other layers remain on CPU during processing.

These comparisons have been included in the revised manuscript to better reflect the practicality of our method.

Weak4: Missing Structured Pruning ... time.

Reply: We have conducted additional experiments to include LLM-Pruner as a structured pruning baseline.

Below is the perplexity comparison on WikiText2 using the LLaMA2-7B model at various densities:

On average, MPIFA reduces the perplexity gap by 87.2% compared to LLM-Pruner.

We also benchmarked the inference speedup of PIFA and LLM-Pruner layers (relative to dense linear layers) on an A6000 GPU across different hidden dimensions:

Memory usage during inference, relative to dense linear:

At the same density (55%), PIFA achieves similar speedup and memory efficiency as LLM-Pruner. When comparing MPIFA at 55% density to LLM-Pruner at 70% density, MPIFA consistently offers lower perplexity, faster inference, and reduced memory usage.

We have included this comparison with LLM-Pruner in the updated manuscript. Additionally, we are evaluating other recent structured pruning methods and will report those results as they become available.

Weak5: Retraining ... is applied.

Reply: Fine-tuning experiments are already included in Table 4 of the original manuscript. All pruned models are retrained for one epoch on a mixed dataset consisting of 2% WikiText2 and 98% C4, to recover performance. This setup ensures a fair and consistent comparison across all methods.

The results show that fine-tuned MPIFA continues to outperform other low-rank pruning methods, and also achieves slightly better performance than fine-tuned semi-sparse models. Details of the fine-tuning setup can be found in Appendix B.3.

Weak6: Structured Compression ... of semi-structured pruning.

Reply: We have updated the manuscript to include discussions of SLEB, ShortGPT, and DISP-LLM in the related work section. We appreciate the reviewer’s recommendation to acknowledge them.

Thank you for your valuable and constructive feedback. We appreciate the opportunity to address your concerns.

Weakness 1: (Expand evaluation) The authors have failed to ... existing tools like LMEvalHarness etc.

Reply: Thank you for raising this concern. We have extended our evaluation in two directions:

1. C4 Perplexity (PPL) Evaluation: We report perplexity on the C4 dataset across various model sizes and parameter densities. The results demonstrate that MPIFA significantly outperforms existing low-rank pruning methods, reducing the perplexity gap by

   • 47.6% on LLaMA2-7B
   • 34.5% on LLaMA2-13B
   • 55.3% on LLaMA2-70B
   • 62.6% on LLaMA3-8B
     on average across all densities, compared to the best-performing baseline.
     Full results: https://anonymous.4open.science/r/PIFA-68C3/c4_ppl.png

   Example results at 50% density:

   Model	SVD	ASVD	SVD-LLM	MPIFA
   LLaMA2-7B	58451	25441	129.8	52.01
   LLaMA2-13B	18196	3537	110.4	42.03
   LLaMA2-70B	7045	OOM	44.10	29.04
   LLaMA3-8B	143573	108117	784.8	257.4

2. Task-Centric Evaluation: We conducted zero-shot evaluation on the SuperGLUE benchmark using 8 downstream tasks via the lm-evaluation-harness (https://github.com/EleutherAI/lm-evaluation-harness) framework. All methods use WikiText2 as the calibration dataset, and follow the same configuration as in the main manuscript.
   Results: https://anonymous.4open.science/r/PIFA-68C3/zero_shot.png

   MPIFA_NS achieves the best accuracy on 6 out of 8 tasks and reduces the average accuracy gap by 42.8% compared to the best-performing low-rank pruning baseline.

We appreciate the reviewer’s suggestion. These additional evaluations have been added to the revised manuscript.

Weakness 2: Wikitext2 dataset doesn't contain enough details (e.g. seqlen etc.).

Reply: Thank you for the feedback. The sequence length used in all experiments, including both WikiText2 and C4, is 2048. We have updated the manuscript to include this information.

Weakness 3: In addition, it will be also interesting to see how the proposed method extend beyond only one single family of models (LLaMa 2/3) - may be on some MoE style models.

Reply: Thank you for the suggestion. We are currently exploring this and will report the results here as soon as possible.

Weakness 4: (Expand related work) Related work requires some upgrade ... arXiv:2502.02723.

Reply: Thank you for the suggestion. We have updated the manuscript and included these articles in the related work section.

Weakness 5: Authors introduce two variables r and d directly in abstract and introduction without explicitly defining it.

Reply: Thank you for the suggestion. We have revised the original phrasing “at $r/d = 0.5$” to “at rank equal to half of the dimension” for clarity.

Weakness 6: (Improve the article's structure) The writing of the paper ... in supplementary.

Reply: Thank you for this valuable suggestion. We have moved Section 3.3 to the appendix while retaining the key conclusions about the computational and memory cost of PIFA in the main paper. Since the final version allows one additional page, we have also moved the previously referenced plots from the appendix (cited in Section 5.3) into the main body, ensuring the draft is more self-contained without exceeding the page limit.

Weakness 7: The paper mentions that PIFA is fully differentiable, suggesting potential integration into the training stage. Have the authors explored this direction?

Reply: Thank you for raising this interesting point. We are happy to share our preliminary findings. While PIFA ensures lossless conversion in the forward pass, it does not guarantee identical gradient descent dynamics compared to traditional low-rank training methods, as different factorizations can lead to different gradients.

Weakness 8: How can this method integrated and benefit some recent quantization techniques like ZeroQuant-v2.

Reply: Thank you for the suggestion. We are currently exploring this and we will provide further updates as soon as possible.