Learning Parameter Sharing with Tensor Decompositions and Sparsity

Dear reviewer,

We greatly appreciate the feedback and the valuable insights you provided, which are instrumental in helping us improve SParS. We have addressed your concerns below:

Limited Comparison

We selected AAFM and GFM, as proposed by Yu and Wu [1], as baselines due to their effectiveness and relevance to our approach. As other compression methods like NAS or distillation require much higher computational resources. However, we agree that expanding the comparison to include other compression algorithms would provide a more comprehensive evaluation. We will work to include additional baselines in future versions.

Insufficient Theoretical Analysis

Our work leverages the theoretical foundation of truncated SVD decomposition and is supported by extensive empirical analysis. The key factor behind SParS’s ability to achieve significant compression lies in the fact that, for a given parameter budget, the rank of SVD is substantially higher in a sparse setup (see Figure 3b). This is further validated by the empirical results shown in Figure 3c. In this context, a detailed theoretical analysis of SParS would essentially delve into why a higher rank in the truncated SVD of ill-conditioned matrices leads to better reconstruction, which is already well studied in the literature.

Application Scope

To address your concerns about the limited evaluation, we have expanded our experiments to include:

• Another vision transformer baseline using SParS on SWIN-B (presented in Meta Rebuttal comment);

• Structured sparsity performance with DeiT-B;

• Structured sparsity latency results.

Structured sparsity-related results can be seen in the revised version of the paper. Moreover, per your suggestion, we are running SParS on the Gemma-2-2B model with the RedPajama dataset. However, we face computational restraints since LLMs require much more computational resources. Therefore, we will try to prepare results for this before the rebuttal deadline and add to "Meta Rebuttal" and revise the paper version.

We hope these responses and additional results effectively address your concerns. Thank you once again for your constructive feedback.

REFERENCES

[1] Yu, H., & Wu, J. (2023). Compressing Transformers: Features Are Low-Rank, but Weights Are Not! In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 9, pp. 11007–11015).

[2] Together Computer. (2023, October). RedPajama: An Open Dataset for Training Large Language Models. Retrieved from https://github.com/togethercomputer/RedPajama-Data

Dear reviewer,

Thank you for your time. Considering the ICLR review guidelines: "What is the significance of the work? Does it contribute new knowledge and sufficient value to the community? Note this does not necessarily require state-of-the-art results. Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc.)," we found it surprising that you found no strength in SParS. To the best of our knowledge, SParS is the first to combine and study in detail the interaction of parameter sharing, tensor decomposition, and sparsity. Further, through this method, we achieve significant improvements over comparable methods from the literature and successfully reach significant compression rates on two prominent vision transformers.

We have tried to address your concerns below:

Novelty

We respect your perspective. However, a similar criticism could, for example, also be applied to LoRA [1], which simply applies a well-known low-rank tensor decomposition approach to transfer learning. However, LoRA’s approach has been recognized for its innovative decomposition application and is not short of novelty. Similarly, while SParS can seem straightforward, we believe it introduces a novel approach to efficient sparsity and parameter sharing. It is the first approach to combine parameter sharing, tensor decomposition, and sparsity to achieve compression at a minimal cost, as demonstrated in the paper (e.g., compressing DeiT-B takes approximately 30 minutes on NVIDIA A100, see section 4).

Design Choices for U and V in MLP Modules

The rationale for sharing the base U across MLP modules while keeping V specific to each module is detailed in Section 2. In short, this choice is grounded in empirical results: sharing U globally and sparsifying V per module under our concatenation scheme yields optimal results for both zero-shot MSE of weight matrices and distilled reconstruction of weights. The reviewer can check this in Figure 2, where we show sparsifying larger matrices yield more optimal reconstruction, and concatenating MLP weights on a longer dimension yields a shared U. However… Singular values are not shared. Can the reviewer point out where their deduction comes from on the paper?

Additional Experiments

In response to your feedback, we have expanded our experimental results to include:

• Another vision transformer baseline using SParS on SWIN-B (presented in Meta Rebuttal comment);
• Structured sparsity performance with DeiT-B;
• Structured sparsity latency results.

Structured sparsity-related results can be seen in the revised version of the paper. Moreover, per your suggestion, we are running SParS on the Gemma-2-2B model with the RedPajama dataset. However, we face computational restraints since LLMs require much more computational resources. Therefore, we will try to prepare results for this before the rebuttal deadline and add to "Meta Rebuttal" and revise the paper version.

We hope these responses and additional results effectively address your concerns. Thank you once again for your constructive feedback.

REFERENCES

[1] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv. https://arxiv.org/abs/2106.09685

Reviewer can find the answer to the question on the inference time of Tables 1 and 2 below.

DeiT–B Inference Latency

SWIN–L Inference Latency

Considering this SParS out of the box utilizes theoretical sparsity, original model has better performance. But as stated in the previous comment, we have applied SParS using Structured Sparsity and performed an inference time analysis, where we see SParS improved latency requirements with <0.5% point loss in generalization. The results can be seen in the “Meta Rebuttal” comment.

Dear reviewer, thank you for your feedback and questions. Your input is valuable in helping us improve our work.

Writing

We have uploaded an updated version of the paper. This update includes improved captions, corrected typos, clarified contributions, and any additional improvements identified during this time. We hope these enhancements will address your concerns about the clarity and readability of our manuscript.

Method

• Focusing on the QKV layers could be an interesting direction. However, as the models get larger, MLP modules dominate the size, going beyond 70%. For example, in Gemma-2-9B, MLP parameters dominate 70.05% of the entire model parameter count. The percentage is higher for larger models, like LLaMa-3-70B.

• Compressed models initialized via local error minimization often show stronger generalization capabilities. Please see Table 2 on the paper.

• We are happy to hear that the reviewer found our method simple. However, we disagree that novel work needs to be complex; good research often involves distilling seemingly complex ideas into simple, robust approaches. SParS exemplifies this by integrating compression techniques across sparsity, tensor decomposition, and parameter sharing—a novel approach to the best of our knowledge. Moreover, evaluating how SParS’ simplicity compares to methods like LoRA [1], a novel and prominent paper in literature, remains essential.

Experiments

• Per your suggestion, we are running SParS on the Gemma-2-2B model with the RedPajama [2] dataset. We will try to prepare results for this before the rebuttal deadline. Moreover, we applied SParS on SWIN-B and used structured sparsity on DeiT-B to address your suggestion for expanded performance evaluation. We provided the corresponding performance and latency results on in the revised version of the paper. SWIN-B results are presented in the Meta Rebuttal comment.

• Regarding NAS and Distillation, NAS methods require significantly more computing and time and often do not leverage pre-trained models. Similarly, distillation relies on larger models for compression, adding to the computational burden. Given our computationally constrained situation, comparing SParS to these methods is not very practical.

• We also agree that computational budget and inference time are crucial considerations. As noted in the Experimental Setup within the Results section, SParS, without additional fine-tuning, completes training in approximately 30 minutes for DeiT-B and approximately one hour for SWIN-L on an NVIDIA A6000 GPU with the given hyperparameters. As for the inference time of Tables 1 and 2, we are attaching them in the next comment due to the character limit.

Thank you once again for your constructive feedback. We hope these updates address your concerns and illustrate the effectiveness and versatility of SParS.

REFERENCES

[1] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv. https://arxiv.org/abs/2106.09685

[2] Together Computer. (2023, October). RedPajama: An Open Dataset for Training Large Language Models. Retrieved from https://github.com/togethercomputer/RedPajama-Data

Thanks, authors. I have no further questions. I will wait for the responses from the other reviewers and make my final decision.

Dear reviewer, thank you. We greatly appreciate your feedback, which is instrumental in refining and advancing SParS. We have tried to address your concerns below:

1. We understand your concern regarding similarities with HydraLoRA [1]. However, there are two key differences: (i) HydraLoRA reduces LoRA [2] parameters but does not compress the original weights, as LoRA adapters are added to the existing weights, resulting in no reduction (but an increment) in the original parameter count. In contrast, SParS compresses the weights directly, significantly decreasing the overall model size. (ii) SParS demonstrates that sparsity is crucial for optimal compression, as shown in our results (see Figure 2 on the paper), where the dense baseline performs suboptimally compared to our sparse approach. These distinctions highlight the broader scope and effectiveness of our method.

2. You noted the limited comparison to AAFM, GFM [3], and Weight Multiplexing [4]. These methods are detailed in the literature review. Briefly: AAFM compresses model weights using truncated PCA on layer activations, GFM adds model output distillation to AAFM, and Weight Multiplexing shares weights between MLPs while learning them via distillation. In contrast, SParS shares existing parameters across layers, leveraging sparsity, with layer calibration via L2 loss minimization. The text thoroughly discusses this, and we believe adding a new paragraph to compare these methods would be a repetition.

3. We ran SParS on SWIN-B as requested by you and added the results below: | MLP Param. Budget (%) | ImageNet Top-1 Accuracy (%) | |----------------------|------------------| | 10% | 56.50% | | 25% | 81.67% | | 40% | 83.96% | | 50% | 84.27% | | Original SWIN–B (224) | 85.2% [2] |

4. Per your suggestion, we are running SParS on the Gemma-2-2B model with the RedPajama dataset. However, we are facing computational restraints regarding this since LLMs require much more computational resources. Therefore, we will try to prepare results for this before the rebuttal deadline. Moreover, we conducted additional experiments using structured sparsity, hopefully addressing your concern for limited generalization evaluation. We provided the corresponding performance and latency results in the revised version of the paper.

5. Block–wise compression $\mathcal{L} = \| \mathbf{UVx} - \mathbf{Wx} \|_2^2$ of SParS works analogously to compression at the activation level (see [6]), similar to the PCA-based compression that Yu and Wu (2023) – indeed this is how SParS can achieve such a strong compression rate.

References

[1] Tian, C., Shi, Z., Guo, Z., Li, L., & Xu, C. (2024). HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. arXiv. https://arxiv.org/abs/2404.19245

[2] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv. https://arxiv.org/abs/2106.09685

[3] Yu, H., & Wu, J. (2023). Compressing Transformers: Features Are Low-Rank, but Weights Are Not! In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 9, pp. 11007–11015).

[4] Zhang, J., Peng, H., Wu, K., et al. (2022). MiniViT: Compressing Vision Transformers with Weight Multiplexing. In Proceedings of the IEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12145–12154).

[5] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv. https://arxiv.org/abs/2103.14030

[6] Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv. https://arxiv.org/abs/2210.17323