SP-LoRA: Sparsity-Preserved Low-Rank Adaptation for Sparse Large Language Model

-My major concern is about the research contribution of this paper. Although the techniques such as memory reutilization are effective for memory-efficient sparsity preserved training, they seem to be general solutions to this kind of problem, thus making this work more of an engineering contribution rather than research one. Maybe a more detailed explanation of the challenges of combining all techniques together can make this work better.

-A better baseline is needed to show the Sec 3.2.1 memory complexity. As kernel fusion or in-place computation are widely used in famous frameworks such as FasterTransformer, it seems to be a poor baseline shown in Formulation 3.

-In Figure 1, SPP(GC) has similar memory usage with SP-lora, although slower than SP-lora, but only 2 seconds per batch. As the number of batches needed for fine-tuning is generally small, SPP(GC) seems to be fast enough for fine-tuning. Therefore, the contribution of SP-lora over SOTA method SPP(GC) seems to be limited.

Although the SP-LoRA method has the above strengths, I think it includes but is not limited to the following weaknesses:

1. Lack of novelty. Sparsity-Preserved Parameter-Efficient Fine-Tuning (SPP) for LLMs has already been proposed, and the Sparsity-Preserved Low-Rank Adaptation (SP-LoRA) introduced here is not novel and lacks groundbreaking contributions.
2. Limited contribution. The author proposes using gradient checkpointing and memory reuse techniques to optimize SP-LoRA’s memory overhead and fine-tuning efficiency. These are well-established techniques implemented using PyTorch APIs. The author merely combines existing techniques, making SP-LoRA appear more like an engineering improvement.
3. Accuracy is worse than or only slightly improved over baselines. The data in Tables 1-5 shows that SP-LoRA’s accuracy is much lower than that of the LoRA and SPP baselines or only offers a minimal accuracy improvement. Although SP-LoRA has memory and time advantages over the baselines, its disadvantage in accuracy makes me very concerned about the effectiveness of SP-LoRA.
4. 2:4 sparsity is not enough. The author only conducted experiments with 2:4 semi-structured sparsity, which is insufficient to validate the effectiveness of the SP-LoRA method. For example, the author should provide more experimental data with 2:8 sparsity. Additionally, there is a lack of experimental data with 50% or 60% unstructured sparsity, as commonly set in methods like Wanda and SparseGPT.
5. Experiments are conducted only on LLaMA. The authors only conducted experiments on the LLaMA family and lacked verification of the effectiveness of SP-LoRA method on models with non-LLaMA architecture (such as OPT model [1]).

Due to SP-LoRA's lack of novelty, limited contributions, lower accuracy compared to the baselines, and insufficient experimentation, I believe this paper is not yet ready to be accepted. I would be happy to discuss my evaluation with the author and look forward to their response.

[1] Opt: Open pre-trained transformer language models. Arxiv.

While the approach that the authors propose is straightforward, it comes with a high memory overhead compared to the baselines. This is not necessarily an issue in itself since much of the paper is devoted to addressing this memory problem. However, since it is the SP-LoRA method that introduces the memory overhead in the first place, I would expect SP-LoRA not just to match but outperform the baselines in some respect. In other words, the memory cost that the masking introduces should be worth the trouble on some other metric. As far as I can tell, SP-LoRA performs similarly to LoRA and SPP. In particular, the memory cost after optimization appears to be the same (Figure 1). Consider including an experiment that shows the inference memory usage of the resulting sparse SP-LoRA network compared to the baselines. Given that LoRA does not preserve sparsity, SP-LoRA may show significant advantages at inference time that could justify the memory overhead during fine-tuning. Alternatively, consider exploring scenarios where preserving sparsity might be particularly beneficial, like when scaling to larger model sizes. Evaluating the memory usage for growing model sizes could reveal a favorable trade-off that could justify using SP-LoRA over the baselines as model size grows.

Additionally, Figure 1 suggests that SP-LoRA reduces time usage compared to SPP. Consider adding a more detailed analysis of the time usage differences across different model sizes or sparsity levels. This would help clarify whether the time advantage is consistent and significant. If there is a general advantage independent of implementation details, it would provide another reason to use SP-LoRA over SPP. Furthermore, Appendix A contains a valuable analysis of the specific differences between LoRA and SPP. It would be worth exploring these differences in an ablation study that compares the usage of a composite matrix vs a standard matrix and the usage of mask vs additional weight term. The study could evaluate training time, memory usage, and task performance across different sparsity levels and thus provide a clearer picture of the trade-offs involved in sparse training when using these two methods.

Suggestions to improve the clarity of the paper:

• The introduction is written with the assumption that the reader is familiar with LoRA's definitions and terminology, which, however, are not introduced until Section 3.1. To make the introduction more accessible to a broader audience, consider giving readers a brief idea of LoRA and the extension this work proposes.
• L73-L78+L82-L87: I appreciate the effort to formulate a concise overview of the method in the introduction, but this early on in the paper without any preliminaries in place, I found it hard to follow and diminished the clarity of the introduction. Consider focusing on the key idea behind your work in the introduction, deferring the mathematical details to Section 3.
• L464 To aid the reader, could you please reference which experimental outcomes specifically show this improvement
• For the Tables:
  • Consider highlighting the best average scores in bold.
  • There is a typo 'Mehtod' in Table 1 and 3.
  • Instead of denoting the sparsity type 2:4 that is the same for all methods, consider listing the weight sparsity levels and non-zero weight count.

1. The manuscript does not discuss any specific kernel implementations or hardware-level optimizations for SP-LoRA, which could limit its practical efficiency on specialized hardware.
2. The evaluations appear to be primarily conducted on smaller to medium-sized LLMs. It's unclear how SP-LoRA would scale to very large language models with hundreds of billions of parameters.
3. The method is mainly tested on models pruned by Wanda or SparseGPT, which may not cover all possible sparsity patterns. It's uncertain how SP-LoRA would perform with other sparsity types or pruning methods.
4. The manuscript could benefit from more extensive comparisons with a wider range of models and other parameter-efficient fine-tuning techniques beyond LoRA and SPP.
5. While the manuscript mentions evaluations on domain-specific tasks like math and code, the results presented for these tasks may not be comprehensive enough to fully demonstrate SP-LoRA's capabilities across various domains.
6. Although memory usage is optimized, the method may still introduce some computational overhead compared to standard LoRA, which could impact training time.