SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity

We sincerely appreciate the reviewer’s thoughtful feedback and recognition of our approach. Below, we address each of the concerns in detail:

"S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity" is a recent paper published in Arxiv in December. However, it is officially published in ICLR 25 after the submission deadline of ICML.

We would like to clarify that our paper already cites the S2FT work. S2FT introduces a structured pruning approach aimed at memory-efficient fine-tuning. According to Figure 5 in their paper, S2FT achieves a speedup of 1.1× over LoRA on LLaMA2-7B for the Commonsense 170K dataset. In contrast, our method achieves a 1.32× speedup on the same setting. While S2FT prioritizes memory efficiency and accuracy, SparseLoRA addresses a complementary direction: computational efficiency with lossless accuracy. We will include a more detailed discussion of S2FT in the final version of the paper.

I can not see why this method needs to be used together with LoRA. It would be great to use it independently and compare it with FFT.

We agree that extending our method beyond LoRA fine-tuning is an exciting direction. However, there are some practical considerations tied to the characteristics of LoRA. A key component of our method is the dynamic prediction of activation sparsity using an SVD-based predictor. This predictor relies on the static nature of the base layer weights—which, in LoRA, remain frozen during fine-tuning. In contrast, in full fine-tuning settings, the base weights are updated throughout training. This undermines the validity of the SVD-based predictor, potentially leading to inaccurate sparsity estimates. At this stage, our method leverages the frozen base layers in LoRA to reliably identify and exploit structured sparsity for speedup. Exploring how to adapt the predictor dynamically for full fine-tuning is an interesting direction we plan to pursue in future work.

While the computation speedup is clear, the memory analysis and comparison to the baselines are missing

Our approach uses LoRA for fine-tuning, so the memory profile remains the same as LoRA. Sparsifying the main branch does not affect memory usage.

Include comparisons with additional PEFT methods like APT and LoSA

APT and LoSA are designed to optimize inference-time sparsity rather than fine-tuning speed. Their methods typically slow down fine-tuning significantly to produce a sparse model with minimal accuracy loss at inference. For instance, LoSA requires 45.34 minutes compared to 13.78 minutes for LoRA, and under the Wanda setting, it takes 73.91 minutes versus 21.40 minutes for LoRA under SparseGPT, as shown in Table 8 of the original paper.

Include comparisons with additional PEFT methods like GaLore. Missing comparison to GaLore in terms of training time and GPU memory

We compare LoRA, GaLore, and SparseLoRA on Commonsense 170K and Math 10K. GaLore training requires A100 GPUs due to VRAM limitations of A6000 under DDP. Runtime is normalized to LoRA, as in our main submission. GaLore incurs a 1.58x training overhead but achieves similar performance to LoRA. The amortized time of GaLore, accounting for projection updates and online SVD, slows fine-tuning by 13.72x. While GaLore focuses on memory-efficient fine-tuning at the cost of computational efficiency, SparseLoRA accelerates fine-tuning with near-lossless performance.

LLaMA3-8B Commonsense 170K [on A100s]

LLaMA3-8B Math 10K [on A100s]

Include results on GLUE benchmark

Following the reviewer's recommendation, we extend LoRA and SparseLORA using Llama3-8B on the GLUE benchmark. SparseLoRA maintains competetive perofrmance on sequence classification accross various subsets of the GLUE benchmark.

LLaMA3-8B GLUE Benchmark

The proposed method doesn’t improve the finetuned model accuracy but only reduces the training time

Our method is designed to accelerate fine-tuning while preserving accuracy—not to improve accuracy over standard fine-tuning methods.

The GPU memory increase/decrease is not discussed

SparseLoRA improves computational efficiency during fine-tuning without altering the memory usage compared to baseline LoRA.

Specify SVD rank (k) values and sparsity levels used in experiments

We use an SVD Rank of 8 across all models and datasets, resulting in minimal runtime overhead (Table 3). Layer sparsity ratios are based on sensitivity analysis per model, with Llama2-7B's layer-wise analysis shown in Figure 7. The specific sparsity ratios employed are:

Model Sparsity Information

The selected layers and sparsity rates remain mostly consistent across models and datasets. Sensitivity analysis identifies layers unsuitable for sparsity (i.e. layers 20 and 24 in Llama2-7B). On Math 10k, layer ranges were increased to account for token splitting overhead. The sparsity assignment is latency-driven and informed by sensitivity analysis, ensuring no added overhead in SparseLoRA.

Inference Setup

SparseLoRA targets fine-tuning accleration and only applies sparsity during fine-tuning; inference remains unchanged compared to baseline LoRA.

Missing DoRA Results

Results for the DoRA method with LLaMA2-13B are missing due to OOM on A6000 GPUs without gradient checkpointing, as DoRA requires significantly more memory.

Template Issues

We will correct table captions and formatting in the revised version.

We appreciate the reviewer’s comments:

Fixed learning rate instead of hyperparameter sweeps.

For concision we only include mean performance. Here V1 refers to results in Table 1-2 of the paper and V2 is the new "conservative" config in our response to Reviewer as16

LLaMA3-8B LR Sweep (Math10K)

LLaMA3-8B LR Sweep (Commonsense 170K)

We conduct two learning rate sweeps with Llama3-8B on Math10K and CSR170K for LoRA and SparseLoRA. SparseLoRA shows minimal performance degradation compared to LoRA, confirming our preformance claims. In the conservative setting, the performance gap between the optimal LoRA and SparseLoRA is 0.2% on Math10K and 0.3% on CSR170K.

SparseLoRA is only evaluated on QKVO projections, but LoRA can be applied to different subsets of projections.

We focus on applying sparsity to speed up the main branch of the model, leaving the LoRA branches unchanged. However, we also provide additional experiments applying LoRA to Q, K, V, up, and down projections, following DoRA, in addition to the settings in our paper.

LLaMA3-8B Performance on Math10K for different projections

Provide iso-FLOP comparisons showing performance vs. computational budget

Following the reviewer’s recommendation, we conduct an iso-flop comparison on Llama3-8B using Math10K and CSR170K. SparseLoRA outperforms LoRA, with a 3% improvement at 5% FLOP budget on Math10K. Experiments run using 1 epoch as 100% FLOPs.

Iso-FLOP Comparison on Math10K with LLaMA3-8B

Iso-FLOP Comparison on Commonsense 170K with LLaMA3-8B

Lack of specifics like number of shots per task etc.

Tasks are evaluated with the mean over 5 shots. The SVD sparsity estimator is used during training to skip parts of the base-branch computation. The SVD computation is offline, but during training, marginal FLOPs are spent on the sparsity metric, accounting for 0.0534% of LoRA FLOPs and 0.8% runtime overhead, as shown in Table 3.

Details on SVD Predictor

We compute the SVD of weight matrices to create our sparsity estimator (Algorithm 1, lines 251-254) once per model. For the self-attention block, we SVD q, k, and v, and for the FFN, we compute up and gate projections, as described in Section 3.1. SVD is performed with rank 8 for all models. The predictors incur minimal runtime during training (Table 3) and negligible memory overhead (see below in Mb).

• Llama2-7B: 26.8
• Llama2-13B: 36.9
• Llama3-8B: 30.0

Motivation for computationally efficient PEFT

SparseLoRA achieves near-lossless performance while reducing computation. Even if PEFT tasks are fast, practitioners favor any free acceleration. For example, many opt for BF16 training over FP32 because, despite its slightly lower precision, the efficiency gains are substantial and the performance drop is negligible. Faster training means more rapid experimentation and ability to scale to larger models or more complex tasks—all of which are significant benefits in practice. SparseLoRA is a step toward this goal and can inspire future improvements in computationally efficient, lossless PEFT methods.

DoRA Settings

In this work, we use a rank of 32 for LoRA fine-tuning; DoRA has shown a smaller performance gap with this setting (Figure 5 in their work). S2FT reproduced similar results on Llama3-8B (arithmetic). Unlike DoRA—which prioritizes stability and performance at the expense of runtime—SparseLoRA enhances efficiency, complementing DoRA for practitioners.

Figure 1

We visualize the results directly from Table 1, specifically highlighting the normalized runtime and average performance on CSR170K for Llama2-7B.

"less than 0.5% overhead to finetuning" by SVD

We meant "0.8%" runtime overhead introduced by SVD estimator. Additional metrics in Table 3.

We sincerely appreciate the reviewer's comments.

L2 norm vs. Random pruning in Self-Attention blocks

L2 norm shows clear benefits for FFN blocks, while its gains over Random pruning in Self-Attention blocks are modest. We use a unified L2-based criterion to avoid over-engineering and ensure broad applicability.

Channel vs. Head Pruning:

Although the performance gap is small (80.2 vs. 80.1), channel pruning offers greater flexibility. Unlike head pruning—which enforces equal channel counts—channel pruning enables nuanced selection of important channels, allowing precise control over computational cost.

SVD Rank and Per-Layer Sparsity Ratios:

For all models, we use an SVD rank of 8, which incurs marginal overhead (see Table 3). Our per-layer sparsity ratios are determined via sensitivity analysis per model (see Figure 7). For the detailed configurations, please check "Model Sparsity Information table" in our response to Reviewer pTrd. Similar settings, with slight adjustments for latency, are applied across models.

Model Specification and Math10K Accuracies

All experiments in Section 4.3 use LLaMA3-8B. The Math10K results for LLaMA3-8B in Table 2 have 79.8% accuracy on average. This matches the result in line 2 of Table 3, line 2 of Table 4, and the “Non-uniform 1.4x” configuration in Table 5. In Table 6, higher accuracies result from applying sparsity only to specific modules rather than across the entire model.

Context-Output Aware Sparsity

Lines 263–264 describe our strategy as “selectively preserving dense computation for output tokens while applying sparsity to the context,” meaning only a portion of output tokens are processed densely, as reflected in Figure 8. We will revise the text to avoid potential ambiguity.

Table 5 Speedup Columns

The “1.4x Speedup” and “1.6x Speedup” columns indicate runtime targets by adjusting sparsity ratios. Both uniform and non-uniform configurations are calibrated to these targets for fair comparison. The 1.4x (1/0.71) non-uniform setting, with 79.8% accuracy on Math10K, directly corresponds to the SparseLoRA LLaMA3-8B result in Table 2; the 1.6x setting explores a more aggressive sparsity trade-off.

"Sequence averaging" in the SVD estimator

“Sequence averaging” reduces sparsity estimation cost by averaging activations over the sequence dimension for SVD inputs to compute a single channel score. This is particularly effective for FFN layers, where token-level differences are less critical, and it minimizes overhead while maintaining performance (see Table 3).

FLOPs and Runtime Optimizations in Tables 1-2.

The reductions reported in Tables 1 and 2 stem from applying contextual channel sparsity guided by our SVD estimator to expensive linear layers. Sequence averaging further minimizes the overhead of sparsity estimation, making the cost of identifying sparse channels negligible.

Trained to Convergence and Uniformity of Sparsity Ratios

No additional training was performed during the sensitivity analysis. As shown in Figure 7, we apply sparsity to the pretrained LLaMA2-7B model at individual layers (with others kept dense) and evaluate its performance on CSR170K. This analysis identifies robust sparsity patterns that are then uniformly applied across all models and tasks.

Comparison with Ma et al. (2024)

Ma et al.’s method differs from ours in many ways: it uses structured pruning with full fine-tuning, resulting in full-model memory usage. In contrast, SparseLoRA integrates structured pruning with LoRA adapters, reducing both computation and memory. Moreover, their importance metrics (Wanda, MaxiP) involve element-wise operations that incur high overhead (e.g., on A6000 GPUs), nearly canceling speedup gains when we experimented Wanda for Table 6. Our SVD-based sparsity estimator relies on efficient matrix multiplications with low-rank weights, ensuring minimal overhead across GPUs.

Performance drop not negligible

SparseLoRA offers a flexible speed–accuracy trade-off. For applications where maximum accuracy is crucial, LoRA remains viable; for efficiency-critical scenarios, SparseLoRA delivers substantial speedups with minimal accuracy loss. Here we provide an updated, more conservative sparsity configuration for LLaMA2-13B and LLaMA3-8B that slightly increase FLOPs (0.59 → 0.62 for LLaMA2-13B and 0.60 → 0.63 for LLaMA3-8B) but yield nearly identical runtime (0.74 → 0.76 for LLaMA2-13B and 0.71 for LLaMA3-8B) and match or improve accuracy:

These results show that, with proper configuration, SparseLoRA can achieve comparable or better accuracy than LoRA while delivering strong runtime benefits.