Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation

We thank the reviewer for recognizing the soundness of our work and providing thoughtful feedback. We address the reviewer’s concerns and suggestions below:

[Concern 1 - The sketch strategy is somehow hard to understand. Specifically, in Section 2.3, it is not very clear how the mapping matrix, i.e., M, is determined.]

Thank you for pointing this out. We provide further clarification below on how the mapping matrix $M$ is learned.

Each column of $M$ is a binary one-hot vector that maps an original parameter (from a row of size $c$) to one of the $k$ entries in the sketched parameter vector, where $k \ll c$. This mapping inevitably introduces some error in the model output. To minimize this error, we learn the columns of $M$ sequentially and iteratively update the remaining unmapped parameters to compensate for the introduced error after each step.

Concretely, for each original parameter, we identify the entry in the sketched parameter vector that is closest to it and assign the corresponding one-hot value in $M$. After fixing this column of $M$, we apply an update $\boldsymbol\delta$ (as defined in Equation 13) to the remaining unmapped parameters to absorb the approximation error. This process is repeated until all columns of $M$ are assigned.

We will clarify this procedure in the final version of the paper.

[W1 - Misleading Equation 6]

We thank the reviewer for pointing out this typo. We will correct this in the final paper.

[W2 - Release CUDA kernel as supplementary material]

Thank you for the great suggestion. We will include the CUDA kernel in the appendix and release our code and models for the final version.

Thank you for the thorough review and thoughtful feedback. Below, we address your questions and concerns.

[Theoretical Claims]

The effectiveness of LoRA and SketchTune depends on the structure of the true update matrix $\Delta$. If $\Delta$ is low-rank, LoRA is favored; if $\Delta$ aligns with the mapping matrix, SketchTune performs better.

Theorem 3.1 addresses how to choose between methods without detailed knowledge of $\Delta$. Assuming only the singular value distribution, the most neutral case is when $\Delta$ is uncorrelated with the mapping. Under this assumption, Theorem 3.1 shows that as $\Delta$ becomes less low-rank, SketchTune is increasingly likely to outperform LoRA.

We adopt the uncorrelated setting in Theorem 3.1 to avoid bias toward either method. Empirically, however, our analysis (Figure 1) shows that $\Delta$ is correlated with the mapping in a way that favors SketchTune.

[Experimental Designs 1 - More Data-Dependent Quantization + LoRA Baselines]

We performed additional comparison with data-dependent AQLM [4] quantization + LoRA baselines. We fine-tuned LoRA adapters with a 2-bit AQLM Llama-2-7B model on GSM8K using a learning rate of 1e-5 and a batch size of 16 for 5 epochs. We report the results in the table below. We are actively trying other hyperparameters to improve performance.

[W1 – Additional Data for Calibration & Sketching Overhead]

The sketching step is efficient, requires only a small calibration dataset, and is performed once per base model. Specifically, we use just 128 sequences of 2048 tokens, which is minimal compared to typical fine-tuning datasets. For further details and results, we kindly refer you to our response to Reviewer WZr9 under [W1].

[W2 - Sparse adapters are not thoroughly compared]

Tables 1 and 2 in our paper include comparisons with a sparsity-based method, S2FT. Additionally, while we aimed to compare with Diff-Pruning[1], we encountered a challenge: the official implementation does not support LLMs such as Llama. In the table below, we instead provide additional comparison with SpIEL[2] and SMT[3], two recent sparsity-based PEFT methods, reporting average accuracy across eight commonsense reasoning tasks.

[Q1 - Time/Compute Overhead]

We ran additional experiments to assess the impact of using smaller C4 subsets for calibration, evaluating model quality via WikiText2 perplexity and MMLU accuracy.

As shown in the table below, larger calibration sets improve model quality but do not significantly increase sketching time, since the dominant cost in sketching comes from clustering to obtain the $k$ centers (Equation 4).

These results highlight a trade-off: even a single sequence yields a usable sketch, but larger sets lead to better performance.

[Q2 - Sparse Adapters]

Yes, SketchTune can be combined with Diff-Pruning for greater parameter efficiency. This involves keeping the sketched parameters frozen and learning a sparse, task-specific update $\delta_\tau$, so that $w_{sketched, \tau} = w_{sketched} + \delta_\tau$.

Following Diff-Pruning, $\delta_\tau$ can be modeled as a gated update using a Hard-Concrete distribution to approximate a binary mask $z_\tau$, with $\delta_\tau = z_\tau \odot w_\tau$, where $w_\tau$ is the dense task-specific update.

Implementing this extension is non-trivial, and we defer it to future work.

[Q3 - Dynamic Remapping]

Since each sketched parameter is shared by multiple model parameters, re-mapping columns or recalculating the Hessian mid-training may help better align the sketch with the evolving gradients, potentially leading to improved final performance. Yet such an approach may introduce non-trivial computational overhead, as it would require additional clustering steps and second-order estimation during training. We defer a thorough investigation of dynamic remapping strategies to future work.

References

1. Parameter-efficient transfer learning with diff pruning
2. Scaling sparse fine-tuning to large language models
3. Sparse Matrix in Large Language Model Fine-tuning
4. Extreme compression of large language models via additive quantization

We thank the reviewer for their thoughtful feedback and for acknowledging the empirical effectiveness of our method. We address your concerns and questions below:

[W1 - Sketch generation process appears to be more expensive than LoRA]

While SketchTune introduces an additional sketching step before fine-tuning, this preprocessing is fast, resource-efficient, and one-time per base model. In the table below, we report additional end-to-end sketching time (INT4, GPR=4) for different sized models, using a single A100-40GB GPU. Thanks to our layer-wise optimization objective (Equation 1), sketching scales efficiently to large models (70B) with a single GPU.

The sketching overhead is comparable to existing compressed PEFT baselines. For example, LoftQ [1] reports a quantization overhead of 21 seconds for a 4096 $\times$ 4096 matrix, while SketchTune’s overhead is only 5 seconds (4.2x speedup).

Moreover, a single sketched model can be reused across multiple downstream tasks. As demonstrated in Tables 1 and 2 of the paper, the same sketched models are effective across different domains, i.e. math and commonsense reasoning. We also plan to release these models on the HuggingFace Model Hub, enabling users to directly download and fine-tune without repeating the sketching step.

[W2 - Learning separate sketches for each of the $g$ subgroups may reduce memory efficiency]

Our sketching approach is memory-efficient and effectively compresses the model weights. Specifically, each row $\mathbf{w} \in \mathbb{R}^{1 \times c}$ is divided into $g$ non-overlapping sub-rows $\mathbf{w}' \in \mathbb{R}^{1 \times \frac{c}{g}}$, each of which is sketched into a $k$-dimensional vector. The resulting sketched row $\mathbf{w}_{\text{sketched}} \in \mathbb{R}^{1 \times gk}$ leads to a compression factor of $\frac{c}{gk}$.

For Llama 2 7B, the row size $c$ is either 4096 or 11008. In our experiments, we set $k \in \\{4, 8, 16\\}$ and $g \in \\{1, 2, 4, 8\\}$, resulting in a compression factor of at least $32\times$ for Llama 2 7B. Notably, the compression factor increases with larger models, due to larger $c$ and fixed $k$ and $g$.

Thus, even with subgrouping, the sketching approach remains highly memory-efficient across model scales.

[Q1 - The derivation of (6) is not clear to me. Please explain how it is derived.]

Thank you for pointing this out. We realize that Equation (6) contains a typo: the derivative fractions are upside down. We apologize for the oversight and will correct this in the camera-ready version. The correct expression is:

$\frac{\partial \mathcal{L}}{\partial w_{\text{sketched}}} = \frac{\partial \mathcal{L}}{\partial y} (MX)^\top$

The equation is derived as follows. Let $X$ be the layer input, $w_{\text{sketched}}$ the sketched row, and $M$ the mapping matrix. The forward pass computes:

$y = w_{\text{sketched}}MX$

During backpropagation, we apply the chain rule to compute the gradient of the loss $\mathcal{L}$ with respect to $w_{\text{sketched}}$:

$\frac{\partial \mathcal{L}}{\partial w_{\text{sketched}}} = \frac{\partial \mathcal{L}}{\partial y} \cdot \frac{\partial y}{\partial w_{\text{sketched}}} = \frac{\partial \mathcal{L}}{\partial y} (MX)^\top$

We will clarify this derivation in the revised paper.

References

[1] Li, Yixiao, et al. "LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models." The Twelfth International Conference on Learning Representations.

We thank the reviewer for the detailed review and insightful feedback. We address your concerns as follows.

[Theoretical Claims 1 - Ambiguous Title in Section 2.1 & More in-depth Empirical Observation Analysis]

We thank the reviewer for the insightful suggestion. We conducted additional analysis to examine whether the weight updates in LLMs are close to full rank. In the table below, we report the minimum rank and the standard deviation, across all layers, required to explain a given percentage of the variance in the weight updates. This was computed by performing SVD on each weight update and counting how many top singular values are needed to reach the target variance threshold.

The results show that the weight updates are relatively high-rank. This suggests that standard LoRA configurations (e.g., rank 32) capture only a limited portion of the variance, and may not effectively approximate the full weight updates. We will incorporate this analysis into the final version of the paper to better align with the section title.

[Experimental Designs 1 – Additional Instruction-Following Benchmark]

We conducted additional experiments by instruction-tuning Mistral-7B on the Alpaca-GPT4 dataset for one epoch, following the experimental setup in S2FT. In the table below, we report MT-Bench scores evaluated by GPT-4, with baseline results taken from the S2FT paper. Despite using a smaller base model, SketchTune outperforms the baselines.

[Experimental Designs 2 – GPR Choice in Tables 1 & 2]

We reported results for GPR=4 in Table 2 due to the time-consuming nature of fine-tuning on the Commonsense170K dataset, which is 17× larger than the Math10K dataset used in Table 1. A single training run takes several days to complete. We selected GPR=4 as a practical trade-off between memory efficiency and final performance. We are happy to conduct additional experiments with other GPR values and include the results in the camera-ready version.

[Comments 1 – Overhead of Sketching Time and Memory Requirement]

In Table 9 of the paper, the reported sketching time refers to the time required to compress a base model into a sketched model using a single RTX 8000-48GB GPU. In the table below, we provide additional sketching time (INT4, GPR=4) and peak GPU memory usage for models of various sizes, using a single A100-40GB GPU. Thanks to the layer-wise optimization objective (Equation 1), model sketching scales efficiently to large models (e.g., 70B) on a single GPU. Additionally, we plan to upload sketched models to the HuggingFace Model Hub, enabling users to directly download and fine-tune without repeating the sketching step.