Input Compensation for Pruned Models

Q3. It is important to note that the proposed IC method does introduce additional learnable parameters, which might not be considered when comparing the models’ sparsity levels. This could lead to an inaccurate estimation of the actual sparsity levels of both the original model and the proposed technique. Therefore, it is crucial to consider these extra parameters when evaluating the sparsity levels and overall performance of the models in question.

When computing the sparsity level, is the additional learnable parameters counted in the IC-related numbers?

A3. The compensation pool contains few parameters, only 2.3M. To address this concern, we re-calculate the number of non-zero parameters to include the pool. We also test baseline methods with a lower sparsity to evaluate whether our IC performs better than baseline methods when using the same number of non-zero parameters. The tables below show the testing accuracy of image classification when using ViT-B/32 and ViT-B/16. As can be seen, when using the same number of non-zero parameters, combining IC with existing pruning methods is better.

Testing accuracy on image classification using CLIP ViT-B/32.

Testing accuracy on image classification using CLIP ViT-B/16.

We sincerely thank the reviewer for the valuable comments on our work. We address the raised concerns as follows. If there are any remaining questions, please let us know and we are happy to address them.

---

Q1. The proposed method for Input Compensation (IC) has some limitations in terms of novelty. The primary idea is similar to weight compensation techniques, except it diverges from such approaches by focusing on input space. It is noteworthy that an IC method can be transformed into a weight compensation method, as the two strategies are "dual" to one another.

A1. Thanks for your insightful comments. Input compensation and weight compensation are different as compared below:

(i) Note that duality between input and weight space only holds for the linear layer, i.e., for an input compensation $\Delta X$, we can design a weight compensation $\Delta W\equiv X^{-1}\Delta X W$ (assume $X$ is invertible) such that: $(X+\Delta X)W= XW + \Delta X W=XW+XX^{-1}\Delta X W=X(W+X^{-1}\Delta X W)=X(W+\Delta W).$ However, for nonlinear models $\mathcal{F}(X;W)$, the duality does not always hold, i.e., for an input compensation $\Delta X$, we cannot find $\Delta W$ such that $\mathcal{F}(X+\Delta X; W) = \mathcal{F}(X; W+\Delta W).$

(ii) Weight compensation is input-independent; however, input compensation is input-dependent. Input-dependent compensation is more flexible.

(iii) For input compensation, we edit the input without modifying the model; thus, we can serve one shared model for all tasks. For weight compensation, however, we need to serve a task-specific model for each task. Hence, for multiple tasks, weight compensation needs much more serving resources than input compensation.

(iv) Input compensation is complementary to weight compensation. Thus, input compensation can be integrated into weight compensation, as shown by the successful applications of Magnitude + IC, Wanda + IC, and SparseGPT + IC.

---

Q2.

While the reported improvements seem to be promising, it is essential to consider their fairness in comparison to the baseline methods. The proposed IC method necessitates re-training and fine-tuning on the target dataset, which may introduce bias in the evaluation results if the baseline methods were not similarly fine-tuned. In such cases, it would be reasonable to conclude that the proposed technique's performance could potentially be attributed to the difference in fine-tuning processes rather than solely due to its inherent merits.

If the baseline models were not fine-tuned in the comparison, please report the fine-tuned version of these models / methods.

A2. In the paper (Lines 458-466), an experiment has been conducted to analyze whether input compensation is beneficial to the sparse models finetuned on the target dataset. As can be seen from Figure 4 of the paper, IC consistently brings large improvements to existing pruning methods when sparse retraining is applied.

We sincerely thank the reviewer for the valuable comments on our work. We address the raised concerns as follows. If there are any remaining questions, please let us know and we are happy to address them.

---

Q1. Similarity to Prompt Tuning: The method bears a resemblance to prompt tuning, with the primary difference being the integration manner (addition versus concatenation). A direct comparison between the proposed method and prompt tuning could clarify their distinct advantages and limitations.

A1. As discussed in Lines 127-129 of the paper, input compensation is different from prompting:

• Prompting inserts extra tokens into the input, but input compensation edits the input directly. Hence, no extra tokens are added. Note that adding tokens increases the computational cost.
• Prompts are input-independent; however, our designed input compensation is input-dependent. Note that input-dependent compensation is more flexible than input-independent prompting.

---

Q2. Experimental Scope: The experiments primarily compare the proposed method with post-training sparsity methods that do not involve backpropagation. This seems limiting, as the method aligns more closely with sparse LLM fine-tuning strategies. A comparison with similar methods that also leverage fine-tuning, such as DsNot[1], would provide a more comprehensive evaluation of its effectiveness.

A2. Thanks for your suggestions. We conducted an additional comparison between DSnoT [1] and DSnoT + IC. The table below shows the WikiText perplexity for the LLaMA model family with 50% sparsity. As can be seen, Wanda + DSnoT + IC consistently outperforms Wanda + DSnoT, validating that IC improves the performance of finetuned sparse LLMs.

---

Q3. Unclear foundation: Although it is interesting to model parameters as a low-rank matrix plus a sparse matrix, how exactly is the sparsity of ( S ) defined? Does it mean that all levels of sparsity (50%, 60%, 70%) are compatible with such a formulation? Furthermore, there are many possibilities for deciding sparse matrices. Can all types of sparse matrices be adequately represented by adding a low-rank matrix (as the proposed IC method)?

A3. We believe the reviewer comments on Lines 209-211, which says the weight matrix can be approximately decomposed into a sparse matrix $\mathbf{S}$ and a low-rank matrix. Which type of sparse matrix can be used to perform this decomposition is not the focus of this paper. As studied in existing works (Yu et al., 2017; Li et al., 2023b; Ding et al., 2023), a very high sparsity (e.g., 70%) leads to weak performance.

Note that this decomposition is only used to motivate our attention-based compensation mechanism in the linear layer and is NOT an assumption when applying input compensation to pruned models.

Q3. The proposed method is not clearly explained. More implementation details should be added.

A3. To address this concern, we clarify the implementation details below.

• For image classification experiments (Section 5.1), $\mathbf{v}_i$'s are learnable pixels on all sides (with the padding size 30). The compensation pool $\mathbf{K}$ and $\mathbf{V}$ are randomly initialized by standard normal distribution. We train the pool for 30 epochs using the SGD optimizer with a learning rate of 40 and momentum of 0.9, while the mini-batch size is 128. We use the supervised loss Eq.(5) and reuse the pruned image encoder as the encoder of IC.

• For NLP experiments (Section 5.2), $\mathbf{K}$ and $\mathbf{V}$ are learnable embeddings, which are randomly initialized by a normal distribution with zero mean and standard deviation 0.01. We train the pool for 20 epochs using the AdamW optimizer with a learning rate of 0.001 and a linear warmup schedule. The mini-batch size is 2. We use the reconstruction loss Eq.(7) and reuse the LLM's input embedding layer as the encoder of IC.

---

Q4. For CLIP experiments, the authors mentioned 'The compensation pool is shared across all ten tasks'. I suppose the pool is learned on one big dataset?

A4. Yes, we merge the training sets of all ten tasks together to learn the pool.

---

Q5. It would be better to provide the training time.

A5. The compensation pool is trained within hours in all experiments. Specifically,

• For image classification using ViT-B/32, it takes about 2 hours to train the compensation pool;
• For image classification using ViT-B/16, it takes about 7 hours to train the pool.
• For NLP tasks, it takes about 17, 38, and 40 minutes to train the pool for LLaMA-1 (7B), LLaMA-2 (7B) and LLaMA-3 (8B), respectively.

---

Q6. For NLP experiments, the authors claimed that they follow OWL[2] and focused on the case of 70% unstructured sparsity. While the base pruning methods may fail at 70% sparsity, it will be more fair to compare the case like 'wanda+owl' vs 'wanda+owl+IC'. Could IC be well combined with OWL?

A6. Thanks for your suggestions. To address this concern, we conducted an additional experiment following OWL to combine IC with Wanda+OWL （i.e., Wanda+OWL+IC). The table below shows the perplexity (smaller is better). As can be seen, IC consistently brings improvements to Wanda+OWL by a large margin of 5.0 on all three LLaMA family of models, verifying the effectiveness of combining IC with OWL.

We sincerely thank the reviewer for the valuable comments on our work. We address the raised concerns as follows. If there are any remaining questions, please let us know and we are happy to address them.

---

Q1. The major concern is the proposed method may introduce a significant amount of additional computation cost, resulting longer inference latency. For CLIP models, while the authors claim that the pruned image encoder is used as the encoder of IC, as illustrated in Fig.1, the proposed method has to run through an additional Encoder(ViT), which is time-consuming. The authors should add the comparisons of computation cost and inference speed.

A1. Thanks for the comments. We agree that running the encoder of IC introduces extra computation costs. However, the cost is low when the encoder is small. As summarized in the table below (image classification experiment with CLIP ViT-B/32), compared with Magnitude, the computation cost of Magnitude + IC increases by only 1% (from 305G to 309G FLOPs), and the inference speed drops by only 20% (from 665 to 535 images/second). Note that the testing accuracy increases largely (as shown in Table 1 of the paper), verifying that this additional computation cost is worthwhile.

---

Q2. For structured sparsity experiments, following [1], using 2:4 pattern instead of 4:8 pattern may be better, as advanced GPUs support 2:4 pruning acceleration.

A2. Thanks for your suggestions. We conducted additional image classification experiments using CLIP ViT-B/32 and ViT-B/16 to evaluate whether IC is useful for structured 2:4 sparsity. The tables below show the testing accuracy. We can see that IC consistently brings large improvements to existing pruning methods, demonstrating that IC is effective when using structured 2:4 sparsity.

Testing accuracy on image classification tasks using CLIP ViT-B/32.

Testing accuracy on image classification tasks using CLIP ViT-B/16.

We sincerely thank the reviewer for further comments. We are glad that our rebuttal has addressed all your initial concerns. For the newly raised concerns, we address them as follows.

---

Q7. It is not clear whether the accuracy can be guranteed when the decode is small.

A7. Note that our IC method does NOT contain a decoder.

---

Q8. Moreover, it is very strange that the accuracy without IC is pretty low.

A8. This observation (i.e., using pruning algorithms alone hurts performance) is consistent with existing works, e.g.,

• Tables 2, 21, and 23 of Wanda (ICLR 2024) show that Magnitude/SparseGPT/Wanda all have much lower testing accuracy than the dense model;

• Table 1 of SparseGPT (ICML 2023) and Table 3 of Wanda show that Magnitude/SparseGPT/Wanda have higher perplexity than the dense model.

Q6. It is stated in Sec 5.1 of the paper that the compensation pool is shared across all tasks. How does the compensation vary across different input types and model architectures?

A6. For different input types and model architectures, the compensations are always learnable $\mathbf{K}$ and $\mathbf{V}$. Compensation is model-agnostic and only depends on the input types: For an input with continuous values (e.g., images in Section 5.1), compensation is added to the input directly; For an input with discrete values (e.g., sentences in Section 5.2), compensation is added to its input embeddings.

---

Q7. In Table 4, only the result under 70% sparsity setting is reported. How's the performance of applying input compensation under the 50% sparsity setting? It would be better to directly compare with the result reported in the original Wanda paper for fair comparison.

A7. Thanks for your insightful suggestion. We conducted additional experiments to evaluate IC under the 50% sparsity accordingly. The table below shows the WikiText validation perplexity of the pruned LLaMA family of models (results with $^\dagger$ are reproduced by us, while results with $^\star$ are reported in the original publication). As can be seen, IC consistently brings a significant improvement to existing pruning methods, verifying the effectiveness of compensating inputs for pruned LLMs again.

Q4. The sparsity calculation needs clarification, especially considering the additional parameters introduced by the compensation pool and the encoder.

A4. Thanks for the valuable comments. Our proposed method reuses the submodule of the pruned model as the encoder and only requires a very small compensation pool as the additional parameters. For example, in image classification, IC reuses the pruned image encoder as the encoder and the compensation pool contains only 2.3M parameters. In natural language processing, IC reuses the input embedding layer of LLM as the encoder and designs a tiny compensation pool with 2.6M parameters.

To account for these additional pool parameters, we recalculate the number of non-zero parameters in IC. For a fair comparison, we also test the baseline methods using the same number of non-zero parameters. The tables below show the testing accuracy on image classification tasks using CLIP ViT-B/32 and ViT-B/16. As can be seen, IC outperforms all the baseline methods using the same number of non-zero parameters, validating the usefulness of our method again.

Testing accuracy on image classification using CLIP ViT-B/32.

Testing accuracy on image classification using CLIP ViT-B/16.

---

Q5. It would be better to list the detailed environment for better reproducibility including hardware and software.

A5. Our experiments are conducted on A100 (80G) with CUDA version 12.6. We list the detailed software environment (package==version) as follows.

accelerate==0.27.2
apex==0.9.10dev
bitsandbytes==0.41.1
debug_utils==1.0
evaluate==0.4.3
fairscale==0.4.13
huggingface_hub==0.23.4
intel_extension_for_pytorch==2.5.0
lm_eval==0.4.4
numpy==1.24.4
optuna==4.1.0
packaging==24.2
peft==0.11.1
pytorch_utils==0.5.5
ray==2.39.0
safetensors==0.4.5
timm==0.9.8
torch==2.1.2+cu118
torch_xla==2.5.1
torchvision==0.16.2+cu118
tqdm==4.66.1
transformers==4.46.2

We sincerely thank the reviewer for the valuable comments on our work. We address the raised concerns as follows. If there are any remaining questions, please let us know and we are happy to address them.

---

Q1. While the main goal of network pruning is to reduce computational costs, the introduction of an encoder and attention-based compensation mechanism appears to add more computational overhead during inference. It would be better for the authors to clearly quantify the additional computation and memory costs introduced by the compensation mechanism.

A1. Thanks for the comments. We agree that running the encoder of IC introduces extra computation costs. However, the cost is low when the encoder is small. As summarized in the table below (image classification experiment with CLIP ViT-B/32), compared with Magnitude, the computation cost of Magnitude + IC increases by only 1% (from 305G to 309G FLOPs) and inference speed drops by only 20% (from 665 to 535 images/second). Note that the testing accuracy increases largely (as shown in Table 1 of the paper), verifying that this additional computation cost is worthwhile.

---

Q2. The optimization process for the compensation pool needs a more detailed explanation, as the approaches described in Sections 5.1 and 5.2 seem quite different.

A2. To address this concern, we clarify the optimization details below.

• For image classification experiments (Section 5.1), $\mathbf{v}_i$'s are learnable pixels on all sides (with the padding size 30). The compensation pool $\mathbf{K}$ and $\mathbf{V}$ are randomly initialized by standard normal distribution. We train the pool for 30 epochs using the SGD optimizer with a learning rate of 40 and momentum of 0.9, while the mini-batch size is 128. We use the supervised loss Eq.(5) and reuse the pruned image encoder as the encoder of IC.

• For NLP experiments (Section 5.2), $\mathbf{K}$ and $\mathbf{V}$ are learnable embeddings, which are randomly initialized by a normal distribution with zero mean and standard deviation 0.01. We train the pool for 20 epochs using the AdamW optimizer with a learning rate of 0.001 and a linear warmup schedule. The mini-batch size is 2. We use the reconstruction loss Eq.(7) and reuse the LLM's input embedding layer as the encoder of IC.

---

Q3. The evaluation lacks testing on the widely adopted structured 2:4 sparsity setting, which is an important baseline for practical applications.

A3. Thanks for your suggestions. We conducted additional image classification experiments using CLIP ViT-B/32 and ViT-B/16 to evaluate whether IC is useful for structured 2:4 sparsity. The tables below show the testing accuracy. We can see that IC consistently brings large improvements to existing pruning methods, demonstrating that IC is effective when using structured 2:4 sparsity.

Testing accuracy on image classification tasks using CLIP ViT-B/32.

Testing accuracy on image classification tasks using CLIP ViT-B/16.