Enhancing Pruned Models by Input Compensation

Dear Reviewer QATu,

We sincerely thank you for your positive rating, thoughtful review, and valuable suggestions that have helped us improve our paper.

We have carefully addressed your concerns as follows. If you have any other concerns or questions, please let us know. We are more than happy to address them and further improve our work.

Best,

Authors

---

Q1. Discussion on research regarding input compensation outside the realm of pruning

A1. Thank you for this insightful suggestion. Input compensation (IC) has been studied in other domains, particularly in control systems and signal processing.

• Control Systems: As discussed in the last paragraph of Related Work, IC is a well-established technique in control theory [1-2] to adjust control signals for reducing the influence of potential disturbances.

• Signal Processing: IC is also relevant to Pre-emphasis in Signal Processing [3], which modifies input signals to counteract the effects of noise and attenuation that can occur in communication channels, especially in analog transmission systems.

We will discuss these connections more explicitly in our revised paper, which will help us better position our contribution within the landscape of input compensation techniques across different fields.

References

[1] Automatic control systems. 1995.

[2] Feedback control of dynamic systems. 2002.

[3] Pre-emphasis and speech recognition. 1995.

---

Q2. However, is it feasible to concurrently train the required K and V for "input compensation" during the pruning process itself? This aspect warrants a thorough discussion both methodologically and experimentally.

A2. Thank you for your insightful suggestion.

We conducted an ablation study (using ten image classification tasks and the CLIP ViT-B/32 model as in the Section 5.1) to explore this variant of IC. As pruning is a non-differentiable process, IC cannot be trained jointly with the pruning process directly. We instead study the performance of jointly training $(K,V)$ with the retained weights after pruning. In our study, we compare two approaches:

• Pruning $\to$ IC (separate training), where we first train the retained weights and then learn the $(K,V)$.
• Pruning $+$ IC (joint training), where we jointly train the retained weights and $(K,V)$.

Table R1 (https://www.dropbox.com/scl/fi/n3qcoi1ckwz4x2mlw0vbk/results.pdf?rlkey=8mt0tzz1f1ikzaa1hzuzns0lc&st=ldzypniu&dl=0) shows that the joint training consistently surpasses separate training across different sparsity patterns. This finding underscores the effectiveness of the joint training strategy, which allows for more cohesive optimization, leading to enhanced performance.

---

Q3. a discourse on the research pertaining to input compensation

A3. See our reply to Q1.

---

Q4. does IC also contribute to an improvement in the degree of sparsity?

A4. Thank you for your insightful question.

IC not only enhances performance at a given sparsity level but can also enable existing pruning methods to achieve higher sparsity without significant performance degradation. As shown in Figure 3, Magnitude+IC with 60% sparsity performs better than Magnitude with 50% sparsity, while Magnitude+IC with 50% sparsity performs comparable to Magnitude with 40% sparsity. This means IC effectively allows for an additional 10% of weights to be pruned while still maintaining higher/comparable performance.

Figure 3 also indicates that IC improves the tradeoff curve between sparsity and performance. For any target performance level, a model with IC can achieve that performance at a higher sparsity level than without IC. By enabling higher effective sparsity, IC allows practitioners to deploy smaller models without sacrificing as much performance as traditional pruning methods would require.

We will add the above discussion to the revised paper.

---

Q5. improve the diagram of the methodology (Figure 1)

A5. Thank you for this valuable suggestion. We have created a more comprehensive diagram (https://www.dropbox.com/scl/fi/n3qcoi1ckwz4x2mlw0vbk/results.pdf?rlkey=8mt0tzz1f1ikzaa1hzuzns0lc&st=ldzypniu&dl=0) with the following enhancements:

1. Detailed Attention Mechanism: The revised figure explicitly shows how the query-dependent compensation is generated from the query and how it interacts with the key-value pairs in the compensation pool through the attention mechanism.

2. Encoder is part of the pruned model: The revised figure explicitly shows that the encoder is part of the pruned model.

This enhanced diagram will be included in the revised paper to provide readers with a more intuitive understanding of our IC methodology.

Dear Reviewer aNPP,

Thank you for your time and effort in reviewing our paper.

We have carefully addressed your concerns and hope you are satisfied with our responses.

If you have any further questions/concerns, please let us know and we are more than happy to address them.

Best,

Authors

---

Q1. The claim regarding efficiency should be reconsidered.

A1. Thank you for this important point. We agree that pruning itself has limitations: Though pruning (especially unstructured pruning) reduces parameter count, it may not always improve hardware efficiency since some general-purpose devices cannot efficiently process sparse computations.

We will do the following to improve the writing:

• Remove the sentence "Quantization requires specialized hardware support, while distillation requires extensive retraining".
• Provide a more balanced discussion of different efficiency-focused methods.
• Add a limitation section to discuss the hardware compatibility issues.

We believe these revisions will present a more accurate view of efficiency-focused methods.

---

Q2. The assertion that the proposed method is orthogonal to existing approaches may not be entirely accurate

The rationale behind the effectiveness of input compensation remains unclear.

input compensation does not appear to be truly "orthogonal" to existing methods but rather a variant that modifies the optimization parameters.

Since input compensation is a variant of weight compensation, the authors should provide an error analysis for pruning to justify its effectiveness, rather than solely reporting accuracy improvements.

A2. We believe there is a misunderstanding regarding orthogonality. When we say IC is "orthogonal" to existing pruning methods, we mean:

• Different Spaces: IC operates in the input space by adjusting inputs, while existing methods based on weight compensation operate in the parameter space by adjusting weights.
• Complementary Approaches: Because they work in different spaces, IC can be combined with existing pruning methods (which are based on weight compensation) to improve their performance. Hence, IC is not a variant of weight compensation.

Testing Accuracy is a Reasonable Metric. Note that both IC and weight compensation fundamentally aim to minimize the output deviation caused by weight removal. Testing accuracy directly quantifies this deviation and is the standard evaluation metric in the pruning literature. Hence, we believe testing accuracy is a reasonable metric to reflect the pruning performance.

Effectiveness of IC. Since IC and weight compensation are complementary approaches, the effectiveness of IC can be verified by comparing "weight compensation + IC" against "weight compensation alone." Our extensive experiments (Tables 1-5) consistently show that SparseGPT+IC outperforms SparseGPT (a weight compensation method) across various sparsity patterns and tasks, confirming the effectiveness of IC.

---

Q3. fairness of the comparisons

A3. Thank you for this concern.

We would like to claim that the comparison is fair due to the following reasons:

• Careful Hyperparameter Tuning: (i) Magnitude and Wanda have no hyperparameters. (ii) We have carefully tuned the hyperparameters of SparseGPT specifically for the vision models. Indeed, the performance of SparseGPT is insensitive to hyperparameters (Hessian dampening, and mask selection blocksize).

• Wanda and SparseGPT are Competitive: Note that our CV experiments are based on ViT models, which are transformer like LLMs. Hence, Wanda and SparseGPT, which are initially designed for LLMs, are still SOTA baselines for CV tasks. As shown in Tables 1 and 2, Wanda and SparseGPT achieve much higher accuracy than Magnitude Pruning, thus, are very competitive.

• Comprehensive Evaluation Across Domains: Besides CV tasks, we also extensively evaluated our method on NLP tasks (Tables 3 and 4), which shows that IC consistently improves performance across both CV and NLP domains.

In the revised paper, we will include the above discussion to address this concern.

---

Q4. computation cost of the encoder.

A4. Thank you for this question.

We have discussed the computation cost in our submission (last paragraph, Page 7). Our method increases FLOPs by only 1% (from 305G to 309G) compared to existing pruning methods. For a detailed breakdown, we provide Table R1 (https://www.dropbox.com/scl/fi/le76r4j7kdq9ixvzzd0rk/results_flops.pdf?rlkey=75b98wu5xaoe3hkqy9mnik6ut&st=cy9k8676&dl=0) to compare Magnitude and Magnitude+IC FLOPs by module. Given the substantial performance improvements (up to 33.9% accuracy increase in Table 1), this minor computational overhead from constructing compensation is worthwhile.

We will include this detailed analysis in the revised paper.

Dear Reviewer MNqZ,

We sincerely thank you for your positive rating, thoughtful review, and valuable suggestions that have helped us improve our paper. We have carefully addressed your concerns as follows.

If you have any other concerns or questions, please let us know. We are more than happy to address them and further improve our work.

Best,

Authors

---

Q1. It would be nice if the authors could have theoretical claims for the proposed framework.

A1. Thank you for your valuable suggestion. We agree that strengthening the theoretical foundations would enhance our paper. We have provided a theoretical analysis for linear models in Section 4.1, where we show the duality between input compensation and weight compensation:

For a linear layer with output $Y = XW$, we demonstrate that if the weight matrix $W$ can be approximated as $W \approx S + AB^\top$, where $S$ is a sparse matrix and $AB^\top$ is a low-rank matrix. This leads to $Y = XW \approx X(S + AB^\top) = XS + (X + XA\hat{B}^\top)S,$ where $\hat{B}\equiv B^\top(S^\top S)^{-1}S^\top$. This equivalence shows that adjusting the input (adding $XA\hat{B}^\top$ to $X$) can have similar effects as adjusting the weights (adding $AB^\top$ to $S$) in linear models.

In the revised paper, we will strengthen our theoretical analysis by providing a formal theorem and proof for the equivalence between input compensation and weight compensation in linear models. However, extending this analysis to non-linear models is non-trivial and requires additional theoretical work. We leave this as a future research direction.

---

Q2. It would be nice if the authors could test their method on large-scale image classification datasets.

A2. We are grateful for your valuable suggestion. We conducted additional experiments on ImageNet, a large-scale image classification dataset with 1,000 classes and over 1.2 million training images. Table R1 (https://www.dropbox.com/scl/fi/8srj7o2yacxmxtg9j75j7/results_on_imagenet.pdf?rlkey=krue8mns0ues2ns9uulcxlnae&st=fhqqqoj5&dl=0) compares the testing accuracy of different methods using CLIP ViT-B/32 . As can be seen, our IC method consistently increases the testing accuracy of existing pruning methods across different sparsity patterns on ImageNet:

• At unstructured sparsity (50%), IC improves Magnitude Pruning from 19.6% to 41.0% accuracy (+21.4%), Wanda from 38.3% to 51.0% (+12.7%), and SparseGPT from 47.7% to 52.9% (+5.2%).

• For the more challenging structured sparsity patterns (4:8 and 2:4), IC shows even more significant improvements. For example, with the 2:4 pattern, IC improves Magnitude Pruning from 7.2% to 32.5% (+25.3%), Wanda from 11.6% to 36.1% (+24.5%), and SparseGPT from 36.0% to 48.7% (+12.7%).

These results on ImageNet further validate that our IC method effectively enhances model performance on large-scale image classification tasks.

Dear Reviewer Wf6n,

We sincerely thank you for your thoughtful review and valuable suggestions that enhanced our paper.

We have carefully addressed your concerns as follows. If you have any other concerns or questions, please let us know. We are more than happy to address them and further improve our work.

Best,

Authors

---

Q1. No theoretical guarantee for the effectiveness of IC in general cases.

A1. Thank you for your valuable question. We agree that our IC method has a limitation that lacks a theoretical guarantee for non-linear models. Though we have provided a theoretical analysis for linear models in our paper, extending this analysis to non-linear models remains a challenging open problem that we leave as a future research direction.

Despite this theoretical limitation, our approach follows the practical tradition of many successful pruning methods that are primarily heuristic in nature. The empirical results across various tasks (image classification, language modeling, and image generation) demonstrate that IC effectively boosts the performance of pruned models in practice. These consistent performance improvements across different model architectures and tasks provide strong evidence for the practical utility of our method, even in the absence of complete theoretical guarantees for non-linear cases.

---

Q2. Could you provide details on the fine-tuning process for these methods, including the number of epochs and datasets used?

A2. Thank you for your question about the fine-tuning process (i.e., sparse retraining). For all methods, we followed a consistent fine-tuning protocol to ensure fair comparison:

• Datasets: The fine-tuning is performed on the same training datasets (i.e., the ten datasets in Section 5.1) used for learning the IC.

• Number of epochs: We retrain the retained parameters for 3 epochs for all methods. We observed that performance typically saturates after 2 epochs, with minimal gains from additional training.

• Optimizer: For all methods, we adopt the AdamW optimizer with a learning rate of 0.000001 and weight decay of 0.01, and a batch size of 128.

This consistent protocol ensures that any performance improvements observed when combining pruning methods with IC can be attributed to the effectiveness of our approach rather than differences in the fine-tuning process.

---

Q3. In the ViT experiments, (Q3-A) was ViT fine-tuned separately for each subtask? (Q3-B) Additionally, was a single IC trained across all subtasks, or was a unique IC trained for each subtask?

A3. Thank you for your questions.

(Q3-A) No, the ViT model was not fine-tuned separately for each subtask. Instead, we fine-tuned a single ViT model across all ten subtasks. This multi-task approach is more parameter-efficient as we maintain only one model instead of ten separate models and better reflects real-world deployment scenarios where a single pruned model needs to handle various tasks.

(Q3-B) Yes, a single IC was trained across all subtasks. All subtasks share the same compensation pool, which provides two benefits:

• Parameter efficiency: Using a shared compensation pool requires much fewer parameters compared with training separate ICs for each subtask.
• Knowledge sharing: As shown in Figure 6, different subtasks can share the same components of the compensation pool, demonstrating effective knowledge sharing.

---

Q4. However, it remains unclear whether this approach provides an advantage over pruning combined with LoRA fine-tuning. To clarify this, it would be beneficial to conduct experiments using LoRA with a comparable number of additional parameters and directly compare its effectiveness with IC.

A4. Thank you for your insightful suggestion.

To address your concern, we conducted additional language modeling experiments to compare IC with LoRA fine-tuning on pruned LLMs using approximately the same number of additional parameters as our IC method.

For LoRA implementation, we use a rank of 16 for both LLaMA-1 and LLaMA-2. To maintain the same number of parameters (only 262K) as our IC method, LoRA is applied only to the first layer of the LLM, which our ablation studies showed to be the most effective configuration.

Table R1 (https://www.dropbox.com/scl/fi/x9ni5gmmetxkog2dslsac/results_lora.pdf?rlkey=jdzet17nezcsoxtrfxbyr888g&st=1pyx8l6t&dl=0) shows that IC consistently outperforms LoRA when combined with various pruning methods (Magnitude Pruning, Wanda, and SparseGPT) across both LLaMA-1 and LLaMA-2 models with the same parameter budget.

This suggests that, when using extremely few trainable parameters, IC is more effective than LoRA. We will add this experiment to the revised paper.

---

Q5. the advantage of IC over pruning combined with LoRA remains unclear.

A5. See our reply to Q4.