Compression for Better: A General and Loss-Driven Compression Framework

Q1: About convolution representation.

A1:

1. The convolution layer can be expressed in the first-order differential form[2]. This first-order approximation is applicable to any differentiable operation unit (including conv layers, etc.), and its effectiveness comes from the differentiability assumption of neural networks[1]. The following is a simple proof,

Assume that the output $h_i$ of a conv layer is defined by the convolution:
$h_i = (w * x)_i =\sum_j w[i - j]x_j$

where $x$ is the network input, $w[i - j]$ represents the convolution kernel weights, and we introduce a new index $j$ to denote the input position corresponding to kernel computation. $h_i$ is the activation at position $i$. Suppose the input $x$ is perturbed by a small amount $\delta$, i.e., $x \rightarrow x + \delta$.

Then, the new convolution output becomes:
$\tilde{h}_i = (w * (x+\delta))_i = \sum_j w[i - j]\Bigl(x_j + \delta_j\Bigr)$

Due to the linearity of convolution, we can split the above sum:
$\tilde{h}_i = \sum_j w[i - j]x_j +\sum_j w[i - j]\delta_j = h_i + (w * \delta)_i$

Thus, the perturbation is:
$\Delta h_i = \tilde{h}_i - h_i = (w * \delta)_i$

To be consistent with the notation in the paper, we denote:
$\delta_i = (w * \delta)_i$

This means that the input perturbation $\delta$ is weighted by the kernel $w$ to yield the output perturbation $\delta_i$. Assuming the loss function $l_k$ is differentiable with respect to the activation $h_i$, a first-order expansion gives:
$l_k(h_i + \Delta h_i) \approx l_k(h_i) + \frac{\partial l_k}{\partial h_i}\Delta h_i$

Replacing $\Delta h_i$ with $\delta_i$ yields: $\Delta l_k \approx \frac{\partial l_k}{\partial h_i} \cdot \delta_i$

For a detailed derivation and differentiation, please refer to https://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/.

[1]Goodfellow I et al. Deep learning[M]. Cambridge: MIT press, 2016.

[2]Dumoulin V. A guide to convolution arithmetic for deep learning[J]. arXiv preprint 2016.

Q2: About accumulation joint impact.

A2: The accumulation of layer-wise errors comes from the off-diagonal elements of the Hessian matrix in the second-order terms. Through extensive experimentation, in the first-order neighborhood, we have found that the impact of the second-order terms on the loss is smaller than $10^{-5}$, and the layer-wise accumulated error is similarly smaller than $10^{-5}$.

Because the computation of second-order terms is time-consuming, LLC neglects these terms. The joint impact and the summation of multiple errors that you referred to are typically results when the second-order terms dominate, which is different from LLC’s approach.

Q3: About the calibration and drop-rate.

A3: The calibration dataset is completely different from the test dataset. We follow the existing mixed precision quantization work and keep the calibration set consistent with the existing methods[Zhang et.al]. Drop-rate refers to the compression rate of the model weights.

Q4: About the error boundary.

A4: After extensive experiments, we have found it is less common for layers quantized to 2 bits. 2-bit quantization does not always satisfy the gradient-based error bound. As shown in Fig 1, when the error caused by ultra-low bit is large, the first-order term still dominates, but the error introduced by the second-order term is non-negligible ($10^{-2}$), it may not ensure lossless compression. This issue has been addressed in the LLC limitation section. Therefore, in such cases, it is necessary to additionally analyze the impact of the high-order term on the model, which is also the next research direction of LLC.

Since the purpose of LLC is to ensure losslessness and select a suitable bit width for the layer, if LLC cannot ensure lossless compression under ultra-low bits, it usually increases the bit width (2bit->4bit) to minimize the loss. The experimental results are detailed in Reviewer 7rwL A2.

Q5:About noise.

A5:$scale_{input}$ and $scale_{weight}$ are the scaling factors used for the FP to Int conversion. In Alg 1, the role of the noise is twofold:

(a)During the calibration process, the noise is used to compute the magnitude of quantization errors to ensure that they remain within the differential neighborhood;

(b)During the quantization process, the noise represents the quantization error and must remain opposite to the gradient direction.

We have updated the algorithm and clarified the symbols.

Q6: About configuration.

A6: LLC adopts a layer-wise quantization approach, where all weights in a given layer use the same bit-width. This is consistent with existing work. For specific configurations of bit-width and rank, we plan to gradually release the code to provide more information.

We have updated the paper as you suggested. However, due to ICML's rebuttal policy, the updated paper will be presented later. I hope the reply can address your concerns and look forward to your feedback.

Q1: About Smoothness Condition.

A1: Except at the zero point, ReLU is smooth and differentiable in most regions; its non-differentiability occurs only when the activation value is exactly zero. Since real-world datasets (e.g., ImageNet) typically feature continuous distributions, the activations input to ReLU are also mostly continuous real numbers. This implies that the probability of activations being exactly zero is extremely low, and thus the piecewise nature of ReLU has a negligible impact on the overall optimization process. Therefore, ReLU does not invalidate the overall smoothness assumption, and the theoretical derivation of LLC remains valid.

Bengio, Y. On the Number of Linear Regions of Deep Neural Networks. [NIPS2014]

Q2: About the correlation of layers.

A2: In Equation 5, we have accounted for the inter-layer perturbation correlations, mathematically manifested as the effect of the off-diagonal elements of the second-order Hessian matrix (denoted as $H_{i,j} = \frac{\partial^2 f}{\partial x_i \partial x_j}$). This term reveals some dependency among layers.

Q3: About limited loss improvement.

A3:

1. First of all, from the perspective of the improvement, the loss improvement is competitive. For example, in ResNet18, as shown in the following table, the data in the table represents the improvement compared with the original model after compression, and the larger the value, the better. HAWQ has a smaller improvement over LLC. The improvement compared with other methods is also competitive. ResNet18|Acc. Drop$\uparrow$|Loss. Drop$\uparrow$ -|-|- HAWQ|-0.21%|-0.74% ACIQ|-0.14%|-0.22% LLC|-0.02%| +0.13%

2. You mentioned that smaller loss does not always seem to lead to better performance. This is because you have already believed that loss (cross entropy, sequence cross entropy, AUC error, etc.) cannot represent the performance of the model. But in fact, cross entropy or accuracy are all performance indicators that can evaluate the model.

You agree with LLC's theoretical framework for loss. As you mentioned, loss is the main criterion in the experiment. Therefore, in LLC's experiments, loss (cross entropy, sequence cross entropy, AUC error, etc.) is mainly used to measure the performance of the model in different tasks. For more discussion on loss, please refer to the answer to m1Ar reviewer. Looking forward to your discussion with us.

Q4: About calibration set.

A4: Following existing pipelines, we use the ImageNet validation set to assess the noise boundary and the test set for performance evaluation. Since the validation and test sets contain entirely different data, the calibration set is used solely for computing quantization parameters and error boundaries, thus avoiding overfitting to the calibration data. All experimental results in the paper are based on the test set.

Q5: About the noise boundary.

A5: The data source of Figure 1 is calculated by Eq. 6. In the early stage, we conducted a lot of experiments to calculate the compression boundary. That is, control the size of the introduced noise, measure the applicable range of the differential expansion, and observe the change in loss. The purpose is to analyze the relationship between the introduced noise and the higher-order terms. For example, when the error introduced by the weight is less than 8e-3, the first-order gradient is the main factor affecting the compression performance. Specifically, in this experiment, we mainly perturb the ResNet18, ResNet34, ResNet50, Bert and other models with different error sizes. After the perturbation, different error ranges for the first-order and second-order terms are obtained.

The calculation of the noise boundary of the second-order term is complicated. We use the Lanzcos algorithm to restore the second-order Hessian term as much as possible. The calculation time takes several days and the cost is high. However, after repeated experiments, we found that the impact of the second-order term is small within the range of $8\times10^{-3}$, and the time cost is large. Therefore, LLC ignores the calculation of the second-order term and focuses on the first-order term. At the same time, LLC calculates the cost of the noise boundary of the first-order term is extremely small, which takes only about 3-4 minutes. The experimental hardware uses NVIDIA A800 GPU.

We show the cost of the quantization experiment in the table below. LLC has higher efficiency in both quantization and decomposition.

We have added the above experimental results to the original text and added a description of the noise boundary in Figure 1 based on your comments. Due to the rebuttal policy of ICML, we are currently unable to display the original paper publicly. This paper will be provided in subsequent stages.

Thank you for your suggestions and hope that our answer can address your concerns. Looking forward to your feedback.

Q1:About when LLC cannot reduce loss

A1:Regarding the limitations of LLC, we emphasize that the loss reduction achieved by LLC is not exaggerated but based on strictly defined mathematical conditions. This sentence has been mentioned in the advantages of the paper you commented on: LLC has clear mathematical conditions.

As described in the paper, LLC can reduce the loss of the model in the differential neighborhood; in section 3, we discussed the effective conditions in the noise neighborhood mapping, which require that the perturbation is small enough to ensure the effectiveness of the total differential expansion. Only in such a differential neighborhood can LLC guarantee the reduction of loss. Once the perturbation exceeds this neighborhood, the influence of high-order terms becomes significant, and LLC can no longer guarantee the reduction of loss, making the total differential analysis no longer applicable, and further analysis of high-order terms is required. Therefore, the effective conditions of LLC are the effective conditions of the total differential expansion. When the significant error caused by extreme compression is introduced, the total differential analysis is difficult to apply, and LLC cannot maintain the loss, which is also mentioned in the limitations of the paper.

Q2:About extreme compression and sensitivity analysis

A2:Our algorithm is still based on mixed-precision compression, and we use quantization as an example for explanation. First, the purpose of LLC is to find the bit width that can reduce the model loss, so the bit width selected is different according to the tolerance of the layer. LLC is best-effort, that is, when the layer can be extremely compressed (meeting the neighborhood condition), LLC will compress it to ultra-low bits. Taking VGG13 as an example, the following table shows the loss changes of LLC for quantization of the 4th and 8th layers,

The error caused by 1bit significantly exceeds the neighborhood range of these two layers, so 1bit compression cannot be performed. LLC will increase the bit width to 2bit compression. At this time, the eighth layer meets the neighborhood condition and the loss is reduced. However, the fourth layer still exceeds the neighborhood range, resulting in increased loss. Therefore, 4bit compression is finally selected for this layer. This example demonstrates that layers with larger error neighborhoods benefit most from LLC.

Secondly, LLC defines "aggressive compression" based on the impact of low-bit errors on the model's loss. Since different layers have varying tolerances to compression, when the error introduced by 2-bit quantization remains within the first-order differentiable neighborhood, LLC can achieve loss reduction with 2-bit compression. As shown in Table 2 of the original paper, some models maintain lossless performance even after 2-bit compression.

Q3: About inter-layer dependency

A3: Mathematically, the inter-layer dependency primarily stems from the off-diagonal elements $H_{i,j} = \frac{\partial^2 f}{\partial w_i \partial w_j}$ of the second-order Hessian matrix. However, experimental results indicate that within the first-order neighborhood, the contribution of the second-order terms is negligible, and the first-order terms remain dominant. Consequently, within this neighborhood, LLC can reasonably neglect inter-layer dependencies, effectively transforming the bit-width selection problem in mixed-precision quantization into an approximately independent grouped knapsack problem.

Q4: About comparison

A4: The following table is a comparison with the latest baseline method. LLC is competitive and the accuracy drop is still minimal.

The following table shows LLC's performance on Tinyllama compression on MMLU. LLC still improves accuracy while reducing loss. Due to rebuttal policy we were unable to submit the latest paper, but more results and details have been added to the original paper.

Due to our lack of familiarity with the neighborhood characteristics of the diffusion model, coupled with the limitations of rebuttal time and calibration datasets, it is currently difficult to quantify and analyze the diffusion model within a limited time. We plan to explore this direction in future work, and hope you understand.

Thank you for your suggestions and we look forward to receiving your feedback.

Q1: About the evaluation metrics in Table 3.

A1: First, in Table 3, the decrease in loss value represents the decrease in cross entropy. Cross entropy and accuracy represent different evaluation methods. Compared with accuracy, cross entropy can compare the closeness between the probability distribution predicted by the model and the true distribution. That is to say, cross entropy can more carefully reflect the performance of the model on different samples, not only whether it is correct or not, but also the degree of certainty of the prediction. At the same time, accuracy will be dominated by the performance of most categories. Compared with HAWQ, although LLC has a slightly lower accuracy, LLC performs better than HAWQ under the evaluation standard of cross entropy. This shows that lower loss values can reveal more detailed improvements in the probability distribution of the model. Secondly, compared with the original model Mobilenet-v2, LLC not only outperforms the original model in accuracy, but also has a lower cross entropy than the original model. HAWQ outperforms the original model only in the accuracy evaluation metric, but is lower than the original model in the cross entropy evaluation metric.

Q2: About loss and accuracy.

A2:

1. Different focus. Loss functions remain a valid evaluation metric, such as cross-entropy, sequence cross-entropy, and AUC error, among others. We believe that both cross-entropy and accuracy reflect model performance, regardless of good or bad, but they emphasize different aspects. For our work, as described in Q1, loss functions like cross-entropy are better suitable, making them a more appropriate evaluation standard.

2. Generality metric. As mentioned in the paper, we aim to develop an evaluation method that is applicable to different tasks and technical frameworks to validate the universality of LLC. Since different tasks or models may employ diverse evaluation metrics, and the loss function, as an inherent property of the model, can uniformly reflect the model’s performance across various tasks, using loss as the evaluation metric is the best choice for verifying the universality of LLC.

3. LLC does not target specific forms of loss, but it can steadily reduce losses. LLC is a compression method for loss reduction, that is, compression may lead to performance improvement. In order to prove the versatility of LLC, LLC has been verified on different tasks, data sets, and compression methods. More importantly, there are differences in the loss indicators used in different tasks in this paper, but LLC can achieve the reduction of loss after compression, proving that it is independent of the loss function. Theoretical analysis and experimental results can show that, under the differential framework, LLC implements a method for reducing loss after compression for a variety of losses (cross entropy, sequence cross entropy, AUC error, etc.) and compression techniques (quantization and decomposition). In other words, LLC also provides an explanation for the performance improvement after compression, which is also used as a guide for future compression work.

4. Reasonableness of loss evaluation. Regarding evaluation metrics, we believe that the loss value best reflects the effectiveness of LLC. Because the starting point of LLC is the analysis of model loss, the purpose is to reduce the loss values (cross entropy, sequence cross entropy, AUC error, etc.) on different tasks. It is undeniable that most papers with theoretical analysis start with loss, and theoretical analysis ultimately achieves the purpose of reducing loss. However, in the experiment, although these works may have advantages in other indicators, they avoid the evaluation of loss (such as HAWQ, etc.). So are there certain flaws in such theoretical verification? If these works lead to higher model loss, does it mean that there are certain flaws in the theoretical analysis and it does not match the experimental results? Based on the above concerns, we use loss to evaluate and verify the rationality of the theory, despite similar papers avoiding experiments on loss.

Thank you for your valuable feedback. We are honored to discuss this issue with you and look forward to your further comments.