Oscillations Make Neural Networks Robust to Quantization

Thank you for the careful reading of our manuscript, your helpful comments and suggestions of relevant literature.

• Thank you for the suggestion on exploring the quantization performance for different $\lambda$ values. We have now expanded the analysis in A.2 to span a larger range of $\lambda$. In the final version, we will provide additional data regarding the quantization performance for different $\lambda$ values. Also see the response to reviewer tzVB for additional discussions. We present an ASCII rendering of these experiments below:

 100 |                                        
  90 |                      @     @      @  
  80 |               @                         @    
  70 |          @              
  60 |                    
  50 |                                               @      
  40 |                    
  30 |                    
  20 |                    
  10 |                                                    @                
   0 |__________________________________________________________
      10^-3      10^-2      10^-1      10^0      10^1      10^2
                             λ (log scale)

• Thank you for providing an extensive list of additional related work, and specifically the questions related to Bondarenko et al. and Hu et al. Initial reading of these two papers do not indicate that they are focused on the specific effects of oscillations in QAT that we are concerned with. For instance, Bondarenko et al. (2023) is primarily concerned with activation quantization during post-training quantization (PTQ). Similarly, Hu et al. (2024) is concerned with handling outliers efficiently which they also argue has an effect on improving the PTQ performance. We will take a closer look at suggested references, and make sure to cite all relevant papers in the final version.

• Although in this paper we focus on a computer vision tasks to demonstrate the effect of oscillations, we believe that the results would transfer to other domains.

• The theoretical analysis presented in this work explicitly focused on weight quantization, for which we provide experimental performance data in Tables 1-3. We do appreciate the reviewer's suggestion on attempting this for quantizing activation maps. This would be exciting future work as it would entail expanding the theoretical analysis to include activation maps, and performing additional experiments specifically to analyze the quantization effects on activation maps.

Bondarenko et al. "Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing" (2023)

Hu et al. "Outlier-Efficient Hopfield Layers for Large Transformer-Based Models" (2024)

Thank you for the careful reading of our manuscript, your helpful comments and pointers to relevant literature.

We agree that the theoretical analyses in Section 4 are straightforward. At the same time, they provide essential intuition and motivation for the empirical results because they give an explicit description of the mechanism behind oscillations and clustering in a simplified setting.

You are completely right that in our model, there can in principle be clustering without oscillation and oscillation without clustering. We will add the following text at the end of Section 4 to clarify this point:

"In this model, the clustering and/or oscillation of individual weights depends on the relative influence of L(w) and $\delta L$".

We will also elaborate on the choice of the regularizer, adding the following in the paragraph following (23): "In this term, we replaced the factor $x^2$ by a hyperparameter $\lambda$, since the precise expression of $x^2$ is specific to the model studied in Section 3 (see also Appendix A). We empirically find that this regularizer is sufficient to induce oscillations and show their positive effect. The exploration of the design space of oscillation-inducing regularizers, including layer-dependent and/or adaptive scale factors, is left to future work."

Oscillations and QAT: Regarding your observation on the necessity of oscillations, we would like to point out that the fact that oscillations are necessary follows from previously published experiments, as described in the first paragraph of Section 7, and we do not claim to make a novel contribution in this regard.

Additionally, regarding the sufficiency of oscillations, our results are empirical and therefore of course cannot unambiguously show that oscillations are sufficient for QAT in all cases. However, as previous work has essentially claimed that oscillations are harmful, we believe that even showing experimentally that they are sufficient (using MLP, ResNet and Transformer) is a significant contribution, especially when combined with previous results on them being necessary at least for part of the training process.

In one of the places in the initial manuscript (L87) we believe the claims relating oscillations and QAT is strong and we will adjust to "... our results suggest that weight oscillations capture many of the beneficial effects of QAT ..." instead of "all the beneficial effects of QAT".

Theoretical claim on STE and region of quantization: As we note in lines 155-158, the scale factor is chosen to cover the range of the tensor to be quantized, so there is no clamping operation in our quantizer defined in Equation (1). This means that all weights will always be within the quantization region and subsequently that the STE is always 1. We will further clarify this in line 191-192 "Using the STE and recalling that the STE gradient simplifies to $\frac{\hat{\partial}q}{\hat{\partial}w}=1$ (note that there is no clamping in our setting, see Equation (2)), ..."

We will cite the correct version of "Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks" and double check that we are citing the correct version of our other references.

We thank the reviewer for suggesting [1,2]. We will include [1] in the background section on weight oscillations. The type of oscillation discussed in [2] refers to fluctuations in the reconstruction loss across network layers during PTQ, which is traced back to differences in module capacity between adjacent layers. This is different to our area of investigation, oscillations during QAT, which is a periodic change in the quantized value of the weights. We will mention this in the background.

[1] Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision, TMLR 2023

[2] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective, CVPR 2023

Thank you for the careful reading of our manuscript and your insightful comments, and for this characterisation of our work: "The analysis of QAT's encouraging latent weights to be clustered around bucket edge is already interesting enough to be shown to the public."

Regarding point (1), we agree that the accuracy should become better as $\lambda$ gets bigger, and eventually deteriorate as the regularizer starts to dominate the loss for large values of $\lambda$. This is indeed the case if we vary $\lambda$ in a wider range. We performed additional experiments and observe this trend. We will add the corresponding figure to the final version. Please find an ASCII rendering of the data below:

 100 |                                        
  90 |                      @     @      @  
  80 |               @                         @    
  70 |          @              
  60 |                    
  50 |                                               @      
  40 |                    
  30 |                    
  20 |                    
  10 |                                                    @                
   0 |__________________________________________________________
      10^-3      10^-2      10^-1      10^0      10^1      10^2
                             λ (log scale)

We also agree with your point (2) that oscillations do not fully reproduce the training dynamics of QAT. While we do discuss this already in lines 406-421, we will further clarify this in the manuscript by adding the following text after line 421: "Preliminary observations indicate that some of the secondary effects of QAT can be beneficial for training dynamics and convergence, see Appendix A.4".

Training dynamics of ViT with OsciQuant: We currently do not have a clear explanation for the behaviour reported in Figures 7 and 8 for ViT. The closest hypothesis we have is the attribution we already make in A.4 using the arguments from Liu et al. 2023, who point out that "the interdependence between quantized weights in query and key of a self-attention layer makes ViT vulnerable to oscillation" which might explain the behaviour observed when we only induce oscillations using our method.

We have now analyzed the training curves for other models like ResNet-18. We do not observe these large drops in accuracy for ResNet-18, which leads us to believe this is an artifact of transformer-based models as suggested by Liu et al. 2023.

Liu et al. "Oscillation-free Quantization for Low-bit Vision Transformers" 2023.

Thank you for your thorough and careful reading of our manuscript.

We are pleased to read that you found our paper to be "well-written and has good theoretical proof" and about your assessment that the "experimental designs and analyses are sound and convincing".

However, we respectfully disagree with the statement that "the contribution, significance and novelty are limited".

Our paper demonstrates for the first time the beneficial effect of oscillations in quantization-aware training (QAT), which challenges a major belief on the mechanisms underlying QAT. We believe our contribution to be both novel, since it is the first to show the beneficial effect of oscillations, and significant, since we expect that our result will significantly impact the thinking and design principles behind the development of future QAT and quantization-aware methods in general, which has traditionally emphasized aligning weights to bucket centers.

We would also like to point out that other reviewers commented positively on the novelty and significance, with Reviewer tzVB writing, "The analysis of QAT's encouraging latent weights to be clustered around bucket edge is already interesting enough to be shown to the public" and Reviewer 3CFm confirming the novelty of our approach in writing "this work is the first to focus on the beneficial effect of oscillation."