CL-Calib: Enhancing Post-training Quantization Calibration through Contrastive Learning

1. The presentation quality needs to be improved, including but not limited to:

• Figure 1 is misleading, Why FP activation is extracted from the intermediate layer of the quantized network but the quantized activation is not? Why backward pass does not flow through the quantized network?
• Many typos need to be fixed, e.g. Sec.1 contrastive learning (CT) => (CL), (e.g., ,) => (e.g.,) (remove one comma). Figure 1 Quantize Parameter => Quantization Parameter. Sec.2 (i.e., ,) => (i.e.,) (remove one comma), Eq. 6 there is one mismatch "}", etc.
• The first subsection of section 3.1 (Instance Recognition (IR) for PTQ Calibration) has nearly no relationship with quantization, I suggest the author move this part to section 2 as the preliminary of contrastive learning.

2. The motivation of this work is not very clear and straightforward. It seems more like a combination of contrastive learning and calibration of PTQ. I fully understand that the proxy contrastive learning task is the same as the optimization problem of Eq.5 as proved in Sec 3.2, but what is the real and core advantage of using this proxy CL task instead of directly optimizing the target?

3. More tasks of PTQ other than image classification should be verified, such as object detection and even some NLP tasks. Also, experiments on Transformer-based model (including ViTs) is a plus. In Table 1, the author mentions that all experiments are implemented by them but why some results of specific methods on specific bit-width or models are missing? (ACIQ-Mix and Bit-split are not evaluated on RegNetX and MNasx2, BRECQ is not evaluated on 4/2, 2/4-bit, etc.)

1. The idea of contrastive learning for tuning quantization parameters is straightforward but also need very careful design of selecting positive and negative samples in different down-streaming tasks. In this paper, this design for tasks such as object detection is not clearly stated and the experimental setting is also not very clear. This makes this work not sounded for these tasks.
2. The training stability of contrastive learning under such limited training images is mentioned but not very fully explained in this paper. How and why the mapping function can solve that problem very well theoretically and experimentally. In this paper, the experimental results did not provide a convincing evidence that this problem can be easily solved. For example, all experimental results should be displayed as a stable performance improvement under different selection of training samples rather than the best result shown in all tables and figures.

1. The improvement in performance with 4-bit quantization is not substantial, and the absence of experiments in 8-bit quantization and language model domains limits the breadth and depth of the study.
2. The study lacks significant innovation, as it does not introduce novel methods or techniques that significantly differ from existing research, making its innovative contribution less prominent.
3. The practical utility of the proposed approach is constrained. Despite achieving results in 2-bit quantization, the performance gap compared to full-precision models remains considerable, restricting its applicability in real-world scenarios.

1, This paper does not provide the overall process. It only optimizes the quantization parameters of activations. Is that all? Nothing else? It confused me. A more clear description is needed. 2, There is some confusion when Eq15 comes to Eq16. The author says: we can further loosen the lower bound to obtain the optimization objective. However, it is more likely a formula that's been forcibly pieced together. 3, There is some missing paper that also uses contrastive learning in PTQ. [1] 4, I do believe the proof in Appendix A.2 has some typos. For example, symbol between Eq15 and Eq16.

[1] CPT-V: A Contrastive Approach to Post-Training Quantization of Vision Transformers