Fisher-aware Quantization for DETR Detectors with Critical-category Objectives

Thank you for your feedback!

We would like to point out a misunderstanding you had on our reported performance of the COCO-Panoptic dataset. As mentioned in Sec. 5.1, we perform box detection task with the COCO-Panoptic dataset and report the mAP box in the results. Here we follow the box detection training process described in the official DETR repo (https://github.com/facebookresearch/detr#training-1).

COCO-Panoptic dataset extends from the 80 categories of COCO to a total of 133 categories grouped into 27 super categories. This significant increase in category numbers leads to a more servere impact on the critical performance. Our methods lead to significant imrovements in this challenging setting. More significant inprovements are also observed on complicated DETR-based architectures, where we see up to 2% critical mAP performance on 4-bit DAB-DETR, and up to 14.5% on Deformable DETR comparing to Uniform quantization baseline. All these verify the potential of the proposed method.

We thank reviewer Djvc for your thorough review and valuable comments. Hopefully, the following responses can address your concerns.

Q1: Importance of our proposed method

A1: Thank you for bringing up these possible methods. As we do not claim Fisher-aware quantization to be the only way of improving the fine-grained performance of the quantized model, here we discuss the advantage of our method over other alternatives.

For adjusting the weight of fine-grained losses, we have compared our proposed scheme with performing regular QAT with the summation of overall and critical objectives in Tab. 12 of Appendix D. For the Fisher-trace regularization, the motivation comes from our Insight 2 in Sec. 4.1, where the QAT gap is caused by the sharp loss landscape leading to a poor generalization. This cannot be simply resolved with the addition of critical objectives in the training loss, as illustrated by the results in Tab. 12, where the summed loss does not consistently lead to critical performance improvements.

As for the Instance-aware dynamic quantization method, it may not bring real efficiency when deployed on a hardware system. All the weights need to be stored at the highest possible precision, and the switching of precision at inference time will bring additional overhead. Furthermore, multiple objects with different classes are likely to exist in the same scene for object detection tasks, which brings further challenges for instance-aware methods on deciding the quantization scheme to use. Our proposed method works with a static quantization scheme for all the inputs, which does not suffer from the aforementioned drawbacks.

Q2: Generalizability of the proposed method

A2: This paper focuses on the quantization performance of the DETR model and its variants given their wide usage as state-of-the-art in object detection benchmarks. We also find the quantization of DETR is more challenging and interesting compared to CNN-based models, as the CNN quantization has been extensively studied in the past. Our focus on DETR quantization is also echoed by the recent work Q-DETR [1] in CVPR2023, which validates our motivation.

Within the scope of DETR-based models, we have conducted our experiments with multiple architectures including DETR-R50, DETR-R101, DAB DETR, and Deformable DETR, and on multiple datasets including COCO (Tab. 3 and 5), COCO-Panoptic (Tab. 4 and 6), and CityScapes (Tab. 9). Our proposed method shows consistent improvements across all these settings. Per the suggestion of reviewer ka68, we further evaluate our Fisher-aware quantization scheme with the Q-DETR training objective, and observe similar improvements in the performance. These results verify that our approach is generalizable across architectures, datasets, and QAT methods.

[1] Xu, S.,et al. (2023). Q-DETR: An Efficient Low-Bit Quantized Detection Transformer. In Proceedings of CVPR (pp. 3842-3851).

Q3: Significance of the proposed method

A3: We would like to note that both the Fisher-aware quantization scheme (both overall and critical) and the Fisher-trace regularization on the DETR model are our contributions derived from our theoretical analysis in Sec. 3. When combined, they lead to an improved critical mAP performance over uniform quantization baselines by up to 1.15% on 4-bit DETR-R50 on the COCO dataset, and up to 10.4% on COCO-Panoptic where the critical performance is hindered more due to the large number of classes (Sec. 5.3).

Also, we would like to argue that the significance of the empirical improvements should not be solely determined by the value itself. Our results of the Fisher-critical quantization scheme are significant in the sense that 1) the improvements are mostly 3x larger than the standard error induced by the training randomness, and 2) the improvement is consistent across different precisions, architectures, and datasets. This verifies the improvements brought by our proposed method are significant and not random. Furthermore, we find the improvement of our method is higher when applied to more complicated architectures. Critical mAP improves by up to 0.5% on DETR-R101, 0.8% on DAB DETR-R50, and 0.4% on Deformable DETR-R50 with Fisher-critical over Fisher-overall, compared to the 0.2% improvement on DETR-R50. This further shows the potential of the proposed method as larger models may require more compression.

Besides the observable improvements brought by our method, we would like to emphasize that this paper is the first to investigate the impact of quantization on the fine-grained critical-category performance of DETR-based object detectors, which is a practical setting yet largely overlooked by the community. As we take an initial step to tackle this challenge in this work, it will motivate future explorations on the fine-grained impacts of other compression methods in the computer vision area and general machine learning research.

We thank reviewer ka68 for your thorough review and valuable comments. Hopefully, the following responses can address your concerns.

Thank you for bringing up the Q-DETR paper to our attention. We acknowledge the contribution of Q-DETR as a pioneer work in designing better QAT objectives and algorithms for DETR. We are happy to cite the paper in our related work section.

As you have clearly identified, Q-DETR and our paper work on different aspects of DETR quantization. Q-DETR focuses on better distillation objectives to improve the overall DETR performance in QAT. In contrast, our work proposes a Fisher-aware mixed-precision quantization scheme design and a Fisher trace regularization, with a special focus on improving critical-category objectives. In this sense, the contributions of both papers are orthogonal to each other. Following your suggestion, we have included an experiment of training DETR models quantized with different mixed-precision quantization schemes using the Q-DETR distillation objective. Due to the limitation in time, we ran all experiments with the DETR-R50 model on the COCO dataset, with a total of 50 epochs using a 1e-5 learning rate. All the other settings are the same as described in Sec. 5.1. Though the reported performance here is lower than that of the Q-DETR paper due to the 10x less training epochs, we believe the trend in relative performance remains if longer training is conducted.

The results are consistent with our findings in Tab. 3 that the proposed Fisher-overall quantization scheme improves over uniform quantization on the overall model performance, whereas the Fisher-critical quantization scheme further improves the corresponding critical performance. This result verifies that the disparate impact on critical performance exists for different QAT objectives, while our method is generalizable to other QAT objectives.

I'm shocked that you didn't know about the first and unique article in your field, and it was published in CVPR.

We thank reviewer WkhP for your thorough review and valuable comments. Hopefully, the following responses can address your concerns.

We would like to note that both the Fisher-aware quantization scheme (both overall and critical) and the Fisher-trace regularization on the DETR model are our contributions derived from our theoretical analysis in Sec. 3. When combined, they lead to an improved critical mAP performance over uniform quantization baselines by up to 1.15% on 4-bit DETR-R50 on the COCO dataset, and up to 10.4% on COCO-Panoptic where the critical performance is hindered more due to the large number of classes (Sec. 5.3).

Also, we would like to argue that the significance of the empirical improvements should not be solely determined by the value itself. Our results of the Fisher-critical quantization scheme are significant in the sense that 1) the improvements are mostly 3x larger than the standard error induced by the training randomness, and 2) the improvement is consistent across different precisions, architectures, and datasets. This verifies the improvements brought by our proposed method are significant and not random. Furthermore, we find the improvement of our method is higher when applied to more complicated architectures. Critical mAP improves by up to 0.5% on DETR-R101, 0.8% on DAB DETR-R50, and 0.4% on Deformable DETR-R50 with Fisher-critical over Fisher-overall, compared to the 0.2% improvement on DETR-R50. This further shows the potential of the proposed method as larger models may require more compression.

Besides the observable improvements brought by our method, we would like to emphasize that this paper is the first to investigate the impact of quantization on the fine-grained critical-category performance of DETR-based object detectors, which is a practical setting yet largely overlooked by the community. As we take an initial step to tackle this challenge in this work, it will motivate future explorations on the fine-grained impacts of other compression methods in the computer vision area and general machine learning research.