Efficient Low-Bit Quantization with Adaptive Scales for Multi-Task Co-Training

Q4 (Questions1): While the model design is good and interesting, the model seems a little complex. Would it be possible to provide some comparative analyses regarding the model's speed and complexity?

A4: Thank you for your concern. Compared to existing QAT methods, our method does not add much complexity. The comparative analyses are discussed in Appendix Section E. We provide explanations and comparisons from two perspectives: computation cost and parameter counts.
(1) As the co-training [1, 2] framework trains a versatile model that performs only one task at a time for a given input, only the corresponding set of quantization scales is used during inference for each task, thereby avoiding any additional computational burden. As shown in Table F, compared to baseline_multi (LSQ+), the model trained with TSQ-MTC achieves the same 4.14G FLOPs with a compression ratio of 87.5%, while delivering a significant performance improvement (+0.21dB on the Urban100 dataset for $\times 2$ SR task).
(2) Since most low-bit quantization methods use layer-wise quantization for activations [4], we only chose to introduce task-specific quantization scales for the activation quantizer to adapt to different input data. The additional parameter count is minimal. Table F indicates that TSQ-MTC maintains an almost identical parameter count (14.38M with a compression ratio of 87.5% vs. 14.34M with 87.6%), with only a minor increase of 0.04M.
Additionally, SLLD is only adopted during training. As a result, our model is both effective and efficient.

Thank you once again for the insightful feedback. If you have any other questions or would like to discuss further, please inform us. We sincerely look forward to your comment.

REFERENCES:

[1] Omnivore: A single model for many visual modalities. CVPR'2022.
[2] Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv'2023 (395 citations).
[3] Q-vit: Accurate and fully quantized low-bit vision transformer. NeurIPS'2022 (77 citations).
[4] Cadyq: Content-aware dynamic quantization for image super-resolution. ECCV'2022.

Thank you for your patience, as the additional experiments required more time.
We sincerely appreciate Reviewer icGh’s valuable feedback and constructive comments. We address each question below.

Q1 (Weaknesses1): There is a minor error in Section 2.1 - regarding the order of description for the "single-modal task-related data scenario" and the "multi-modal data scenario" are misaligned with the order of "former scenario" and "latter scenario".

A1: We apologize for the mistake and thank you for pointing it out. We will correct it in the revised manuscript.

Q2 (Weaknesses2): The paper only considered the low-level multi-task scenarios; however, However, it lacks effective exploration of high-level visual tasks. Specifically, Can the proposed method still maintain effectiveness when processing high-level visual multitasking data such as NYUD-v2, Pascal Context and Cityscapes.

A2: Thank you for raising this question. TSQ-MTC is primarily designed for co-training and, for the first time, we introduce an effective quantization approach tailored for this framework. For validation, we select low-level vision tasks (IPT) and high-level classification task (ResNet), which are suitable for co-training and practical to implement. As you suggested, we share a strong interest in verifying the effectiveness of our method on more high-level vision tasks. However, these tasks often involve resource-demanding pre-training, such as unified architectures [1] and multi-modal large models [2]. Due to the resource and time constraints during the rebuttal period, it may not be feasible to fully explore these tasks at this stage. Therefore, the distribution differences of features in distinct high-level visual tasks require further investigation. According to our present observations, the distribution differences are likely to be widespread and may also occur in high-level vision tasks, for which our method can provide an effective solution. We are currently designing a validation experiment, but we may not be able to reach a conclusion within the rebuttal period. If we are able to complete the experiment during the rebuttal period, we will share the results with you. Otherwise, we will discuss the possibility of extending our work to higher-level tasks in the future.

Q3 (Weaknesses4): The experiment involved a comparison of model quantization techniques, including LSQ (2019) and PAMS (2020). Nevertheless, a comparison with the most recent Quantization-Aware Training (QAT) methods is conspicuously absent.

A3: Thank you for your valuable suggestion. The QAT methods compared in the main paper primarily relate to the quantization scales we focus on. As suggested, we include Q-ViT [3] as a new comparison method, which addresses the distribution degradation in quantized ViT. Q-ViT is a recent, highly-cited, and effective QAT method, which can serve as a representative comparative approach. The experimental results are presented in the tables below.

As shown in the tables above, Q-ViT, as a popular and recent QAT method, achieves significant performance improvements on Transformer-based models. However, the performance of our method still surpasses that of Q-ViT (e.g. +0.16dB on Urban100 for $\times 4$ SR). Notably, the motivation and implementation of our method differ from those of Q-ViT, and combining the two approaches could further enhance the performance of quantized models.
We will include these experimental results in the revised paper.

For your concern about the performance of our method on high-level vision tasks, we design and conduct a validation experiment during the rebuttal period.

As you suggested, we train a multi-task model on the NYUD-V2 [1] dataset based on the code of DiSparse [2] within co-training. The tasks includes semantic segmentation (seg) for RGB input and surface normal prediction (sn) for RGBD input. Each task is assigned a corresponding input head and output tail, while sharing a ResNet-based backbone. We first train a converged full-precision model to initialize the 4-bit quantized model. Our experiment compares TSQ-MTC with baseline-multi (LSQ+) with four evaluation metrics. Due to time constraints, our full-precision model may not be trained with the optimal parameters. The results are shown in the table below.

It can be observed that our method significantly improves the performance of the semantic segmentation task (mIoU increased by 0.25%) while maintaining the performance of the surface normal prediction task. This demonstrates that our method effectively alleviates quantization conflicts between tasks.

We hope the above experiments address your concern. We will continue to complete the experiments on high-level vision tasks and include the results in the final version of the paper.

REFERENCES:

[1] Indoor segmentation and support inference from rgbd images. ECCV'2012
[2] DiSparse: Disentangled Sparsification for Multitask Model Compression. CVPR'2022.

Thank you for your patience, as the additional experiments required more time.
We sincerely appreciate Reviewer gsJW’s valuable feedback and constructive comments. We address each question below.

Q1: Could the authors provide the computational cost of training and inference compared to existing QAT methods?

A1: Thank you for your question. Our method does not significantly increase the computational cost. The relevant discussion, including FLOPs and parameter count, are provided in Appendix Section E. We provide some additional details for both training and inference.
(1) During training, the layer-wise activation quantization scales (scalars) do not occupy excessive GPU memory. Additionally, the model performs only one task per input batch with task-specific quantization scales. Therefore, TLMAQ does not increase the computational burden. For example, as shown in the table below, the 4-bit IPT quantized by TSQ-MTC uses 76,722 MiB of GPU memory with a batchsize of 45, whereas Q-ViT [1], a popular and recent QAT method also based on LSQ+, requires 78,996 MiB. The training times for both methods are roughly the same, taking about 4-5 days (on 5 NVIDIA A100 GPUs).

(2) For inference, we compare the parameter counts and FLOPs of the 4-bit IPT trained by baseline-multi (LSQ+) and TSQ-MTC in Table F. Our TSQ-MTC outperforms baseline_multi while maintaining a nearly identical parameter count (14.38M with a compression ratio of 87.5% vs. 14.34M with 87.6%), with only a slight increase of 0.04M. Moreover, our method has the same FLOPs (4.14G) and acceleration performance as LSQ+.
We will emphasize the computational efficiency of TSQ-MTC in the revised paper.

Q2: Could the authors report the variance of the results of the proposed SLLD method in the ablation studies?

A2: Thank you for your suggestion. Regarding your concern, we conduct our discussion by analyzing performance observations and providing further validations.
(1) Performance Observations: In our experiments, we find that improving the performance of quantized IPT on certain datasets can be challenging, such as the denoising task on the CBSD68 dataset (Table A). SLLD, as an additional enhancement to our method, is primarily aimed at addressing the observed information distortion issue (Figure B). Therefore, the improvement is less pronounced than that of TLMAQ. Addationally, after incorporating SLLD, the performance consistently improves or remains stable. It also demonstrates clear performance improvements on certain complex datasets, such as Urban100 (+0.03dB for $\times 3$ SR).
(2) Further Validations: To address your concern about potential experimental fluctuations, we conduct additional experiments to obtain the variance. Given the limited time during the rebuttal period, we resume training from the weights at the 40th epoch and calculate the mean and variance of the test results for the final weights (50 epochs) of SLLD and non-SLLD training. Each set of statistics includes five test results on the same task and dataset. The calculated standard deviation results are as follows:

It can be seen that the standard deviation of our method is not significant enough to affect its performance. We hope the table above clarifies your concerns and provides further evidence of the robustness of our approach.

Thank you once again for the insightful feedback. If you have any other questions or would like to discuss further, please inform us. We sincerely look forward to your comment.

REFERENCES:

[1] Q-vit: Accurate and fully quantized low-bit vision transformer. NeurIPS'2022.

We sincerely appreciate Reviewer LoRG’s valuable feedback and constructive comments. We will revise our organization and emphasize those contents in the main paper to better highlight the effectiveness of our method. We address each question below.

Q1: In table 1, quantitative results for super-resolution tasks are shown. But I am still curious about the results of deraining and denoising tasks.

A1: Thank you for your efforts. Our TSQ-MTC also achieves the best performance on denoising and deraining tasks, where detailed results can be found in Appendix Section B (Table A and Table B). We compare the 4-bit IPT quantized by TSQ-MTC with Minmax, PAMS [1], LSQ [2], and baseline-multi [3]. For instance, our TSQ-MTC reaches 42.02dB in derain task on Rain100L dataset, while baseline-multi achieves only 41.89dB. We will highlight the discussion of experiments in the main paper to more clearly and accurately present the denoising and deraining results.

Q2: How about the parameters change of your method?

A2: Thank you for your concern. The additional parameter count of our TSQ-MTC is negligible. Since most low-bit quantization methods use layer-wise quantization for activations [4, 5], we only chose to introduce task-specific quantization scales (scalars) for the activation quantizer to adapt to different input data. In Appendix Section E, we present the parameter counts and FLOPs of the models trained by baseline_multi and TSQ-MTC, along with their corresponding performance. As shown in Table F, TSQ-MTC outperforms baseline_multi while maintaining a nearly identical parameter count (14.38M with a compression ratio of 87.5% vs. 14.34M with 87.6%), with only a slight increase of 0.04M. This result underscores the efficiency of our method. We will also add these descriptions to the revised version of our paper to highlight the advantages of TSQ-MTC in model compression.

Q3: Can you provide the some comparisons with some SOTA single task methods to further demonstrate the superiority of your method?

A3: Thank you for your suggestion. We compare our 4-bit TSQ-MTC to several state-of-the-art single-task super-resolution quantization methods, such as PAMS [1], CADyQ [5] and QuantSR-T [6] in Appendix Section B (Table C). Our method demonstrates a clear performance advantage over single-task quantized super-resolution models. For example, in the $\times 2$ super-resolution task on the Urban100 benchmark, TSQ-MTC achieves a PSNR of 33.67dB, notably outperforming QuantSR-T, which reaches only 32.20dB. We will also polish our descriptions and highlight these comparisons to the revised version.

Thank you once again for the insightful feedback.

REFERENCES:

[1] Pams: Quantized super-resolution via parameterized max scale. ECCV'2020.
[2] Learned step size quantization. ICLR'2019.
[3] Lsq+: Improving low-bit quantization through learnable offsets and better initialization. CVPR Workshop'2020.
[4] Nonuniform-to-uniform quantization: Towards accurate quantization via generalized straight-through estimation. CVPR'2022.
[5] Cadyq: Content-aware dynamic quantization for image super-resolution. ECCV'2022.
[6] Quantsr: accurate low-bit quantization for efficient image super-resolution. NeurIPS'2024.