Q-VLM: Post-training Quantization for Large Vision-Language Models

We appreciate your suggestion and agree that comparisons with more advanced techniques and cross-attention based VLM architectures would provide a deeper insight into the effectiveness of our method. Below are our detailed responses.

Q1: Ablation study about the projectors.

[Reply] For quantization about the projector between the vision encoder and the LLM, we did not quantize it as it contains only 20M parameters which is negligible compared to the 7B parameters of the entire model. Additionally, remaining the projector to FP would not result in significant increase in inference speed by 0.06h (6.13h vs. 6.07h) compared with 4-bit projector. We conduct experiments on OpenFlamingo architecture with encoder-decoder structure which emphasize the leverage of the model inherent architecture for multimodal feature alignment and our Q-VLM still achieves outstanding performance compared with baseline methods. The results are presented as follows based on Tables in Overall Author Rebuttal Q1:

We conclude that our Q-VLM designed in LLaVA-like architectures can be effectively adapted to other transformer-based VLM architectures with different projectors due to the effectively vision-language alignment information preserved in projectors.

Q2: Performance on more baseline methods.

[Reply] In response to Overall Author Rebuttal Q1, we have conducted experiments and include a more in-depth discussion with more advanced techniques ZeroQuant-V2 compared with proposed Q-VLM. ZeroQuant-V2 fails to acquire the optimal rounding strategy with severe outliers under low bitwidth led to significant quantization loss. On the country, our Q-VLM leverages Information Entropy as a proxy and mines the cross-layer dependency to achieve optimal block partitioning. As a result, our method outperforms ZeroQuant-V2 by 1.71 (79.79 vs. 78.08) in answering accuracy on ScienceQA dataset under 4-bit in LLaVA-7B model. Additionally, our method enhances inference speed, exceeding ZeroQuant-V2 by 0.06h (6.13h vs. 6.07h) due to utilizing stored rounding parameters instead of dynamic per-token quantization. The additional baseline provides a more comprehensive evaluation framework to highlight the strengths of our approach.

Q3: Performance on other multi-modal architectures.

[Reply] In response to Overall Author Rebuttal Q1, we additionally explored our Q-VLM into another multi-modal architecture OpenFlamingo. Q-VLM designed in LLaVA-like architectures can be effectively adapted to cross-attention based VLMs due to the consistent core mechanism of cross-attention and the robust multimodal alignment capabilities pre-trained on large-scale vision-language pairs. Since OpenFlamingo is a cross-attention based VLM, exploiting cross-layer dependency is particularly suitable. Our method outperforms Q-LoRA by 2.08 (47.84 vs. 45.76) under 4-bit in OpenFlamingo-3B model. Q-VLM achieves high accuracy on different cross-attention based multi-modal architectures, which means that our method maintain effectiveness and generalizability.

Hi. Thank you for writing the rebuttal! I confirm I have read the rebuttal, which addresses most of my concern. So I would keep my score.

Thank you for careful reading and valuable comments. We will check the paper carefully, and modify the presentation of the ambiguous parts in the final version. We provide answers to the questions as follows:

Q1: Evaluation on Diverse Datasets.

[Reply] LVLMs encounter diverse range of tasks and challenges in real-world applications. We evaluate our proposed method on various datasets with different architectures to verify the effectiveness and generalizability. The experiments are presented in Table 3 and as follows:

ScienceQA challenges LVLMs with domain-specific question answering requiring deep understanding of scientific concepts and visual data interpretation. VQA v2 and MM-Vet[1] tasks LVLMs with the need to accurately answer open-ended questions about diverse images, testing their ability to integrate visual and linguistic information. VizWiz leverage LVLMs to interpret and answer questions about everyday images captured by visually impaired users, often dealing with low-quality and diverse real-world content. Our method outperforms Q-LoRA by 1.31 (49.28 vs. 47.97) under 4-bit in LLaVA-v1.5-13B model on VizWiz dataset, which shows the superiority of mining cross-layer dependency to effectively reduces quantization errors and enhances generalizability for low-quality and diverse real-world content. Q-VLM achieves the highest accuracy among various post-training quantization methods for LVLMs across different VQA datasets, indicates that our method can be robustly deployed in diverse downstream tasks.

Q2: Evaluation on Different Tasks.

[Reply] Our Q-VLM has achieved the highest accuracy across different visual question answering tasks including ScienceQA, VizWiz, VQA v2 and MM-Vet, where our Q-VLM outperforms Q-LoRA by 0.93 (31.64 vs. 30.71) under 4-bit in LLaVA-v1.5-13B model on MM-Vet dataset.

Additionally, we conduct experiments on other multi-modal architecture OpenFlamingo with Hateful Memes dataset for classification task to further demonstrate effectiveness and generalizability of our proposed method in the Overall Author Rebuttal Q1. Our method outperforms Q-LoRA by 2.08 (47.84 vs. 45.76) and 1.03 (51.05 vs. 50.02) in OpenFlamingo-3B model under 4-bit and 6-bit respectively. The Hateful Memes dataset is a collection of multimodal data containing images paired with text captions, specifically designed for research in detecting hate speech in memes. Our Q-VLM excels in the classification task on the Hateful Memes dataset by effectively combining vision-language information to accurately identify hate speech in multimodal memes.

Q3: Correcting Some Typos.

[Reply] Thank you for your valuable comments. We have thoroughly revised the paper, carefully improving the writing, and correcting misleading wording and grammatical errors. We hope these revisions make the writing more professional and easier to understand.

[1] Yu, Weihao, et al. "Mm-vet: Evaluating large multimodal models for integrated capabilities." arXiv preprint arXiv:2308.02490 (2023).

We would like to thank the reviewer for the careful reading and valuable comments. We address the questions and clarify the issues accordingly as described below.

Q1: Confusion about the quantization strategies.

[Reply] We apologize for the confusion. The detailed construction of the baseline quantization methods is described in Section 4.3 as we leverage RPTQ and Outlier Suppression for activation quantization in single language and vision modality model. For our methods, the language model employs cross-layer dependency mining method exclusively, while the vision model not only utilizes cross-layer dependency mining but also incorporates additional optimization for vision encoder with vision-language synergistic optimization. Our vision encoder optimization method utilizes Jacobian matrices to assign various importance weights to different layers for vision-language synergistical optimization. We did not quantize the projector, as it contains only 20M parameters which is negligible compared to the 7B parameters of the entire model. Additionally, remaining the projector to FP would not result in significant increase in inference speed by 0.06h (6.13h vs. 6.07h) compared with 4-bit projector.

Q2: Explain why the proposed methods particularly suitable for LVLM.

[Reply] According to the Overall Author Rebuttal Q2, we point out our visual encoder optimization method is designed specifically for LVLMs due to the multi-modal data distribution compared with LLM. CNNs primarily rely on the layer-by-layer transmission of local features within the receptive field and the pooling operations that gradually lose detailed information. This makes it challenging for them to achieve global information capture and direct associations like the self-attention mechanism of LVLMs.

Q3: Ablation study about the number of training data.

[Reply] Thanks for your advice. We have conduct ablation studies to evaluate the effectiveness of varying amounts of calibration images used during the post-training quantization on both accuracy and calibration cost. Leverage more calibration images can reduce the rounding function overfitting, while may result in extreme distribution data selection and increased calibration time overhead with large-scale learnable parameters. The results are presented as follows:

Observing the accuracy and calibration cost for different amounts of calibration images settings, we analyze that medium calibration images achieve the optimal performance. Limited amounts of calibration images face the challenges of overfitting, while high calibration images select extreme distribution data with marginal improvements in accuracy and substantially increased calibration cost due to large-scale learnable parameters. We suggest that utilizing 64 calibration images achieve optimal trade-off between quantization accuracy and rounding function generalizability.

Q4: Confusion about the mining cross-layer dependence.

[Reply] We detailly describe the significance of leveraging entropy as a proxy which identifies sensitive layers and facilitates optimal layer allocation through efficient cross-layer dependency correlation in Rebuttal for TnuV Q2. In data-limited PTQ quantization, it is impractical to minimize discrepancies by reconstructing the network final output to achieve the optimal solution for large-scale second-order errors. Directly searching the optimal rounding function is NP-hard because the search space increases exponentially with the layer number. Compared with conventional layer-wise quantization which minimizes quantization errors for each layer in the greedy way to solve that NP-hard problem, our method considers cross-layer dependency and employs entropy as a proxy for block-wise quantization to achieves a satisfying trade-off between discretization errors and search cost. Our cross-layer dependency mining method outperforms BRECQ by 0.56 (78.66 vs. 78.10) and 1.04 (79.89 vs. 78.85) in the 7B and the 13B models respectively. Consequently, our method achieves optimal block partitioning and effectively utilizes cross-layer dependency.

Dear Reviewer,

Thanks for reviewing this paper! The authors have provided rebuttal to address your concerns. Could you have a look and let the authors know if you have further questions?

Thanks, Your AC

Thank you for your valuable feedback. We appreciate the opportunity to clarify the points you raised regarding our methodology and its contributions.

Q1: The benefit and motivation for using entropy as a proxy are unclear.

[Reply] We are sorry for the confusion. For block-wise quantization, we partition the entire model into different blocks and search the optimal rounding function by considering the output quantization errors for each block achieves better trade-off between the search cost and the quantization accuracy. Our goal is to obtain the optimal block partition for rounding function search, where we leverage Information Entropy to judge the sensitive layers with homogeneous distribution. For sensitive layers, the noise in the former layer caused by the deviation from the global optimal value usually leads to higher deviation for the current layer, so that jointly search these layers decrease the block output quantization errors. As a result, discretization error difference (DED) between layer-wise search and joint search is obvious for sensitive layers with high entropy. The benefit and motivation for using entropy rather than quantization errors as a proxy for block-wise searches lie in several key considerations.

First, we analyze that larger entropy indicates more homogeneous data distribution, which is a well-established principle in information theory. Consequently, DED and activation entropy are strongly correlated with an $R^2$ value of 0.97. However, greater quantization error does not necessarily imply more homogeneous data distribution and does not show a positive correlation with DED, having an $R^2$ value of 0.81, which is empirically verified in the figure presented in the Author Rebuttal.

Meanwhile, the search cost of quantization errors doubles compared with entropy as a proxy, as the calculation of quantization errors requires multiple forward passes for both the FP model and the quantized model. The weak correlation and the unbearable search cost render quantization error unsuitable as a metric for measuring cross-layer dependency.

Furthermore, we conducted experiments comparing the proxy effectiveness of quantization error and entropy across different models under various bitwidths. Entropy outperformed quantization errors by a significant margin (78.66 vs. 77.97), showing a strong cross-layer dependency within each block. This allowed us to achieve optimal block partitioning by mining the cross-layer dependency.

Q2: The idea of mining cross-layer dependency is not new.

[Reply] Existing works such as BRECQ incorporates Fisher information and jointly optimizes two layers within each residual block, it does not capture interactions across neighboring residual layers. Additionally, they do not provide the optimal configuration of the reconstruction granularity, with their choice of block-wise optimization stemming solely from experimental results. Consequently, these methods leverage fixed block partitions for rounding function search, leading to suboptimal performance in large models. We conduct experiments on LLaVA model with 7B and 13B parameters under 4-bit in ScienceQA dataset. Our cross-layer dependency mining method outperforms BRECQ by 0.56 (78.66 vs. 78.10) and 1.04 (79.89 vs. 78.85) in the 7B and the 13B models respectively. Conventional quantization methods[1-3] for LLMs indicate that as model scale increases, systematic outliers with large magnitude emerge in activations, leading to significant quantization errors and accuracy degradation. Therefore, BRECQ's fine-grained block partitioning cannot effectively address outliers in LVLMs under low-bitwidth. On the country, our cross-layer dependency mining method which leverage Information Entropy achieve optimal block partition and sufficiently utilizes the cross-layer dependency.

Q3: Performance on more baseline methods.

[Reply] In response to Overall Author Rebuttal Q1 as well as Q2 from your previous responds, we have conducted experiments with additional state-of-the-art PTQ methods including BRECQ and QLoRA for the vision branch and more recent methods such as ZeroQuant-V2 for the language branch. We further performed experiments combining ZeroQuant-V2 and BRECQ as baseline methods. Our Q-VLM method outperforms the baseline method by 1.71 (79.79 vs. 78.08) for answering accuracy in ScienceQA dataset under 4-bit in LLaVA-7B model. Both ZeroQuant-V2 and BRECQ are insufficient handling severe outliers in LVLM under low bitwidth which significantly degrades performance. In contrast, our Q-VLM method effectively mines cross-layer dependency by leveraging Information Entropy and utilizing Jacobian matrices to assign various importance weights to different layers for vision-language synergistical optimization.

Q4: Why the proposed methods are particularly suitable for LVLM.

[Reply] According to the Overall Author Rebuttal Q2 and the ablation study in Table 1, we conducted experiments to demonstrate individual effect of cross-layer dependency mining method on both LLaMA language model and the vision model. Solely deploy cross-layer dependency mining method indicate limited performance improvement. As the visual encoder optimization method with vision-language synergistic optimization, we conclude our Q-VLM are particularly suitable for LVLM.

[1] Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.

[2] Smoothquant: Accurate and efficient post-training quantization for large language models.

[3] Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.

Dear Reviewer,

Thanks for reviewing this paper! The authors have provided rebuttal to address your concerns. Could you have a look and let the authors know if you have further questions?

Thanks, Your AC