Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification

Dear Reviewer FzrR,

Thank you very much for your effort on our paper and the valuable feedback!

We noticed that you adjusted your scores (333->232, 5->3) before we submitted our rebuttal. We are currently preparing a detailed rebuttal and would appreciate knowing if you have any additional concerns or questions regarding our paper. Your insights are invaluable to us as we work to improve it. We are more than willing to respond to any inquiries and address any feedback. Thank you very much!

Sincerely Yours,

The Authors

Q2-1. Benchmark settings for Tab. 5.

We apologize for the confusion. The main purpose of Tab. 5 is to analyze the impact of different sparsification ratios on performance, which is why we selected two sub-benchmarks from the main benchmarks for experimental analysis. The choice of VQAv2 and GQA was due to their prominent positions in the main benchmarks, with VQAv2 also being a widely recognized large-scale VQA benchmark.

Q2-2. Additional vision-centric benchmarks for Tab. 5.

Thanks for your suggestions. We have included the results for the MMVP benchmark in Tab. 5 of the revised paper. The results for MMVP in Tab. 5 are as follows.

Q3. Distinctions and motivation of different sparsification strategies in prefill and decoding stages.

We apologize for the confusion.

Dynamic-LLaVA sparsifies both image tokens and output text tokens during inference. However, since MLLM has not yet started decoding to generate output text tokens during the prefill stage, only the number of image tokens is reduced at this stage. During the decoding stage of MLLM, since both image tokens and output text tokens are generated, we sparsify them simultaneously.

Additionally, we conducted an experiment of sparsifying text tokens (instructions/prompts) during the prefill stage. You can find the specific results in the "comment for Q4 of reviewer UyMq."

Q4. Whether changes in input length contribute to unstable performance.

First, we need to clarify that during training, Dynamic-LLaVA sparsifies output text tokens that exceeding a certain length ($\text{LEN}^{OT}$) and optimizes the output predictor. This process does not affect the input text tokens (since we do not sparsify the input text tokens).

We have conducted extensive experiments on a variety of benchmarks with diverse input lengths, as shown in Tab. 1, Tab. 2, and Tab. 3 of our paper. Dynamic-LLaVA maintains excellent performance across these benchmarks, ensuring that there are no significant performance fluctuations due to changes in input length.

Additionally, regarding the sparsification ratio of output text tokens (which is similar to the ratio during training), the results in Tab. 3 indicate that on benchmarks with output text tokens of varying lengths, Dynamic-LLaVA still achieves an overall 50% reduction of output text tokens while maintaining stable generation ability. Ablation studies of $\text{LEN}^{OT}$ can be found in Tab. 6 of our paper.

Reference

[1] Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Arxiv2406.

[2] Eyes wide shut? exploring the visual shortcomings of multimodal llms. CVPR2024.

[3] Grok. 2024.

[4] Sharegpt4v: Improving large multi-modal models with better captions. ECCV2024.

[5] IVTP: Instruction-guided Visual Token Pruning for Large Vision-Language Models. ECCV2024.

[6] Llava-prumerge: Adaptive token reduction for efficient large multimodal models. Arxiv2403.

[7] VoCo-LLaMA: Towards Vision Compression with Large Language Models. Arxiv2406.

[8] Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs. Arxiv2409.

[9] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV2024.

[10] HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments. Arxiv2408.

[11] SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. Arxiv2410.

[12] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. 2024.

[13] Tokenpacker: Efficient visual projector for multimodal llm. Arxiv2407.

Dear Reviewer FzrR,

Thank you for your valuable suggestions! We have conducted extensive additional experiments and made revisions to our paper. Detailed responses are provided in the following rebuttal.

Sincerely Yours,

The Authors

W1. Writing.

We recognize that excessive notation may hinder broader reader comprehension. In the revised paper, we are incorporating more intuitive visualizations and adding simplified descriptions to improve the readability.

In the revised version of our paper, we have supplemented the ''Method'' section with simplified explanations (Sec. 3.3), additional details (Appendix A.2), and visual flowcharts (Fig. 3 and Fig. 4).

W2. Additional vision understanding benchmarks.

To more comprehensively evaluate Dynamic-LLaVA's performance in vision understanding tasks, we incorporated three vision-centric vision understanding benchmarks following the Cambrian-1 [1], including MMVP [2], RealWorldQA [3], and CVBench-2D [1]. The results are presented in the table below.

On these three new vision-centric Vision Understanding benchmarks, Dynamic-LLaVA not only leads in performance but also significantly reduces the inference resource cost of MLLMs.

Detailed results can be found in Tab. 1 of the revised paper.

W3. Additional complex generation ability benchmarks.

Thank you for your suggestion. We have added experiments on a complex generation ability benchmark, ShareGPT4V-VQA (single-round) [4], which is designed for generating long texts. The average output text length of this benchmark exceeds 1,000 tokens. The detailed introduction of ShareGPT4V-VQA (single-round) is presented in Appendix A.4.4. The results are presented in the table below.

On this generation ability benchmark with long output texts, Dynamic-LLaVA has significantly enhanced inference efficiency while having minimal impact on the text generation abilityof MLLMs.

Detailed results can be found in Tab. 3 of the revised paper.

Q1. Baseline choice.

We primarily use LLaVA-1.5 as our main baseline, employing the same 12 benchmarks utilized in LLaVA-1.5. This choice is motivated by the fact that most works [5-11] in the field validate their results on these benchmarks for LLaVA-1.5, allowing for a fair comparison by adopting the same settings.

For the latest version of LLaVA, such as llava-next [12], due to its incomplete open-source (only weights are released, without open-sourcing datasets and training codes), it is challenging for our method and other methods in the field, especially those that are training-required, to conduct experiment. We will include additional baselines in the future revision due to the limited rebuttal time.

You might be interested in the results presented in our results in Tab. 2, where we validated our method on a recent Efficient MLLM LLaVA-tokenpacker [13]. This model serves as a challenging baseline since it already uses fewer image tokens to accelerate inference. The results demonstrate that our method can further enhance inference efficiency with minimal impact on performance.

Dear Reviewer FzrR,

Thank you once again for reviewing our paper and providing valuable feedback. We have carefully considered your suggestions and made multiple revisions to enhance the clarity, depth, and contribution of the paper. Your constructive insights and feedback have played a significant role in the process of improving our paper.

We sincerely hope you will continue to engage in the discussion. Should you have further questions or concerns, we are more than willing to provide additional explanations or supporting materials. Your insights are critical to refining our research and ensuring its relevance and impact.

Furthermore, we hope these revisions and clarifications will encourage you to reassess your evaluation, as these updates directly address your constructive comments. If you have any additional questions or concerns, please feel free to reach out to us. We are committed to ensuring that all issues are thoroughly addressed.

Sincerely Yours,

The Authors

Dear Reviewer xRRX,

We greatly appreciate your high regard and assessment of our work, as well as your valuable suggestions regarding the presentation style of our paper! We recognize that our current writing overly relies on formulaic expressions. To address this, we have made concerted efforts to include more detailed explanations and illustrative diagrams. We have implemented the following modifications to enhance the readability of method part in our revised paper.

Sincerely Yours,

The Authors

W. Specific modifications to ''Method'' section of the paper.

• We have moved the ''Parallel Sparsification Inference Optimization'' section focused on engineering optimizations to the Appendix A.1.
• In Sec. 3.3.1, we have incorporated a more detailed explanation of the overall framework, including the specific mechanisms by which learnable predictors operate within the model and the details of the layers affected (L248-L252).
• In Sec. 3.3.2, we further explain how subsequent layers of the model compute after token sparsification at $l$-layer during the prefill and decoding stages, to clarify the rationale behind our reduction of computational overhead (L280-283). Meanwhile, we have also detailed the processing by the model in subsequent layers under the scenario of decoding with KV cache (L296-299).
• In Section 3.3.3, we provide an explanation of the consistency in masking forms during training and inference. Additionally, we further discuss the pipeline of the end-to-end sparsification training comprehensively in Appendix A.2 and add a more intuitive figure to facilitate understanding (Fig. 3).
• In Appendix A.3.2, we provide discussions about the differences between Dynamic-LLaVA and other KV cache compression methods, and add Fig. 4, which visually shows the distinctions between Dynamic-LLaVA (when decoding with KV cache) and a KV cache compression method (H2O[1]).

Reference

[1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. NeurIPS2023.

Dear Reviewer xRRX,

Thank you once again for reviewing our paper and providing valuable feedback. We have carefully considered your suggestions and made multiple revisions to enhance the clarity, depth, and contribution of the paper. Your constructive insights and feedback have played a significant role in the process of improving our paper.

We sincerely hope you will continue to engage in the discussion. Should you have further questions or concerns, we are more than willing to provide additional explanations or supporting materials. Your insights are critical to refining our research and ensuring its relevance and impact.

Furthermore, we hope these revisions and clarifications will encourage you to reassess your evaluation, as these updates directly address your constructive comments. If you have any additional questions or concerns, please feel free to reach out to us. We are committed to ensuring that all issues are thoroughly addressed.

Sincerely Yours,

The Authors

Q2-3. Only use the current token feature information to make decision.

First, we want to clarify that Dynamic-LLaVA evaluates each current token's features to determine whether its activations which through $W_K$ and $W_V$ should be added to the KV cache. It does not use the information of the current token to determine the activations of all previous tokens show be discarded or not. Therefore, making decisions solely based on the activations of the current token is both intuitive and feasible. Our experimental results also partially validate this approach.

It is noteworthy that this method of making decisions based solely on the features of the current token enables independence from past KV cache usage. Consequently, this allows for accelerated MLLMs during prefill and decoding without KV cache.

Q2-4. Analysis of the wrongly evicted tokens.

We present visualizations of the decisions made by Dynamic-LLaVA's predictors for images and output text in Fig. 6, Fig. 7, and Fig. 8. As shown in these figures, Dynamic-LLaVA primarily discards background and redundant content in the vision context while retaining the main subjects comprehensively. For output texts, the discard decisions mostly involve articles, prepositions, and suffixes of long words. We believe that the decisions made by Dynamic-LLaVA are reasonable in most cases.

Q3. Applicability for Group Query Attention.

We believe this is entirely feasible. Allow us to illustrate using PyTorch dimensions as an example. In the computation of the Group Query Attention, the dimensions of $Q,K,V$ are typically [$B$, $N$, $H_{num}$, $H_{dim}$]. The grouping mechanism occurs along the third dimension, i.e., $H_{num}$. Dynamic-LLaVA reduces the size of the second dimension, $N$, which improves computation efficiency without affecting the computation of the Group Query Attention operation.

Q4. Long-text context scenarios during the prefill stage.

Good question! In the table below, we present the results for sparsifying the text context during the prefill stage. We add a predictor for sparsifying text tokens, with an architecture identical to that of the output predictor. The sparsification ratio for text tokens is set to 30%, and we use exactly all same settings as Dynamic-LLaVA.

It can be observed that sparsifying only 30% of the text context leads to a significant performance drop. We believe this is because MLLMs rely on the instruction (prompt) to generate responses, and sparsifying the text context may result in critical content being omitted from the question, leading to incorrect answers. We consider this a direction worth exploring in future work.

We discuss and present the results in detail in Appendix A.3.3 and Tab. 9 of the revised paper.

Reference

[1] Sharegpt4v: Improving large multi-modal models with better captions. ECCV2024.

[2] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. NeurIPS2023.

[3] Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurIPS2022.

[4] FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR2024.

W3. Training time and lantency.

We apologize for not providing this part in our paper.

In the table below, we report the additional training time for Dynamic-LLaVA (measured on 8 A100 80G) and the inference latency when using KV cache (measured on one A100 80G, with the prefill time and the average decoding latency per token for generating 1000 tokens and the fixed batch size of 1).

Additional training time:

Inference latency when using KV cache:

The latency results measured on GPU indicate that the learnable lightweight predictors contribute a negligible increase to inference latency (in practice, less than 1%). Meanwhile, the proposed Dynamic-LLaVA framework exhibits an average reduction of 1.57 ms per token in generation latency compared to LLaVA-1.5 when decoding with KV cache. This improvement in inference speed is due to Dynamic-LLaVA significantly reducing the length of KV cache, which speeds up the attention operations.

Detailed discussions and results are presented in Appendix A.5.2, Tab. 14 and Tab. 15 of the revised paper.

W4. More recent related works about KV Cache compression.

Thank you for your suggestion! We have added the latest KV Cache compression methods and a detailed discussion in Sec. 2.2 and Appendix A.3.2.

W5. Mathematical representations.

We apologize for the confusion. As shown in the "response to W1", we only use the predictors once to generate a mask, so to simplify the notations, we did not use the layer index $l$.

W6. Writing.

Thank you for your suggestions! We have revised the presentation of the paper to add simplified explanations, visual flowcharts, and more discussions. Our changes to the paper are highlighted in blue.

Q1. Sparsification strength adjustment strategy.

Specifically, we first adopted the recommendation from previous work (FastV) and set the sparsification to occur after the second layer. Meanwhile, based on our empirical observations, the layer after which Dynamic-LLaVA performs sparsification is not a sensitive parameter, which enhances its adaptability. Detailed ablation results regarding the choice of layer $l$ can be found in Tab. 6.

Q2-1. Lantency of decoding with KV cache.

Please refer to the "response for W3". In short, the predictors introduce negligible inference overhead, accounting for less than 1% (compared to the total computation cost of MLLM).

Q2-2. Efferency of our ''online'' KV cache compression method and traditional ''attention-based'' KV cache compression method.

Please refer to the "response for W3", the additional inference overhead introduced by Dynamic-LLaVA is practically negligible in inference. Additionally, for traditional "attention-based" KV cache compression methods (such as H2O[2]), which require attention scores for KV cache compression during decoding, the attention operations of efficient inference operators are often implicit (e.g., FlashAttention[3, 4] and torch.nn.functional.scaled_dot_product_attention). This can necessitate an additional computation of attention mechanism to obtain attention scores, which may slightly impact inference speed in scenarios with long KV cache length. This phenomenon is shown in the table below, using the torch.nn.functional.scaled_dot_product_attention operator.

However, Dynamic-LLaVA relies solely on the features of the current token for prediction when decoding with KV cache, thus avoiding this issue.

Detailed discussions and results are provided in Appendix A.5.2 and Tab. 14 of the revised paper.

W2. Benchmark statistics and inference benefit discussion.

Thanks for your suggestions!

First, we provide statistics for some of the vision understanding and generation ability benchmarks in the table below.

As can be seen, vision understanding benchmarks often require outputs in the form of choice answers such as "A," "B," "C," or single words or phrases to accurately assess an MLLM's understanding performance. However, in real-world scenarios, MLLMs are typically required to perform both understanding and generation, necessitating more detailed and extensive responses. This aligns with our constructed LVIS-VQA and ShareGPT4V-VQA [1] (which were added following the suggestion of Reviewer FzrR) benchmarks.

The total token length (prefill and decoding) of LLaVA-1.5, FastV, and Dynamic-LLaVA across multiple benchmarks is presented in the table below.

The improvement in inference efficiency that Dynamic-LLaVA provides is particularly significant on these three generation ability benchmarks compared to previous MLLM token reduction methods (e.g., FastV), which have longer output text lengths. The results indicate that as the output length increases, Dynamic-LLaVA progressively exhibits a significant advantage in terms of token reduction percentage compared to FastV, as FastV only reduces image tokens during the prefill stage.

For instance, when the average output text token length is 2 (VQAv2), the efficiency gains achieved by FastV are similar to those of Dynamic-LLaVA (-74% vs. -79%). However, when the average output text token length is 1555 (ShareGPT4V-VQA-single), the efficiency gains achieved by FastV are significantly lower than those of Dynamic-LLaVA (-20% vs. -57%). Additionally, Dynamic-LLaVA also demonstrates superior performance.

Detailed discussions and results are presented in Appendix A.3.1 and Tab. 8 of the revised paper.

Dear Reviewer UyMq,

Thank you for your valuable suggestions and thoughtful evaluation of our work! We recognize that some parts of the current paper may lead to misunderstandings or confusion. To address this, we have made detailed revisions to the paper, adding more explanations and illustrative figures to clarify the overall working mechanism. Additionally, we have conducted extensive experiments and provided detailed explanations to address your concerns.

Sincerely Yours,

The Authors

W1. Predictor mechanism.

We apologize for any confusion. We clarify the predictor mechanism as follows.

The learnable predictors are applied only once, after the $l$-th decoder layer of the MLLM. Specifically, once the predictor makes the decisions on tokens, this decisions are shared across all subsequent layers. In both the prefill and decoding without KV cache stages, after the predictor reduces the token set $S_l$ to $S_l^*$, all subsequent layers use this reduced token set $S_l^*$ for computation. In decoding with KV cache, the decision of whether to add the generated activations of the current token to the KV cache is likewise propagated to subsequent layers. This one-time token reduction strategy circumvents the need for complex adjustments to sparsity ratios of MLLM decoders. In practice, we follow FastV and set $l$ to 2. The ablation study for $l$ is presented in Tab. 6.

We have added more descriptions of the predictor mechanism in Sec. 3.3.1 and Sec. 3.3.2 of the revised paper.

Dear Reviewer UyMq,

Thank you once again for reviewing our paper and providing valuable feedback. We have carefully considered your suggestions and made multiple revisions to enhance the clarity, depth, and contribution of the paper. Your constructive insights and feedback have played a significant role in the process of improving our paper.

We sincerely hope you will continue to engage in the discussion. Should you have further questions or concerns, we are more than willing to provide additional explanations or supporting materials. Your insights are critical to refining our research and ensuring its relevance and impact.

Furthermore, we hope these revisions and clarifications will encourage you to reassess your evaluation, as these updates directly address your constructive comments. If you have any additional questions or concerns, please feel free to reach out to us. We are committed to ensuring that all issues are thoroughly addressed.

Sincerely Yours,

The Authors

We found that even discarding only 50% of the prefill KV cache, directly applying H2O to an MLLM leads to severe performance degradation (see the second row of the table above). This result is similar to those reported in ''Table 5(g) of the work [5]'' concerning the application of StreamingLLM [12] on MLLMs, indicating that directly applying LLM KV cache compression methods to MLLMs can lead to a sharp decline in performance. Thus, we enhance H2O to start reducing KV cache after the $10$-th layer (Dynamic-LLaVA starts sparsifying after the $2$-nd layer). Despite retaining less KV cache (25% vs. 67%), Dynamic-LLaVA still achieves better performance. Moreover, Dynamic-LLaVA additionally reduces the inference overhead of the MLLM by 75%.

For the generation ability task, the table below shows the performance of H2O and our method. ''FastV-7B [5]+H2O$_{r=0.5}$ [11]'' indicates that the prefill stage is sparsified by FastV [5], and the KV cache during the decoding stage is reduced by H2O.

Applying H2O directly in the generation process of MLLMs leads to a significant degeneration in the fluency and quality of the generated text. Even when H2O is used solely during the decoding stage of MLLMs, Dynamic-LLaVA still able to generate slightly more fluent and higher-quality text while using less KV cache throughout the entire generation process. Consequently, compared to H2O, Dynamic-LLaVA has a slightly better generation ability with higher prefill inference benefits, additional decoding inference benefits (when decoding without KV cache), and stronger vision understanding performance.

Detailed results and settings regarding the performance of H2O can be found in Tab. 1, Tab. 3, and Appendix A.4.2 of the revised paper.

Dear Reviewer LdRV,

Thank you very much for your valuable suggestions! We have conducted extensive experiments and revised our paper accordingly. We also apologise for some misunderstandings you may have had about our paper. We have tried to address these issues in the following rebuttal, and have added further clarifications to the paper to avoid potential misunderstandings.

Sincerely Yours,

The Authors

W1-1. Dynamic-LLaVA is a training-required method.

Currently, not all MLLM token reduction methods are training-free (i.e., they cannot be directly applied to MLLMs without training). For instance, methods such as IVTP [1], PruMerge+ [2], VoCo-LLaMA [3], and TRIM [4] are training-required methods.

Additionally, it is important to note that PruMerge+ [2], which you mentioned, also requires additional post-training. For LLaVA-1.5, both Dynamic-LLaVA$_{I/T}$ and PruMerge+ [2] undergo an extra one epoch of instruct-following finetuning for MLLMs. However, Dynamic-LLaVA$_{I/T}$ outperforms PruMerge+ in 10 out of 12 valid results across LLaVA-1.5-7B and LLaVA-1.5-13B (see Tab. 1 in the main paper), while also offering additional decoding inference benefits.

We present comparison results for FastV, which undergo one additional epoch of sparsification training, maintaining the same hyperparameter settings as Dynamic-LLaVA. The "*" denotes FastV that has undergone additional sparsification training. Detailed results and settings can be found in Tab. 1 of the revised paper and Appendix A.4.2.

It is evident that, after undergoing the same additional sparsification training, the performance of Dynamic-LLaVA$_{I/T}$ significantly surpasses that of FastV.

Dear Reviewer LdRV,

Thank you once again for reviewing our paper and providing valuable feedback. We have carefully considered your suggestions and made multiple revisions to enhance the clarity, depth, and contribution of the paper. Your constructive insights and feedback have played a significant role in the process of improving our paper.

We sincerely hope you will continue to engage in the discussion. Should you have further questions or concerns, we are more than willing to provide additional explanations or supporting materials. Your insights are critical to refining our research and ensuring its relevance and impact.

Furthermore, we hope these revisions and clarifications will encourage you to reassess your evaluation, as these updates directly address your constructive comments. If you have any additional questions or concerns, please feel free to reach out to us. We are committed to ensuring that all issues are thoroughly addressed.

Sincerely Yours,

The Authors

Dear Reviewer UyMq,

Thank you for your thorough review of our paper and rebuttal! We are honored that our rebuttal has addressed most of your concerns and that you have subsequently increased your score. Below, we provide responses to the remaining concerns you may have.

Sincerely Yours,

The Authors

Rationale for adding additional training results for FastV [1].

We included additional training results for FastV in the ''Response to W1-1'' for Reviewer LdRV to provide a comparative analysis with the SoTA method FastV, which apply the same sparse training as Dynamic-LLaVA. This comparison aims to demonstrate the superior performance of our proposed method. As shown in the ''Response to W1-1'' for Reviewer LdRV, Dynamic-LLaVA exhibits significant improvement compared to both the untrained and trained versions of FastV.

Settings we used for FastV

In both the paper and rebuttal, we have kept the settings for FastV consistent. Specifically, we set K=3 and R=0.75, as specified in ''Tab. 1 of FastV [1]''. This corresponds to the factors you mentioned (specific layers pruned, the total number of model layers, and whether multi-head or group-query attention is used), which are as follows:

• Specific layers pruned: The first three decoder layers use all tokens for computation, while layers beyond the third use only 25% of the tokens.
• Total number of model layers: LLaVA-1.5-7B has 32 decoder layers, while LLaVA-1.5-13B has 40 layers.
• Multi-head or group-query attention: We use multi-head attention (consistent with the official LLaVA-1.5 implementation).

For the calculation of FLOPs, the detailed process can be found in Tab. 10 and Appendix A.4.5.

Fairness of the settings for FastV

The settings K=3 and R=0.75 that we used are a powerful configuration for FastV (likely the optimal one), as recommended in Tab. 1 of the FastV paper [1].

Additionally, we explored the performance of FastV reported in other papers (LLaVA-1.5-7B).

We ensure that the settings we used are fair. You can obtain the results we reported by using our Anonymous GitHub code (here) to set ratio = 0.25 (equivalent to modifying the official FastV GitHub repository (here)).

[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV2024.

[2] SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. Arxiv2410.

Dear Reviewer LdRV,

Thank you again for your time and effort dedicated to reviewing our paper. Please let us know if our responses have addressed your concerns. If you have remaining concerns, please don't hestitate to let us know and we are happy to address them. If your concerns have been addressed, we respectfully ask you to consider raising the score.

Sincerely Yours,

The Authors

Dear Reviewer LdRV,

As the discussion deadline is approaching, we would like to kindly follow up to confirm whether our responses have adequately addressed your concerns. We sincerely hope you will participate in the discussion.

Sincerely Yours,

The Authors

Baselines and benchmarks.

We primarily use LLaVA-1.5 [14] as our main baseline, employing the same 12 benchmarks utilized in LLaVA-1.5. This choice is motivated by the fact that most works in the field [1-7] validate their results on these benchmarks for LLaVA-1.5, allowing for a fair comparison by adopting the same settings. To evaluate the generation ability of MLLMs, we use the LVIS-VQA single- and multi-round benchmarks. Our goal is to assess whether Dynamic-LLaVA, while reducing output text tokens, significantly impacts the generation ability of MLLMs. The results indicate that our method enhances inference efficiency with minimal impact on generation performance. In the rebuttal, we will supplement our results with additional baselines and benchmarks.

Writing.

We recognize that excessive notation may hinder broader reader comprehension. In the revised paper, we are incorporating more intuitive visualizations and adding simplified descriptions to improve the readability.

[1] IVTP: Instruction-guided Visual Token Pruning for Large Vision-Language Models. ECCV2024.

[2] Llava-prumerge: Adaptive token reduction for efficient large multimodal models. Arxiv2403.

[3] VoCo-LLaMA: Towards Vision Compression with Large Language Models. Arxiv2406.

[4] Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs. Arxiv2409.

[5] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV2024.

[6] HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments. Arxiv2408.

[7] SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. Arxiv2410.

[8] Language models are unsupervised multitask learners. OpenAI blog.

[9] Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America.

[10] Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. ACL Workshop 2005.

[11] Fast inference from transformers via speculative decoding. ICML2023.

[12] Online speculative decoding. Arxiv2310.

[13] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. NeurIPS2024.

[14] Improved Baselines with Visual Instruction Tuning. CVPR2024.

Distinctions from LLMs' KV Cache Compression Methods.

The proposed Dynamic-LLaVA framework differs fundamentally from LLMs' KV cache compression methods in the following ways:

1. Dynamic-LLaVA introduces a tailored sparsification inference scheme specific to various inference modes, i.e., prefill, decoding with KV cache, and decoding without KV cache. Only in the case of decoding with KV cache can it be regarded as a form of "online KV cache compression". For prefill and decoding without KV cache, where KV cache activations are not involved, Dynamic-LLaVA can still significantly accelerate MLLM's computation, a benefit that KV cache compression methods cannot achieve. We have noted the reviewers' focus on the decoding with KV cache scenario and the classification of our method as a KV cache compression approach. In the revised paper, we will include a more detailed comparison with LLM KV cache compression methods and discuss the computational benefits in the prefill and decoding without KV cache scenarios. This includes the importance of prefill-stage inference acceleration for MLLMs [5] and potential applications for decoding without KV cache, such as the widely used speculative decoding strategy [11, 12].
2. Considering the complete generation process of MLLMs with KV cache, Dynamic-LLaVA achieves computation and GPU memory savings in the prefill stage, as well as GPU memory savings during decoding (disregarding computational acceleration from KV cache reduction for extremely long sequences [13]). In contrast, typical KV cache compression methods generally offer only GPU memory savings during decoding. This distinction arises because Dynamic-LLaVA evaluates each current token's features to determine whether its activations through $W_K$ and $W_V$ should be added to the KV cache, rather than removing entries from the past cache (detailed in Eq. 6 of the main paper). KV cache compression methods, however, often rely on past generated KV cache. E.g., H2O [13] uses the current Q with the past KV cache in decoding to compute attention scores and decide which cache activations to discard, rather than directly deciding to retain or discard activations for the current token. We will expand upon this discussion in the upcoming revision of the paper.
3. Dynamic-LLaVA is an MLLM inference acceleration framework that considers the distinct properties of different modalities and incorporates tailored sparsification strategies accordingly. As noted in [5], “While these methods have boosted the inference efficiency of LLMs, they are designed for text-only language models, and whether their effectiveness can be transferred to LVLMs remain under-explored.”. Thus, the direct applicability of LLM KV cache compression methods to MLLMs is yet to be verified. In our detailed rebuttal, we will provide results on the application of LLM KV cache compression methods in MLLMs.

Predictor mechanism.

The predictor used in Dynamic-LLaVA is applied only once, after the l-th decoder layer of the MLLM. Specifically, once the predictor makes a decision on tokens, this decision is shared across all subsequent layers. For instance, in both the prefill and decoding without KV cache stages, after the predictor reduces the token set $S_l$ to $S_l^*$, all subsequent layers use this reduced token set $S_l^*$ for computation. In decoding with KV cache, the decision of whether to add the current token to the KV cache is likewise propagated to subsequent layers. Following [5], we set l=2, meaning that token reduction occurs after the 2nd layer. Ablation studies on the position of l are provided in Tab. 6 of the main paper.

As for the additional inference overhead introduced by the predictor, we first present theoretical FLOP values. For LLaVA-1.5-13B, the predictor's FLOPs in the prefill stage are under 0.01T, compared to the MLLM’s 19.6T FLOPs. In Table 4 of the main paper, we also report latency measurements on a real GPU for prefill and decoding without KV cache stages. During the decoding with KV cache phase, the predictor's computational overhead is similarly minimal and can be considered negligible. We will provide further details in our comprehensive rebuttal and the revised paper.