$\gamma-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models

Comment 4: The author should consider rearranging Table 5 or highlighting the optimal results, as the current version is somewhat challenging to compare the results of different methods.

Response: Thanks for your kind reminder. We will carefully adjust the layout and format of Table 5 to improve its readability.

Comment 5: In this paper, the binary mask Mq assigned the question tokens to 0. Therefore, during the training process, the value of Mq should be fixed. But for different tasks or datasets, the location of the question tokens may be different. How did the author determine and design Mq on different tasks or datasets?

Response: In practice, the positions and lengths of questions in a batch are different, so we assign corresponding binary masks to them. Therefore, $M_q$ will be changed based on the input sample.

Comment 6: I would like to know how the "All layer" in Table 1 is specifically implemented.

Response: "All layer" means that all dense layers are replaced with the MoD ones.

Comment 7: How sensitive is ARank to the choice of hyperparameters, such as the batch used for estimation?

Response: Actually, ARank is highly stable to the change of tasks and samples, and a small batch of samples (50 in our paper) can already accurately estimate its value. In Fig 3, we visualize the impact of tasks and batch size on ARank, which may help you further understand the robustness of ARank.

Comment 8: What are the criteria for determining the threshold for converting dense layers to MoD layers based on ARank values? Is there a more principled way to set this threshold?

Response: In $\gamma$-MoD, the threshold is still a hyperparameter that needs to be manually selected, and this setting in our paper is sufficient to validate the effectiveness of $\gamma$-MoD. However, as you suggested in Comment 2, the adaptive threshold would be a better alternative, which we will further explore in future work.

Comment 1: The metric ARank seems to be a repurposed version of HRank [1] specifically for attention maps. This connection should be discussed in detail.

Response: Thanks for your feedback. ARank is inspired by HRank [A] but still quit different in theoretical and practical contributions. In particular, HRank aims to use the rank of feature maps to estimate the weight redundancy for network pruning. In contrast, we are the first to validate that the layer redundancy can also be estimated by the rank of their attention maps, thus guiding the deployment of MoD. Compared to HRank, we have extended the theory and the practical use of the ''rank''. Based on your advice, we will add these discussions in our new revision.

[A] Hrank: Filter pruning using high-rank feature map, CVPR 2020

Comment 2: The ablation study of the routing mechanisms is not enough. This paper only explores a binary routing mechanism (skip or not skip). Exploring more sophisticated routing strategies, such as weighted routing or adaptive routing thresholds and so on, could make the article more comprehensive.

Response: Thanks for this constructive advice. In fact, binary routing mechanism could best meet the autoregressive property of MLLMs, where the latter tokens cannot see the former ones. Therefore, it's pretty difficult to weighted aggregate several tokens into one, which may break the autoregressive property.

However, we do agree that the adaptive routing thresholds should be a promising approach. Driven by this, we consider two adaptive metrics based on the sequence length and the ARank, defined by $\delta_s*(1+ \log(\frac{seqlen}{avgseqlen}))$ and $\delta_s*(1+ \log(\frac{ARank}{avgARank}))$ , respectively. Experimental results show the adaptive metrics can slightly benefit the model performance on multiple benchmarks. As you suggested, these results would be added in our final version.

Comment 3: While the paper compares γ-MoD with dense and MoE-based MLLMs, it lacks a comparison with other sparsity techniques like pruning or quantization.

Response: We fully understand your concerns and have compared $\gamma$-MoD with token pruning method (Fast-V [B]) and quantization method (AWQ [C]) in the table below. In this table, we test the speed of 4-bit model (awq-4bit) using the lmdeploy toolkit [D], while others are evaluated using Pytorch. From these results, we still observe the advantage of $\gamma$-MoD in performance and efficiency over pruning and quantization, especially in reducing the next-token time.

We believe that these comparisons is highly beneficial to our paper, and will update them in our final version.

[B] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models, ECCV24

[C] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, MLSys 2024

[D] LMDeploy: A Toolkit for Compressing, Deploying, and Serving LLM

Thank you for your prompt reply! As pointed out in the paper, our $\gamma-$MoD is used for the instruction fine-tuning stage, so the pre-trained weights of the first stage can be directly reused in the publicly released stages (e.g. https://huggingface.co/YanweiLi/MGM-Pretrain). This means that we can start deploying $\gamma-$MoD based on the existing publicly available first stage pre-trained weights, thus skiping the pre-training stage. Actually, we also adopt this way in our experiments.

Note that we miss these details in our paper, so feel sorry for any misleading. We will carefully update these details in final version. We hope these further explanations can resolve your concerns and look forward to more discussions.

1. $\alpha$ actually defines the number of positive labels used to calculate the cross entropy loss. For example, if we have a sequence length of 100 and $\alpha$ is 0.5, there will be 50 labels assigned to 1 for routing learning.
2. We double-check the equoation, and confirm that you are correct. Sorry for the obvious typos. We'll double-check it one more time.

Comment 5: Some ambiguities in the methodology: Key information are missing from the article, some statements are quite confusing.

Question 1: What is the applied value of the pre-defined sparse target in Equation 4? What is the relationship between the sparse target α and the "routing ratio" and the "skip ratio" in the experiment section?

Response: The pre-defined sparse target $\alpha$ is the routing ratio that the router is required to approximate during training, e.g., 0.5 for $\gamma-$MoD-0.5. During inference, the actual routing ratio may be slightly different to the training target $\alpha$ .

Question 2: Normally, the sparse target factor α in Equation 4 satisfies α∈[0,1] , but α is actually the sum of multiple indicator functions with values {0,1}. The left side (value of the sum of indicator function) should be normalized.

Response: Yes, α should satisfy $\alpha \in$ [0,1]. We apologize for the lack of normalization in Equation 4 and will carefully revise it in the new version.

Question 3: In Equation 8, the authors put one-hot vectors R^ and 1−R^ inside log. This obviously doesn't seem right. Moreover, it is hard to understand what does the routing objective (Equation 8) do in γ-MoD training. The authors should provide a more detailed explanations of that learning objective.

Response: We would like to thank you again for these careful reviews. Equation 8 defines a binary cross-entropy loss for routing learning, which encourages that the router can learn to discard unimportant tokens based on a pre-defined sparse target $\alpha$. Our routing objective is equivalent to that of the original MoD[B]. Based on your feedback, we will carefully revise the description for it to avoid misleading.

Comment 3: Further generalizability of γ-MoD: The experiments in this article primarily focus on LLaVA structures, likely due to the availability of open-source pre-training and instruction tuning datasets, which are necessary for γ-MoD as it modifies the instruction tuning process. However, as a general method for introducing the MoD scheme to MLLM models, additional experiments on other MLLMs are needed to further demonstrate the generalizability of γ-MoD.

Response: We appreciate this constructive advice. Following your suggestion, we adopted γ-MoD to another popular MLLM termed Mini-Gemini [B] in the table below, and validated the generalizability of γ-MoD on this new MLLM architecture.

Source codes and pre-trained weights are also anonymously released at: https://anonymous.4open.science/r/Gamma-MOD-code-AC17. As the first MoD-based work for MLLMs, we sincerely hope that our open-source codes will benefit future works in the community.

[B] Mini-gemini: Mining the potential of multi-modality vision language models, arXiv preprint arXiv:2403.18814, 2024.

Comment 4: Significance when compared to existing works: In the experiments section, this article finds that most image tokens can be skipped in MLLM inference. This discovery corresponds with [C], in which image tokens are simply pruned from off-the-shelf MLLMs without further training. experiment results in [C] indicates that this simple procedure reduces the FLOPs in MLLMs by over 50% without introducing performance drops, similar to that reported in γ-MoD. If so, pre-train MLLMs from scratch and modify the instruction tuning methods would be too costly for accelerating an MLLM.

Response: Thanks for this comment. We fully agree that [C] is a promising approach to directly reduce the computational cost of pre-trained MLLMs, and we will carefully discuss and compare it in the final version. Nevertheless, as an inference-time acceleration method, [C] is quit orthogonal to the contribution of $\gamma-$MoD. In terms of significance, we summarize the differences in two aspects:

1.Efficiency during next-token generation: [C] mainly aims to reduce the number of visual tokens in the pre-filling stage. In stark contrast, $\gamma-$MoD can skip both visual and textual tokens in both the pre-filling and next-token generation stages. As shown in the table below, the next-token generation time often accounts for a large proportion of the inference time, and $\gamma-$MoD can reduce the time of this stage by up to 58%.

2. Training efficiency: Instruction tuning is an an inevitable process for deploying a new MLLM. As an adaptation approach, $\gamma-$MoD can further reduce the training time by up to 31% (see Tab 4), which is indeed practically useful (as recognized by Reviewer 1ATp).

Based on above discussions, we sincerely hope that you would reconsider the scope and contribution of our paper.

[C] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (ECCV 2024)

Comment 1:Method designs: In the introduction (Line 87), this article suggests that γ-MoD is a plug-and-play method that can be integrated to current MLLMs. Existing MLLMs follow a 2-stage pre-training and instruction tuning training pipeline, and γ-MoD modifies the instruction tuning pipeline to convert LLaVA models in γ-MoD-LLaVA structure. In my opinion, this design is arguably plug-and-play, since open-source MLLM weights are mostly instruction tuned, so that γ-MoD cannot be directly applied on published MLLM checkpoints. To evaluate the validity of γ-MoD, the authors also pre-trains MLLMs on the LCS-558K dataset first. This hinders the applicability of γ-MoD, as training the γ-MoD model also requires conducting the complete pre-training pipeline.

Response: Thanks for this comment. We would like to clarify that $\gamma$-MoD is not a plug-and-play inference module but a plug-and-play adaptation method that can be seamlessly integrated into the standard training process of MLLMs. In fact, $\gamma$-MoD is not used for a well pre-trained MLLM, but serves for a common situation that we want to train a new MLLM in an efficient way. Considering for this, the strengths of $\gamma$-MoD focus on both training and inference efficiency, see Tab 4 (+31% and +53% speedups by $\gamma$-MoD for training and inference, respectively). However, we do agree that the description of plug-and-play is not rigorous enough, and we will carefully revise it in the final version.

Note that one detail we would like to kindly clarify that the pre-training process is not actually needed for $\gamma$-MoD, since $\gamma$-MoD is deployed in the instruction tuning stage. In practice, we use the published pre-training weight in our experiment, please see our anonymously released code: https://anonymous.4open.science/r/Gamma-MOD-code-AC17.

Comment 2: Regarding Weakness 1, since the authors consider the disadvantage of MoD to be demanding training from scratch (Line 49-50), the authors should demonstrate how to apply γ-MoD to existing published MLLM checkpoints, rather than pre-training one from scratch like in the experiment section.

Response: Thanks for this kind advice. In Line 49-50, we stated that recent MoDs[A] require to pretrain LLMs from scratch, instead of MLLMs. Therefore, our motivation is that how to directly adapting MoDs to MLLMs without the expensive pre-training of a new LLM. In this case, directly applying γ-MoD to existing published MLLM checkpoints is indeed orthogonal to our contribution, as discussed in Comment 1.

However, we fully respect your concerns and have attempted to apply γ-MoD to published MLLM checkpoints in the table below (Direct $\gamma$-MoD). In particular, we modified the trainable router to an rule-based one, which skips tokens based on its attention scores. As expected, the Direct $\gamma$-MoD can achieve promising results on some benchmarks like SQA, but is still inferior to our Default $\gamma$-MoD on challenging benchmarks like TextVQA. Based on your suggestion, we will further clarify our motivation and contribution in the final version.

[A] David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024.

Dear Reviewer 6GUt:

We highly appreciate your recognition to our response, and your insightful reviews have greatly contributed to improving the clarity and overall quality of our work. Now we have updated the revised paper based on your and other reviewers' concerns. The main modifications include:

1. Detailed discussions and comparisons with FastV in the related work (Line 148-152) and Tab 5. (Reviewer 6GUt)
2. Generilization results on Mini-Gemini in Tab 3. (Reviewer 6GUt)
3. Clarifications about the pre-training stage in Line 340-346. (Reviewer 6GUt)
4. Clarifications about the plug-and-play adaptation module in Line 340-346. (Reviewer 6GUt)
5. Results of adaptive thresholds in Tab 1. (Reviewer wmRq)
6. Discussions of differences with HRanks in Line 297-300. (Reviewer wmRq)
7. Comparison with quantization and pruning methods in Tab 5. (Reviewer wmRq)
8. Other details: typos like mistake equoations and highlights of the best value in the table, etc. (Reviewer 6GUt , wmRq)
9. Clarifications for the robustness of ARank in Line 269-304. (Reviewer 1ATp)

If issues are well addressed in our new revision, we sincerely hope you could kindly raise the score.

Best,

The Authors

Comment 1: ARank’s accuracy heavily depends on the diversity and representativeness of the input samples, which may lead to inaccurate redundancy assessment, especially in varied multimodal tasks.

Response: Thanks for this professional comment. We fully agree that, as a sample-dependent metric, the estimation of ARank will fluctuate as samples change. However, when we use a small batch of samples to estimate ARank, the variance of ARank is indeed negligible for the determination of the MoD layer. Figure 3 in our paper strongly demonstrates this property, where the Arank values of different layers are almost unchanged regardless of samples and tasks. In this case, ARank could be a stable metric for the choice of MoD layers.

To further eliminate your confusion, we would like to provide the layer-wise ARank values of different tasks in the table below, which further suggests the robustness of ARank.

Comment 2: ARank’s reliance on low-rank attention maps to identify redundancy may overlook the contextual importance of low-rank layers, which could contain compressed but critical information.

Response: Thank you for this insightful advice. ARank is defined from the perspective of layer redundancy, so it does not consider the impact of contextual information. However, ARank seems to be already a good indicator for preserving important contextual content. As shown in Figure 4, the visual contents relevant to the question are well preserved while most of the unimportant contents are discarded.

Nevertheless, we still agree that the consideration of contextual information will be highly beneficial to our methods. In fact, we have already adopted a context-preserving strategy in our method, namely masked routing learning, which aims to preserve significant context (textual instruction) in MoD layer. Based on your suggestions, we will explore more strategies to combine the context importance with $\gamma-$MoD.

Comment 3: How does γ-MoD ensure that ARank’s redundancy metric accurately reflects the importance of layers across diverse multimodal tasks, especially given the unique challenges posed by visual information?

Response: Thanks for this comment. As discussed in Comment 1, although ARank is a sample-dependent metric, it is quit stable across samples and tasks. We think that ARank may reflect the inherent redundancy of a given pre-trained model, so that it does not change significantly across samples and tasks. Our visualizations of Fig 3 and the effectiveness of ARank on 9 multimodal benchmarks have reflected this property. We hope these discussion can further help you understand the robustness of ARank.

Dear Reviewer 1ATp,

Thanks again for your great efforts and constructive advice in reviewing this paper! After reading our rebuttals, Reviewer 6GUt and wmRq have raised their scores to 6 and 8, respectively. (Reviewer 6GUt also gives the highest score for soundness. ) To date, all of your and other reviewer's concerns are updated to our new revision. And our source codes and pre-trained weights are anonymously released at: https://anonymous.4open.science/r/Gamma-MOD-code-AC17.

With the discussion period drawing to a close, we are particularly concerned about whether our revisions adequately address your concerns，and sincerely hope to resolve all potential issues to earn a more positive rating from you.

Best regards!