A Framework of Distilling Multimodal Large Language Models

We sincerely appreciate your constructive comments and suggestions. Thank you for your efforts in assessing and helping to improve the quality of our manuscripts. Below are our responses to your concerns:

Q4.1 Some notations are confusing

1. In Eq.(1), $\phi_l (y_{m} \mid \mathbf{y}_{<m})$ is a conditional probability distribution that describes the distribution of the random variable $y_m$ given the previous observations;

2. Your understanding is correct

• $\phi_{l}\left(Y_{j} \mid \mathbf{y}_{<m}\right)$ is a probability.
• $\phi_{s}(y_m \mid \mathbf{y}_{<m})$ is a distribution.

For discrete distributions $P$ and $Q$, the definition of KLD is $\text{KLD}(P | Q)=\sum_{i} P(i) \log \left(\frac{P(i)}{Q(i)}\right)$. The Eq.(2) indicates that when calculating KLD, we sum the probabilities of each word in the vocabulary, where

• $\phi_{l}\left(Y_{j} \mid \mathbf{y}_{<m}\right)$ represent the probability of the $j$-th word $Y_j$ in the vocabulary at position $m$ for the teacher model,
• $\phi_{s}\left(Y_{j} \mid \mathbf{y}_{<m}\right)$ represent the probability of the $j$-th word $Y_j$ in the vocabulary at position $m$ for the student model.

3. Thank you for your review. $L_{PT}$ should be revised to $L_{reg}$ in Eq.(1), representing the autoregressive training objective. We have made this correction in the revised version.

Q4.2 Can you provide the result of PT+SFT+DFT to further validate the effectiveness of DPT?

We additionally supplement the experiments with PT+SFT+DFT, and the results demonstrate that DPT+SFT+DFT surpasses PT+SFT+DFT by 1.3%, proving the effectiveness of the DPT stage.

The equivalence in final performance between DPT+DFT and PT+DFT may be attributed to the fact that the distillation process in the DFT stage plays a crucial role in both combinations, bringing the model to a similar level of comprehension ability. Specifically, although DPT provides better multimodal mapping in the initial stage, the powerful capability of DFT compensates for the shortcomings of the PT stage, resulting in a final performance comparable to DPT+DFT and PT+DFT.

Furthermore, the performance of DPT+SFT is significantly better than PT+SFT (Table 2 in the subscription), which demonstrates that DPT, compared to PT, is more conducive to MLLMs achieving the mapping from the visual space to the text space. Consequently, in the SFT stage, without the guidance of a teacher network, s-MLLM can optimize from a higher baseline, enhancing the model's final performance.

Q4.3 In Section 3.3, “we further design a three-stage distillation scheme (Sec. 3.3.1)”, (Sec. 3.3.1) should be (Sec. 3.3.2).

Thank you for your careful review. We have made corrections to the citations in the revised paper.

We sincerely appreciate your constructive comments and suggestions. Thank you for your efforts in assessing and helping to improve the quality of our manuscripts. Below are our responses to your concerns:

Q3.1 It appears that the methods do not present particularly novel or distinctive contributions to the field. Therefore, systematic experimentation would be essential to demonstrate clear superiority of the proposed methods.

Thank you for your careful review. We hope to further clarify our motivation and the comparison with other methods:

• Motivation: Knowledge distillation has not been fully explored in the training of $s$-MLLMs. Therefore, we believe that exploring effective training strategies based on knowledge distillation is timely and necessary, providing new ideas for the community to develop small models.

• Comparison with Existing Work:

  • Some existing works have already explored knowledge distillation in MLLMs. For example, LLaVADI experimentally verified that using teacher models for data augmentation is beneficial for enhancing the capabilities of s-MLLMs; LLaVA-MoD focused on improving the structure of s-MLLMs and aimed to address the hallucination ability of the models.
  • Our approach focuses on improving the training strategy. We base our method on the most common training data, without the need for additional data augmentation or complex model structure design, aiming to enhance the performance of s-MLLMs through improvements in the training strategy.

• Experimental Improvements: To better demonstrate the superiority of our method, we have further supplemented and refined the experimental section according to your suggestions. Please refer to the subsequent responses for detailed information.

Q3.2 It fails to conduct comparative experiments against the Preference Distillation technique proposed in LLaVA-MoD.

Thank you for your comments. To address your issue, we replace the DFT stage in our DPT+SFT+DFT training stages with Mimic Distillation (MD) and Preference Distillation(PD). As illustrated in the following table, when employing the DPT+SFT+MD strategy, our method demonstrates an average improvement of 1.0%. However, when PD is additionally incorporated, the performance decreases to 55.1%, with our method showing an advantage of 2.8%.

Our finding is consistent with LLaVA-MoD: "Preference distillation remarkably reduces the hallucination of s-MLLM, while it does not yield consistent improvements in comprehension capability." It appears that PD primarily addresses the model's hallucination ability, but this also results in a decline in the model's comprehension capability.

Q3.3 Additionally, the paper lacks discussion and comparative experiments with LLAVADI [2], another concurrent work on MLLM distillation.

Thank you for the valuable comments. We have discussed LLaVADI in the related work section of the revised paper and added a comparison with LLaVADI in Table 1.

Regarding the experimental comparison, we would like to state that since LLaVADI has not been open-sourced and due to the lack of some experimental details (such as the hyperparameters of the weights of different training objectives during training, the source of data when using teacher models for data augmentation, etc.), it is quite difficult to reproduce their results. Therefore, we only compare the final performance of the two models.

We report the average of the six benchmarks reported in LLaVADI. It can be seen that our 2B model outperforms their 3B model by a margin of 0.9%; our advantage is more pronounced with 1B parameters, showing a margin of 4.2%.

Q3.9 A study on hyperparameters (loss weights) seems necessary.

In our study, the hyperparameters mainly involve the weights of $L_{res}$, $L_{vis}$, and $L_{rel}$ in the training objectives $L_{DPT}$ and $L_{DFT}$ during the DPT and DFT stages, including ${\alpha, \beta, \gamma}$ during DPT and ${\alpha^\prime, \beta^\prime, \gamma^\prime}$ during DFT. We default to set $\alpha=\alpha^\prime=1.0$, $\beta=\beta^\prime=1.0$, and $\gamma=\gamma^\prime=0.5$.

To further verify the impact of hyperparameter adjustments on the model, we conducted several sets of hyperparameter combination experiments. As shown in the following table, using different hyperparameter combinations, the final experimental results show little difference in average performance. This indicates that our model is robust to the choice of hyperparameters.

We would like to state that our goal is not to find the optimal hyperparameters, but rather to use unified parameters to ensure the robustness of the model. However, this can be a direction for future improvement, such as adaptively balancing the weights to achieve the optimal model results.

Q3.4 A direct performance comparison with existing LLM/MLLM distillation methods under the same experimental settings would be more convincing.

It is indeed necessary to validate the effectiveness of our distillation strategy under the same experimental settings. To be more convincing, we have conducted direct comparisons with existing LLM and MLLM distillation methods under the same experimental settings. Specifically:

• For the comparison with MLLM distillation methods, please refer to our response for Q3.6.
• For the comparison with LLM distillation methods, please refer to our response for Q3.7.

We hope these additional experiments and comparisons can furthur demonstrate the effectiveness and advantages of our method. Thank you again for your suggestions.

Q3.5 Why does this paper emphasize that LLaVA-KD's performance is better than BLIP series?

Thank you for your comments. According to your suggestions and Q2.2 raised by Reviewer sLGE, we agree that it is unfair to compare with the BLIP series models under the condition of using higher quality data for training. Therefore, we have removed the statement regarding the comparison with BLIP2 and revised the relevant parts in the revised paper.

Moreover, we would like to emphasize that, compared to recent advancements in some s-MLLMs, including Imp, Moe-LLaVA, LLaVA-MoD, etc., LLaVA-KD still demonstrates superiority.

Q3.6 Can we directly compare the performance of the proposed method with the distillation methods presented in LLaVA-MoD and LLAVDI under the same experimental settings?

LLaVA-MoD: To compare with the distillation method proposed by LLaVA-MoD under the same experimental settings, we first attempt to replace the DFT stage in the DPT+SFT+DFT training stages with mimic Distillation and preference distillation. As shown in the following table, when we replace our DFT with Mimic Distillation, LLaVA-KD achieves a 1.0% improvement. The additional introduction of the preference distillation training stage results in a further decline in performance, which is consistent with the description in the original LLaVA-MoD paper:"Preference distillation remarkably reduces the hallucination of $s$-MLLM, while it does not yield consistent improvements in comprehension capability." In this way，LLaVA-KD achieves a 2.8% improvement. These results further demonstrate the effectiveness of our distillation algorithms.

Additionally, we also transfer the Pre-Training (PT) + Mimic Distillation strategy of LLaVA-MoD to our LLaVA-KD method. As shown in the following table, it can be seen that our proposed training strategy demonstrates significant advantages.

LLaVADI: Since this paper has not been open-sourced and some important experimental details, such as loss weights and how the teacher model generates instruction-tuning data, have not been provided, reproducing this method presents significant difficulties. We hope the reviewers understand that our research focuses on exploring suitable training strategies for MLLMs, and we have conducted a fair comparison with the latest method, LLaVA-MoD, under the same experimental settings, demonstrating the superiority of our distillation algorithms and training strategies.

Q3.7 How does the performance compare when directly applying the LLM distillation methods mentioned in Section 2.2's 'KD for LLM' paragraph to MLLM distillation?

In LLaVA-KD, we focus on discussing the impact of distillation algorithms and training strategies on s-MLLMs and propose a three-stage training strategy along with the MDist and RDist distillation algorithms. Existing LLM training strategies mainly focus on studying the application of standard KL loss in LLM distillation and improving it. Referring to previous work LLaVADI, which also explores the application of LLM distillation algorithms in MLLMs, we followed the training strategy of PT + DFT (Distilling Response logits with Standard KL and Reverse KL loss [1]).

As shown in the following table, we find that the advancements in LLMs seem less applicable to MLLM distillation. This finding is consistent with the conclusion of LLaVADI.

At the same time, by comparing LLaVA-KD's PT + DFT (Row 2) with the above results, LLaVA-KD demonstrates superior performance on 9 benchmarks, with an average advantage of 0.8. This also reveals the effectiveness of our proposed MDist+RDist distillation algorithms in MLLM distillation.

Finally, adopting our three-stage training strategy achieves the best performance, which highlights the effectiveness of the training strategies we explored.

Q3.8 What is the performance of PT+SFT+DFT? The comparable performance between PT+DFT and DPT+DFT raises questions about the necessity of DPT over PT.

The following table compares the performance of DPT+SFT+DFT with PT+SFT+DFT. We can observe that DPT+SFT+DFT surpasses PT+SFT+DFT by 1.3%.

Regarding the equivalence in final performance between DPT+DFT and PT+DFT, we believe this may be attributed to the distillation process in the DFT stage. This stage plays a crucial role in both combinations, bringing the model to a similar level of comprehension ability. Specifically, although DPT provides better multimodal mapping in the initial stage, the powerful capability of DFT compensates for the shortcomings of the PT stage, resulting in a final performance comparable to DPT+DFT.

Secondly, we notice that the performance of DPT+SFT is significantly better than PT+SFT (Table 2 in the subscription). This demonstrates that DPT, compared to PT, is more conducive to MLLMs achieving the mapping from the visual space to the text space. Consequently, in the SFT stage, without the guidance of a teacher network, s-MLLM can optimize from an elevated baseline, enhancing the model's final performance.

$1$ Gu, Yuxian, et al. "MiniLLM: Knowledge distillation of large language models." The Twelfth International Conference on Learning Representations. 2024.

Q2.4 The large MLLM used in this study has only 4B parameters, which may be insufficient to draw conclusions about scaling.

Thank you for your valuable comments. To address your concern, we attempted to use a 7B teacher model for distillation. As shown in the following table, we found that using the 7B teacher model for distillation resulted in an average performance drop of 0.5%. This observation aligns with the findings in [1], which claim: (1) "With increasing teacher size, ..., the trained student accuracy first increases and then decreases," and (2) "The teacher becomes so complex that the student does not have sufficient capacity or mechanics to mimic its behavior despite receiving hints." Therefore, when there is a significant size disparity between the teacher and student models, it can potentially hinder the transfer of knowledge. We believe that selecting an appropriate teacher model is crucial, rather than merely pursuing larger model sizes.

$1$ Mirzadeh, Seyed Iman, et al. "Improved knowledge distillation via teacher assistant." Proceedings of the AAAI conference on artificial intelligence. 2020.

Q2.5 It is also unclear whether distilling a 1B model to another 1B model would be effective within this framework.

Generally speaking, the core objective of knowledge distillation is to transfer the knowledge of a large model to a smaller model, in order to maintain model performance as much as possible while reducing computational resource requirements. Typically, this means using a model with a larger number of parameters (e.g., 7B parameters) to distill a model with a smaller number of parameters (e.g., 1B parameters). If the two models have the same number of parameters (both 1B), then the main advantages of distillation cannot be realized.

Currently, most successful distillation cases[2][3] are achieved in the transition from large models to small models, whereas in cases with the same number of parameters, the effects of distillation are usually not significant.

We would like to further understand what the aim to verify with the 1B to 1B distillation. If you could provide more background information or specific research motivations, we would greatly appreciate it and could conduct more targeted discussions and experiments.

Q2.6 The authors do not report results for small MLLMs (0.5B and 1B) trained with the standard LLaVA procedure.

In fact, TinyLLaVA (Qwen1.5-0.5B) and TinyLLaVA (Qwen1.5-1.8B) in Table1 are small MLLMs trained using the standard LLaVA procedure, following the Pre-Training and supervised Fine-Tuning strategy. Specifically:

• With 1B parameters (using Qwen1.5-0.5B as the LLM), our method brings an improvement of 3.7% (57.3 vs. 61.0).
• With 2B parameters (using Qwen1.5-1.8B as the LLM), our method brings an improvement of 5.9% (59.3 vs. 65.2).

These results indicate that our method can enhance performance across models of different scales.

$1$ Tan, Shicheng, et al. "Gkd: A general knowledge distillation framework for large-scale pre-trained language model." arXiv preprint arXiv:2306.06629 (2023).

$2$ Gou, Jianping, et al. "Knowledge distillation: A survey." International Journal of Computer Vision 129.6 (2021): 1789-1819.

$3$ Zhao, Borui, et al. "Decoupled knowledge distillation." Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022.

We sincerely appreciate your constructive comments and suggestions. Thank you for your efforts in assessing and helping to improve the quality of our manuscripts. Below are our responses to your concerns:

Q2.1 The rationale behind optimizing the similarity matrix of visual tokens between large and small MLLMs is unclear.

Relation Distillation of visual tokens is aimed at facilitating $s$-MLLMs to learn the responses of $l$-MLLMs to visual tokens, thereby enhancing the s-MLLMs' understanding of multimodal information.

Our motivation is based on the observation that in order to fully understand multimodal information, MLLMs need to capture the complex relationships between different visual tokens in an image. For example, identifying the relationship of a cat lying under a table in an image, or the positional relationships between person. These all require the model to recognize and understand the interactions between visual tokens.

By optimizing the visual token similarity matrix of the small model, we can explicitly transfer the relational information captured by the large model to the small model. This process enhances the small model's ability to process visual information, thereby improving its performance in multimodal tasks.

Q2.2 The authors' performance claims are misleading.

Thank you for your suggestions and viewpoints. We agree that it is unfair to compare with the BLIP series models when trained with higher quality data. Therefore, we have removed the statements regarding the comparison with BLIP2 and made corresponding modifications in the relevant sections.

In addition, we would like to emphasize that, compared to recent advancements in some $s$-MLLMs, including Imp, Moe-LLaVA, LLaVA-MoD, etc., LLaVA-KD still demonstrates superiority.

Q2.3 The performance of LLaVA-KD does not surpass that of LLaVA-MOD.

According to the test results reported by LLaVA-MoD in their paper (see Table 10 of the LLaVA-MoD paper), we conducted a comparison on the same benchmarks. We observed that when using teacher and student models of the same scale, our method demonstrates notable improvements. Specifically, for the 1B and 2B student models, our average results across seven benchmarks indicate an advantage of 1.4 and 1.5, respectively.

Furthermore, we would like to emphasize that the training data of LLaVA-MoD is nearly 3.8M more than our method (5M vs. 1.2M). Despite this, our method still demonstrates superior performance, which is sufficient to prove our superiority.

Dear Reviewer sLGE

We sincerely appreciate your time and effort. We have carefully considered your suggestions and have responded to each of your questions.

As the Discussion phase is about to close, we would like to know if we have addressed your questions and concerns so far. We will continue to address your concerns until the Discussion phase deadline. If you could consider raising your evaluation of our paper after reviewing our responses, we would be very grateful.

Thank you very much for your consideration.

Dear Reviewer sLGE,

Thank you for your time and valuable feedback. As the discussion phase is nearing its end, we remain open to addressing any remaining questions or concerns. We would greatly appreciate it if you could consider improving the evaluation after reviewing our responses. Thank you very much for your consideration.

Sincerely, Paper #686 Authors

Dear Reviewer sLGE,

Thank you for your time and valuable feedback. As the discussion phase is nearing its end, we remain open to addressing any remaining questions or concerns. We would greatly appreciate it if you could consider improving the evaluation after reviewing our responses. Thank you very much for your consideration.

Sincerely, Paper #686 Authors

We sincerely appreciate your constructive comments and suggestions. Thank you for your efforts in assessing and helping to improve the quality of our manuscripts. Below are our responses to your concerns:

Q1.1 If models like Imp-2B can perform similarly without a pre-trained l-MLLM, what justifies the need for computationally intensive methods like the one proposed?

Thank you for your comments. We would like to clarify three points:

1. In terms of performance comparison, Imp used 1M training data, which is 1.5 times the amount of LLaVA-KD training data. However, LLaVA-KD still demonstrated nearly a 1.0% performance advantage on 9 commonly used benchmarks. This indicates that LLaVA-KD exhibits stronger overall multimodal capabilities.
2. In terms of training data comparison: Imp-2B indeed performs well under the same scale of small models, but its performance is somewhat dependent on an additional 300k unavailable data.
3. In terms of computational resources, compared to the existing recent MLLMs methods based on knowledge distillation like LLaVA-MoD (16 A100(80G)), our method requires relatively lower computational resources (8 L40s(48G)).

To prove our point, we trained the Imp-2B model based on the LLaVA1.5 dataset (the same as ours) according to the official code, as shown in the following table. We found that: (1) When using the LLaVA1.5-665k dataset for SFT, Imp-2B only showed an average performance of 63.7%, which is a 0.8% decrease compared to the original Imp-2B model. This indicates that the 300K training data contributed to the performance improvement of Imp-2B. However, this part of the data is currently not open-sourced, which is not conducive to further exploration by the community based on this method. In contrast, our method is based solely on the most common LLaVA training set, demonstrating greater flexibility in data dependency. (2) When trained on the same data, LLaVA-KD achieves notable performance, showing a 2.1% advantage.

In summary, our method not only outperforms Imp-2B in terms of performance but also does not rely on unavailable high-quality data and requires relatively lower computational resources. We believe that our method achieves a good balance in terms of performance, training data requirements, and computational resource requirements, providing a more practical solution for the development of s-MLLMs.

Q1.2 Further discussion on distilling visual tokens

In LLaVA-KD, distilling visual tokens encompasses two aspects. One is $L_{vis}$ in the MDist, where $s$-MLLM directly mimics the output of $l$-MLLM's visual tokens by constraining the KLD. The other is $L_{rel}$ in RDist, where $s$-MLLM simulates the relationship matrix of visual tokens from $l$-MLLM. As shown in the table below, we can observe that distilling visual tokens brings an average improvement of 0.7% during the DPT+SFT phase, and an additional 0.7% during the DFT phase, which means that visual token distillation results in a total performance improvement of 1.4% across the two distillation phases.

Furthermore, the method of distilling visual tokens achieves performance improvement without significantly increasing computational resources. In contrast, other methods may require additional computational resources; for example, according to the Table1 in the Imp, the method adds 300k training data, resulting in a 1.1% performance improvement for Imp-3B.

Q1.3 A comparison with all models of similar size. Additionally, it is unclear why certain datasets, such as VQAv2, POPE, and MMMU, were excluded in Table 5.

In the original Table 5, we would like to make a comprehensive comparison with LLaVA-MoD, a MLLM method based on knowledge distillation (KD). LLaVA-MoD primarily reports its performance on seven benchmarks: GQA, VisWiz, SQA, VQA, MME, MMB, and $\text{MMB}^{\text{CN}}$. Therefore, we excluded VQAv2, POPE, and MMMU from Table 5 to maintain direct comparability with LLaVA-MoD.

Based on your suggestion, we have added comparisons of LLaVA-KD with all similarly sized models, including Imp-2B and other relevant models, in the revised Table 5. Additionally, as shown in the table below, we have reported AVG (+POPE), AVG (+POPE+VQAv2), and AVG (+POPE+VQAv2+MMMU) to comprehensively evaluate the model's capabilities. It can be seen that our method demonstrates consistent superiority among models of the same scale.

Dear Reviewer xZ9J:

We sincerely appreciate your time and effort. We have carefully considered your suggestions and have responded to each of your questions.

As the Discussion phase is about to close, we would like to know if we have addressed your questions and concerns so far. We will continue to address your concerns until the Discussion phase deadline. If you could consider raising your evaluation of our paper after reviewing our responses, we would be very grateful.

Thank you very much for your consideration.

Dear Reviewer xZ9J,

Thank you for your time and valuable feedback. As the discussion phase is nearing its end, we remain open to addressing any remaining questions or concerns. We would greatly appreciate it if you could consider improving the evaluation after reviewing our responses. Thank you very much for your consideration.

Sincerely, Paper #686 Authors