Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models

We appreciate your thoughtful comments and valuable feedback. Below, please find our specific responses (R) to weaknesses (W) and questions (Q).

W1: Only test on 7B model.

R: Thank you for the comment. We have added the experimental results on the LLaVA-1.5-13B model as shown below.

Overall, HAWAII-13B (with CLIP, EVA, and ConvNext as teachers) achieves best performance on all the benchmark datasets, compared to LLaVA-1.5-13B and MoVE-KD-13B [1], which are the two most relevant baselines in our methods. We will add experimental results using smaller models in our final version.

W2: Testing on a wider range of models.

R: Thank you for the suggestion. Following MoVE-KD [1], which serves as the most relevant and established knowledge distillation baseline for our model, we use LLaVA-1.5 as our base model to ensure a fair comparison. Now, we are employing our methods to LLaVA-Next. The experimental results will be updated in this thread in the following days. Moreover, we are also planning to use Qwen-2.5-VL as an additional base model. The results will be added to the final version of our paper.

Q1: Why rank 16 for LoRA?

R: We use 16 for training efficiency. Due to time limitation, we could not complete the experiments with the LoRA rank of 32. But the corresponding results will be added to the final version for a fairer comparison to MoVE-KD [1].

Q2: How long does training take?

R: We used 8 A6000 with 48G GPU memory for each. It takes 20 hours for pretraining and 48 hours for supervised fine-tuning on the 7B model. We also tested on 8 A100 with 80G GPU memory for each. It is much faster, with 9 hours for pretraining and 20 hours for fine-tuning. We will add these training details in the final version.

[1] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders. CVPR 2025.

Thank you for the thoughtful feedback. We hope our responses have addressed your concerns. Please don’t hesitate to let us know if you have any remaining questions.

Also, we collected results using LLaVA-Next as the base model as follows.

W2: Testing on a wider range of models (LLaVA-Next-7B).

We have collected the experimental results with LLaVA-Next-7B being the base model, as shown below.

Our proposed HAWAII shows consistent improvements on most of the benchmark datasets compared to the baseline LLaVA-Next-7B and MOVE-KD (LLaVA-Next-7B), which is the most important baseline method on visual distillation of MLLMs. Due to the submission constraints of VizWiz and MMBench, the evaluation cannot be completed at this time. We are also working on implementing HAWAII to Qwen-VL-2.5. We will update the results on those benchmarks and Qwen-VL-2.5 in the final version.

We appreciate your thoughtful comments and valuable feedback. Below, please find our specific responses (R) to weaknesses (W) and questions (Q).

W1: Apply HAWAII to more methods.

R: Thank you for the suggestion. Following MoVE-KD [1], which serves as the most relevant and established knowledge distillation baseline for our model, we use LLaVA-1.5 as our base model to ensure a fair comparison. Now, we are employing our methods to LLaVA-Next. The experimental results will be updated in this thread in the following days. Moreover, we are also planning to use Qwen-2.5-VL as an additional base model. The results will be added to the final version of our paper.

W2: Evaluation on challenging benchmarks like MM-Vet, MM-Vet-v2, and MMMU-Pro.

R: Thank you for the suggestion. We have added evaluation on MM-Vet, MMVet-V2, and MMMU-Pro using lmms-eval. It shows that our model HAWAII outperforms LLaVA-1.5-7B on MMMU-Pro, MMVet, and MMVet-2.

We will test all the models in our paper on these three datasets and the results will be added to the final version. Please let us know if there are any other valuable benchmark datasets. We are more than happy to test our model and report the results.

W3: Why not other LoRA adapters?

R: Thank you for the question. We use LoRA in our design because of its generalizability. We have conducted an experiment using DoRA as shown below.

The results show that, DoRA has comparable performance on most of the benchmark datasets compared with LoRA. It outperforms LoRA on POPE, MME, and AI2D with worse performance on other benchmarks. We will add the experiments with more LoRA adapters for testing the scalability of our methods in the final version.

W4: Exploration on LLM distillation.

R: Thank you for the comment. As mentioned in the limitations of our paper, we acknowledge that further improvements might be achieved by LLM distillation. In this paper, we focus on visual distillation and propose novel coarse-grained and fine-grained distillation for better transferring knowledge from multiple visual experts into one unified visual encoder. We will leave the combination of LLM and visual distillation as future work.

Reference:

[1] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders. CVPR 2025.

I've carefully read the rebuttals and thank you for your efforts. The concerns that I was posing initially are mostly addressed. I've updated the scores towards leaning accept. I suggest that authors reflect on not only my comments but also others' ones, and furthermore please make sure your promise is realized through making Qwen2.5-VL's score and other challening benchmarks.

Thank you for the thoughtful feedback and updated evaluation. Please let us know if you have any further questions.

Also, we collected results using LLaVA-Next as the base model as follows.

Additional experimental results on LLaVA-Next-7B.

We have collected the experimental results with LLaVA-Next-7B being the base model, as shown below.

Our proposed HAWAII shows consistent improvements on most of the benchmark datasets compared to the baseline LLaVA-Next-7B and MOVE-KD (LLaVA-Next-7B), which is the most important baseline method on visual distillation of MLLMs. Due to the submission constraints of VizWiz and MMBench, the evaluation cannot be completed at this time. We are also working on implementing HAWAII to Qwen-VL-2.5. We will update the results on those benchmarks and Qwen-VL-2.5 in the final version.

Looks great to me. Please make sure the final version includes more recently released models

We appreciate your thoughtful comments and valuable feedback. Below, please find our specific responses (R) to weaknesses (W) and questions (Q).

W1: Novelty.

R: Thank you for the comments. In this paper, we propose HAWAII that distills knowledge from multiple visual experts into a single vision encoder. To mitigate conflicts among different teachers and switch between different teacher-specific knowledge, we introduce teacher-specific LoRA adapters, each corresponding to one specific visual teacher. We also include token importance scores to emphasize the most informative tokens from each teacher adaptively. Moreover, we also employ a set of general-knowledge LoRA adapters to learn the common knowledge among different teachers and training data.

We would like to clarify that although the term hierarchical knowledge distillation has been used in other domains (e.g., [3, 4]), the underlying techniques differ significantly. Their methods focus on aligning features extracted from the hierarchical architecture of one teacher, such as features from the shallow layers to the deep layers. On the other side, our proposed method, HAWAII, treats hierarchical knowledge distillation as fine-grained (distilling knowledge from one teacher only for solving conflicts between teachers) and coarse-grained knowledge distillation (distilling knowledge from the summarized knowledge of multiple teachers), which is totally different from those methods in other domains.

To the best of our knowledge, we are the first to use two sets of LoRA adapters to distill knowledge from multiple visual experts into one visual encoder. We might miss some references in relevant areas. Please let us know if there are any works that have a similar design of two sets of adapters for distilling knowledge.

W2: No baseline on models with multiple visual encoders.

R: While we appreciate the importance of such baselines with multiple experts [1], our method fundamentally differs in design: we distill knowledge to a single unified vision encoder, rather than maintaining multiple experts during both training and inference. Directly comparing these two paradigms would therefore not be fair, as the former with multiple experts typically involves higher computational cost and a larger number of parameters. We have collected the latency of LLaVA-1.5-7B, Eagle-X4-Plus (with four visual experts), and our HAWAII-7B as shown below. We use Pytorch Profiler to measure the FLOPs and CUDA time using the teaser figure from the llava paper and “What is shown in this image?” as input. We use one A6000 GPU with 48 GB GPU memory to inference the models.

It shows that, our HAWAII, which only uses one vision encoder, has smaller FLOPs and CUDA times compared to the Eagle model, which contains multiple (4-5) visual experts. We will explicitly highlight this distinction in the paper and include a discussion in the final version.

W2: No ablations on how different vision experts affect performance.

R: Thank you for the comment. We have presented an ablation study on different settings of visual teachers in Table 1 of our paper. It shows that having more visual teachers improves performance, and due to the heterogeneous knowledge in different visual teachers, the performance boost in different benchmark datasets varies. For performance using different visual experts [1], we have included the following table for your reference.

Our HAWAII uses CLIP as the student encoder with CLIP, ConvNext, and EVA being the teachers. Our HAWAII achieves best performance on all the benchmark datasets except AI2D. *represent results from [1]. We will include a more detailed analysis in the final version.

W3: Apply HAWAII to more methods.

R: Thank you for the suggestion. Following MoVE-KD [2], which serves as the most relevant and established knowledge distillation baseline for our model, we use LLaVA-1.5 as our base model to ensure a fair comparison. Now, we are employing our methods to LLaVA-Next. The experimental results will be updated in this thread in the following days. Moreover, we are also planning to use Qwen-2.5-VL as an additional base model. The results will be added to the final version of our paper.

Q1: How are the teacher models selected?

R: Thank you for the question. We select CLIP, ConvNeXt, and EVA-02 as our main design, following MoVE-KD [2], which serves as the most relevant and established knowledge distillation baseline for our model to ensure a fair comparison. We also include Pix2Struct and SAM for analyzing the impact of different visual experts, as shown in Table 1.

Moreover, they serve different purposes: CLIP (general-purpose), ConvNeXt (image classification), Pix2Struct (OCR/document understanding), EVA-02 (multi-task generalist), and SAM (segmentation). These teachers differ in architecture, training data, and task specialization, providing a broad and complementary set of capabilities. And this diversity is intended to balance generality with scalability. Such a diverse set of visual teachers brings comprehensive knowledge from different domains, benefiting the model. We will clarify this in our paper.

Q2: Is knowledge distillation applied during both pretraining and fine-tuning stages?

R: Yes. It is applied in both pretraining and supervised fine-tuning stages. We will clarify this in the paper.

Reference:

[1] Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders. ICLR 2025.

[2] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders. CVPR 2025.

[3] SignKD: Multi-modal Hierarchical Knowledge Distillation for Continuous Sign Language Recognition.

[4] Hierarchical Self-supervised Augmented Knowledge Distillation. arXiv:2107.13715.

Thank you for the thoughtful feedback. We hope our responses have addressed your concerns. Please don’t hesitate to let us know if you have any remaining or further questions.

Also, we collected results using LLaVA-Next as the base model as follows.

W3: Apply HAWAII to more methods (LLaVA-Next-7B).

We have collected the experimental results with LLaVA-Next-7B being the base model, as shown below.

Our proposed HAWAII shows consistent improvements on most of the benchmark datasets compared to the baseline LLaVA-Next-7B and MOVE-KD (LLaVA-Next-7B), which is the most important baseline method on visual distillation of MLLMs. Due to the submission constraints of VizWiz and MMBench, the evaluation cannot be completed at this time. We are also working on implementing HAWAII to Qwen-VL-2.5. We will update the results on those benchmarks and Qwen-VL-2.5 in the final version.

We appreciate your thoughtful comments and valuable feedback. Below, please find our specific responses (R) to weaknesses (W) and questions (Q).

W1: How do adapters for different teachers interact during training.

R: Thank you for the question. During training, we activate one LoRA expert at a time to get the specific student feature $I^{S}_{i}, \forall i \in [N_t]$ for fine-grained distillation (Sec. 2.3.2). Also, for calculating coarse-grained distillation loss and text generation loss, only one teacher-specific LoRA adapter and one general-knowledge adapter are activated based on the routers (Eq. (2)). Overall, each teacher-specific LoRA adapter will be updated to align with its corresponding teacher independently in fine-grained distillation (Sec. 2.3.2), while coarse-grained distillation (Sec. 2.3.1) will update one teacher-specific LoRA adapter selected by the router. That means they will not interfere with each other during training.

W2: Generalization beyond training distributions.

R: Thank you for the suggestion. Following previous works [1,2], in our experiments, we use LLaVA-1.5 pretraining and SFT data for training the model, which means we report zero-shot performance on most of the benchmark datasets. Meanwhile, we are also interested in having more evaluations on synthetic data. Please let us know if there are any relevant, valuable benchmarks we could test our models on.

W3 and Q3: No exploration on non-visual distillation.

R: Thank you for the comment. As mentioned in the limitations of our paper, we acknowledge that further improvements might be achieved by LLM distillation. In this paper, we focus on visual distillation and propose novel coarse-grained and fine-grained distillation to better transfer knowledge from multiple visual experts into one unified visual encoder. We will leave the combination of LLM and visual distillation as future work.

Q1: Optimal number of visual teachers.

R: In Table 1 of our paper and the following table, we did observe that under different settings of the teachers, the model performs differently on diverse benchmark datasets, which might be due to the diverse knowledge brought by teachers. Thus, instead of finding an optimal number or set of visual teachers for all the diverse tasks, we suggest that users select the teachers that best suit the downstream tasks for better performance.

Q2: Optimal number of LoRA adapters.

R: The number of teacher-specific LoRA adapters is dependent on the number of visual teachers. And the number of general-knowledge LoRA adapters is determined by the users. We have added an ablation on the number of general-knowledge LoRA adapters, as shown below

In our paper, we use 3 general-knowledge experts as the main setting for HAWAII. Results show that, the setting of 3 general-knowledge LoRA adapters outperforms other settings with 1 and 5 adapters on most of the benchmark datasets. We also observe that on MMMU, increasing the number of general-knowledge LoRA adapters decreases performance, while on GQA, using 3 or 5 adapters works the best.

Q3: Include domain-specific visual experts.

R: Thank you for the suggestions. In our paper, following MoVE-KD [2] and Eagle [3], we use CLIP, EVA, and ConvNext as the basic settings of teachers, with Pix2Struct and SAM as additional teachers. They serve different purposes: CLIP (general-purpose), ConvNeXt (image classification), Pix2Struct (OCR/document understanding), EVA-02 (multi-task generalist), and SAM (segmentation). These teachers differ in architecture, training data, and task specialization, providing a broad and complementary set of capabilities. And this diversity is intended to balance generality with scalability. Such a diverse set of visual teachers brings comprehensive knowledge from different domains, benefiting the model. Moreover, we would like to explore some other choices for the visual experts, such as medical or driving models. Please let us know if there are any specific models you would suggest.

Reference:

[1] Visual Instruction Tuning. NeurIPS 2023.

[2] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders. CVPR 2025.

[3] Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders. ICLR 2025.