Efficient Multi-modal Large Language Models via Progressive Consistency Distillation

Dear Reviewer Guzs,

Thank you for your insightful feedback. We appreciate your thoughtful comments and are pleased to address each of your concerns in detail below.

W1: Quantitative experiments are needed for detailed explanation on Fig.1.

RW1: Thank you for your feedback. While Fig.1 qualitatively illustrates the difficulty of directly training under high compression ratios, we fully agree that quantitative evidence is important. To address this, we conduct controlled experiments comparing direct training and progressive learning under the same training budget (4000 steps) on LLaVA-665K. For direct training, we randomly drop 60% of tokens and fine-tune the pretrained LLaVA-v1.5-7B for 4000 steps. For progressive training, we divide training into four stages (0%, 20%, 40%, 60% pruning), each lasting 1000 steps. In each stage, the student adopts the current pruning ratio, while the teacher (shared weights) retains the previous stage’s pruning ratio to provide a consistency signal. Both methods use random pruning to isolate the effect of progressive learning. Experimental results in Table below show that progressive distillation significantly outperforms direct training, supporting the motivation of our method.

We hope this helps better illustrate our motivation, and we will include these experimental results in the final manuscript.

W2: A detailed comparison of TCD and LCD

Thank you for the insightful question. TCD and LCD are two complementary designs within our unified EPIC framework: TCD effects a temporal progression by gradually elevating the compression ratio, whereas LCD embodies a spatial progression by shifting the compression layer from deep to shallow, inspired by the observation that visual tokens receive diminishing attention in deeper layers. We will enrich the revised manuscript with further comparative details to highlight their distinct yet synergistic roles.

W3: Differences and connections with curriculum learning and deeper analysis on effectiveness of progressive learning

Thank you for the insightful comment. We appreciate the opportunity to clarify the conceptual differences and underlying mechanisms.

Connections and Differences with Curriculum Learning. We agree that Layer Consistency Distillation (LCD) aligns with the easy-to-hard philosophy inherent to curriculum learning (CL). Both frameworks aim to reduce training difficulty and enhance convergence by gradually increasing task complexity. However, LCD diverges from standard CL in the following ways:

• Different Learning Objects: Traditional CL operates on input samples or labels, selecting or ordering training data based on their estimated difficulty. In contrast, LCD progressively shifts the token compression layer from deep to shallow Transformer layers, which affects the model’s internal feature space rather than external data complexity.
• Mechanistic Basis: LCD is grounded in empirical observations (e.g., [1]) that visual tokens receive minimal attention in deeper layers, meaning early compression in these layers causes less disruption. The progressive schedule is thus designed to gradually introduce stronger perturbations in a structurally informed manner, rather than relying on heuristic difficulty measures.
• Distinct Objective: While CL generally aims to improve generalization or training efficiency, LCD is specifically tailored to mitigate the feature-space perturbations caused by token compression. It allows the model to transition smoothly from the uncompressed optimum to the compressed one, which we find critical for performance under high compression.

Deeper Analysis on the Effectiveness of Progressive Learning. In Section 3.2, we provide a theoretical prototype demonstrating the benefits of progressive consistency distillation in a simplified 1D setting. We model the shift in optimal outputs (centers) induced by increasing compression ratios, and show that incorporating a teacher with a slightly lower compression ratio leads to a smoother optimization path with lower total variation. This translates into better training stability and convergence.

Empirically, we support this with ablations in Section 5.1 (Table 4), where removing progressive layer scheduling leads to performance drops up to 1.6% under heavy compression (64 tokens). This highlights that progressive learning is not only theoretically grounded but also essential in practice.

We hope this addresses your concerns, and we are happy to clarify further.

W4: The sensitivity analysis about hyperparameter λ.

RW4: Thank you for raising this insightful question. As mentioned in the main text (line 178), TCD and LCD default to λ = 0.7. Following your suggestion, we conducted a sensitivity analysis on λ by evaluating token consistency distillation with λ ∈ {0.1, 0.3, 0.5, 0.7, 0.9}. As shown in the table below (evaluations performed at 77.8% pruning ratio with DART), we observe that model performance degrades significantly when λ is either too large or too small. When λ approaches 0, the distillation loss is almost disabled, depriving the model of valuable guidance from the teacher. Conversely, when λ approaches 1, the SFT loss is suppressed, which can impair the model's autoregressive generation capability. Our experiments demonstrate that λ in [0.5, 0.7] yields the most favorable results, which balances the KL guidance from the teacher with the task-specific supervision from SFT.

We will incorporate these insights into the revised version of our manuscript. Once again, we sincerely thank you for your valuable feedback, which has significantly strengthened the quality and clarity of our work.

W5: Add validation on other MLLM architectures

RW5: Thank you for the insightful suggestion. We have evaluated our method on a range of MLLMs, including LLaVA-Next-7B (featuring a different architectural design), and the Qwen-VL series (distinct architecture). All models are trained on the same LLaVA-665K, under identical training configurations. As shown in the table below, our approach maintains over 95% of the vanilla model's average performance even when 77.8% of visual tokens are pruned, and achieves nearly full performance at a 66.7% pruning ratio, surpassing direct SFT by up to 4.9% under 66.7% pruning. These results clearly demonstrate the effectiveness and generalization capability of EPIC across diverse model architectures and scales. We will incorporate these findings into the revised manuscript.

(Due to space limits, more results on LLaVA-v1.5-13B and Qwen2-VL-0.5B can be found in RW1 to Reviewer aYx1, or we can share the results in the discussion thread upon request. Thanks for understanding)

Reference

[1] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

Best Regards,

The Authors

Dear Reviewer Guzs,

We are delighted to hear that our responses have addressed your concerns. Thank you very much for your positive feedback, and for the time and effort you devoted to reviewing our paper and helping us improve it.

Regarding your note on new experiments during the rebuttal phase: we have carefully reviewed the official NeurIPS guidelines, including both the conference website and the related emails, and we confirm that adding new experiments to address reviewers’ concerns is indeed permitted. Accordingly, we have included additional results in our rebuttal to better support and strengthen our claims in response to your valuable feedback.

We truly appreciate your thoughtful comments and will incorporate your suggestions into the final version. Thank you again for your valuable support!

Best regards,

The Authors

Dear Reviewer BQCk,

Thank you for your insightful feedback. We appreciate your thoughtful comments and are pleased to address each of your concerns in detail below.

W1: Similar ideas in DFKD[1].

RW1: Thank you for the thoughtful feedback. We appreciate the opportunity to clarify the distinctions between EPIC and prior KD methods. While there are conceptual connections to data-free KD methods such as the one proposed for image super-resolution (DFKD[1]), we would like to highlight some key methodological and architectural differences that set EPIC apart.

1. Model Architecture & Modality
   • DFKD for SR employs a GAN-based generator to synthesize low-resolution images for a CNN super-resolution network.
   • EPIC operates on Transformer-based MLLMs, where vision and language tokens interact via cross-modal attention.
2. Task Nature
   • SR is a pixel-level regression task from LR → HR images.
   • EPIC targets multi-modal understanding, where visual tokens must retain semantic fidelity within a language modeling objective—a fundamentally different optimization landscape.
3. Distillation Paradigm
   • Traditional KD transfers knowledge from teacher to separate student networks.
   • EPIC introduces weight-sharing consistency distillation: the same mllm acts as both teacher and student, eliminating the need for extra parameters or model replicas.
4. Progressive Strategy Design
   • While both EPIC and DFKD adopt progressive mechanisms, their motivation and implementation are fundamentally distinct. DFKD's progressive strategy focuses on stage-wise expansion of student CNNs, suitable for pixel-level SR tasks. EPIC introduces a progressive mechanism tailored for MLLMs, which gradually adjusts (i) token compression ratios and (ii) the token compression layer to construct a progressive learning trajectory. This mechanism is designed to ensure training stability and semantic preservation, which, to the best of our knowledge, has not appeared in prior distillation literature, including DFKD.
5. Theoretical Contribution
   • We provide the first formal analysis that progressive consistency paths yield lower total variation under convexity, specifically derived for Transformer token compression—a result inapplicable to GAN-CNN pipelines.

We will include a comparative discussion with DFKD in the revised manuscript to clarify these aspects and hope our response is helpful.

W2&Q1: Theory can be improved and detailed ablation setup

RW2&Q1:

• Theoretical part: We believe those assumptions are standard in convex optimization literature and very common for theoretical tractability[2]. Those assumptions are essential to our theory, and we could explore how to safely remove them in our future work. Also we believe our theory serves as an intuitive motivation and shows that progressive curriculum learning makes a smoother optimization trajectory (Line 153-156) and makes our work rigorously grounded.
• Detailed ablation setup: We sincerely apologize for any confusion caused. The experiment without the distillation loss was trained using the same LLaVA-665K data, with all other training parameters and procedures kept identical to those of EPIC. For the variants without progressive compression ratio or progressive compression layer, we fixed the second layer as the compression layer. Overall, our ablation studies were conducted under a strictly fair and consistent experimental setup. We will include these details in the revised manuscript, and we would be happy to add any further information if needed.

W3: Clarify method via structured/algorithmic presentation.

RW3: Thank you sincerely for your valuable suggestion. We will include an algorithmic flow and provide a more detailed methodological diagram in the revised version to enhance clarity and presentation.

W4&Q2&Q5: More results on different models across varying parameter scales.

RW4&Q2&Q5: Thank you for this insightful suggestion. We have conducted experiments on LLaVA-v1.5-13B (a larger MLLM), Qwen-VL series (different model architecture). All experiments are trained on the same LLaVA-665K and have the same training configuration. As shown in the table below, even when 77.8% of visual tokens are pruned, our approach retains over 95% of the average performance of the vanilla model, and with a 66.7% pruning ratio, it achieves performance on par with the vanilla model. Under the same pruning ratio, our method achieves up to a 4.9% higher average performance compared to direct SFT.. These results demonstrate the effectiveness of EPIC across different MLLMs and parameter scales and we will include these results in the revised manuscript.

(Due to space limits, more results on LLaVA-Next and Qwen2-VL-0.5B can be found in RW1 to Reviewer aYx1, or we can share the results in the discussion thread upon request. Thanks for understanding)

W5: Dataset/model appendix length issue

Thank you for your feedback. In the revised version, we will simplify and relocate the dataset and model descriptions to better highlight our key findings and essential details.

Q3: Why some compressed models outperform original models

We appreciate the insightful question. The observation that compressed visual token models occasionally outperform their original counterparts is not an isolated phenomenon, but one supported by several plausible mechanisms:

• Reduced redundancy improves focus: Compression removes redundant tokens (e.g., background, textures), preserving semantically salient regions and helping the model attend to key visual cues.
• Implicit regularization:Excessive visual tokens may introduce noise or task-irrelevant details, leading to overfitting on non-generalizable patterns during training. Compression acts as an implicit regularizer, forcing the model to focus on more robust and generalizable features.
• Optimization benefits from the progressive training strategy (EPIC): Our method employs progressive consistency distillation, gradually guiding MLLM to adapt to compressed visual representations. This "from-easy-to-hard" learning paradigm facilitates smoother optimization and convergence to better minima.
• Task-specific needs: Many tasks require only a few critical regions; excessive tokens can distract, harming performance.

Q4: Why isn't the combined method included in Table 1?

RQ4: We sincerely thank you for raising this important point. The reason we did not include the combined version of TCD and LCD—termed Integrated Progressive Consistency Distillation (ICD)—in Table 1 is due to the following considerations, rather than any suboptimal performance of the combined approach:

• ICD results are fully reported: The performance of ICD is presented in Table 7 of Appendix D. Notably, with only 192 tokens retained (a 66.7% reduction), ICD achieves an average score of 61.7% across 10 benchmarks—surpassing the vanilla LLaVA (61.4%)—demonstrating its effectiveness.
• Table 1 is designed for fair comparison with existing methods:The primary purpose of Table 1 is to compare our method with existing token-compression approaches, most of which employ only a single compression strategy. To ensure a fair and consistent comparison, we present results for TCD-only and LCD-only, demonstrating the effectiveness of each strategy and highlighting the general applicability of the EPIC framework.
• Space and clarity: Table 1 already includes 10 benchmarks and over 10 methods; adding ICD would increase clutter. To maintain clarity, we present the core components (TCD and LCD) in the main text and defer ICD—an integrated variant—to the appendix for a logical progression.

Reference

[1] Zhang, Yiman, et al. "Data-free knowledge distillation for image super-resolution." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[2] Boyd, Stephen P., and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.

Best Regards,

The Authors

Dear Reviewer aYx1,

Thank you for your insightful feedback. We appreciate your thoughtful comments and are pleased to address each of your concerns in detail below.

W1: Add larger MLLMs.

RW1: Thank you for great suggestion. Evaluating across different parameter scales can further validate the effectiveness of EPIC. We have evaluated our method across MLLMs with varying architectures and sizes. All experiments are trained on the same LLaVA-665K SFT dataset and use the same experimental configuration as in Table 1. As shown in the table below, our method demonstrates strong performance across different parameter scales and diverse model architectures. Even when 77.8% of visual tokens are pruned, our approach retains over 95% of the average performance of the vanilla model, and with a 66.7% pruning ratio, it achieves performance on par with the vanilla model. Under the same pruning ratio, our method achieves up to a 4.9% higher average performance compared to direct SFT. We will include these results in the revised version.

W2: More wall-clock comparison

RW2: Thank you for the valuable suggestion. We have added inference efficiency evaluations across more benchmarks and conducted fair CUDA-time comparisons with state-of-the-art baselines, all under identical single A100 GPU hardware settings. As shown in table below, our three token compression strategies achieve inference latency comparable to TokenPacker and MQT-LLaVA when retaining 64 visual tokens. In some cases (e.g., Random), our methods even outperform them in speed due to the absence of additional computation overhead.
Furthermore, we would like to emphasize that the central contribution of our work lies in the EPIC progressive learning framework, whose another key advantage is in training-time efficiency. As shown in Table 5 of the submission, our method achieves faster training compared to all baselines. The inference-time speedup primarily depends on the integrated token compression strategy, and our framework is modular and compatible with a wide range of such methods. In summary, our approach achieves competitive inference efficiency while significantly improving the training pipeline. We will include these results in the revised version.

Q1: A cost-accuracy plot on direct SFT and ours method.

RQ1: Thank you for your suggestion. Due to policy limits, we present the data for the cost-accuracy plot in the table below. We trained both LLaVA-v1.5-7B with direct SFT and our method—using the same LLaVA-665K—on a setup with 8×A100 GPUs, saving checkpoints at multiple training steps and record corresponding time. The results show that EPIC incurs a moderate and acceptable increase in training time. Remarkably, the model trained with EPIC achieves performance on par with, or even surpassing, the vanilla model, despite removing 66.7% of the visual tokens during inference. This outcome is indeed encouraging, demonstrating that high multimodal representation efficiency can be achieved without sacrificing accuracy. We will present the above data in a cost-accuracy plot in the revised version. Thank you once again.

Best Regards,

The Authors

Dear Reviewer mKBF,

Thank you for your insightful and constructive feedback. We sincerely appreciate your thoughtful comments and are pleased to address each of your concerns in detail below.

W1: Some cognitive motivations for the proposed approach are given without any justifications. For instance, it is not clear to me how the D. Marr works enable us to argue that LLMs only need a small amount of information-dense text tokens.

RW1: We sincerely apologize for any confusion this may have caused. In fact, our reference to D. Marr’s work[1] aims to highlight the higher information density of text tokens compared to visual tokens. This distinction is clearly emphasized in his research. Unlike LLMs, MLLMs must process not only text tokens but also a significantly larger number of visual tokens derived from images or videos. Therefore, we believe it is reasonable to argue that LLMs only require a small number of information-dense text tokens in most scenarios. We will clarify this connection between D.Marr’s work and our motivation in the revised version.

W2: While the motivation is to improve the efficiency of compression of visual tokens in MLLMs, it's not clear to me to what extent the proposed approach is specific to this type of model. Couldn't it be generalized to purely visual models or tokens of other modalities?

RW2: Thank you for this insightful question. We focus on MLLMs because visual token redundancy is especially critical in high-resolution and long-video scenarios, directly affecting performance and user experience. That said, our compression framework is not limited to MLLMs. It can be extended to pure vision models, such as ViTs, by applying token reduction within the visual encoder under our training paradigm. Beyond vision, our approach could extend to other modalities. For example, CoT tokens in reasoning models often exhibit redundancy, and applying targeted compression strategies within a compatible training framework could enhance efficiency in language-centric models as well. We conducted experimental validation by integrating the token compression operation into the purely visual model—ViT—within the EPIC framework, updating the ViT weights, while keeping all other experimental settings identical to those in Table 1. The token pruning strategy for training and evaluation is DART. As shown in the table, applying our method within the visual model (ViT) still achieves strong performance, attaining 99.5% of the average score of the vanilla model. This demonstrates the broad applicability and effectiveness of our approach.

We believe these are promising directions and plan to explore them in future work.

Q1: The authors propose to use a linear increase of the compression ratio in the EPIC framework. The choice is relevant, but alternative progressive schemes could also be proposed. For instance, recent approaches have been proposed for learning a curriculum temperature of KD distillation that could be relevant. How the proposed approach could be enriched with other curriculum strategies?

RQ1: Thank you for the insightful suggestion. In our current work, we adopted a linear schedule due to its simplicity and demonstrated its effectiveness across multiple benchmarks. Our progressive learning strategy is specifically designed in conjunction with token compression, aiming to improve the efficiency of MLLMs by dynamically adjusting either the compression ratio or the compression layer throughout training. We also agree that richer curriculum strategies could further enhance training. The curriculum temperature technique you mentioned is a general curriculum learning approach used in KD. Importantly, it is complementary to our framework and can be seamlessly integrated to further guide student learning.
We therefore followed [2] to incorporate curriculum temperature into EPIC, conducting experiments under the same setup as main table 1 (token pruning strategy: DART). As shown in the results below, the addition of curriculum temperature leads to performance improvements on certain benchmarks, yet the overall average performance exhibits only minor fluctuations compared to TCD. This suggests that simply stacking progressive learning strategies may not be sufficient to unlock their full potential—more sophisticated integration may be required. Alternatively, our method might already be approaching the performance upper bound under token compression, where curriculum temperature does exert an effect, but its benefits are not clearly reflected in the metrics.

We will include a discussion of this potential integration in the revised version and consider it a valuable direction for future work.

Reference

[1] Marr, David. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
[2] Li, Zheng, et al. "Curriculum temperature for knowledge distillation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 2. 2023.

Best Regards,

The Authors