LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token

Q3: About missing reference of streaming video LLM works such as "VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024"?

A3: Thank you for your careful review and for suggesting the related work. VideoLLM-online is a valuable work, while it focuses more on similar generation with streaming videos, with different motivation and emphasis compared to LLaVA-Mini. In the new version, we will include a discussion of VideoLLM-online in the related work section.

 

Thanks again for your careful and valuable comments. Hope our responses answer your questions well and reassure your concerns.

Thanks for your careful and valuable comments, and we will refine the paper following your suggestions. Following, we will respond to your questions in detail.

 

Q1: For each downstream task, this paper trains a new, independent compression and modality pre-fusion modules? Would it be possible to train generic compression and modality pre-fusion modules in stage 1 that can be generalised to various downstream tasks?

A1: We apologize for any confusion. In fact, LLaVA-Mini handles all visual tasks with a single model, including vision projection, the compression module, the modality pre-fusion module, and the LLM backbone. In LLaVA-Mini, we do not train separate compression and modality pre-fusion modules for different downstream visual tasks.

The misunderstanding may have arisen from our description of the two-stage training process. Here, I give further explanation. In Stage 1, LLaVA-Mini learns to align vision and language representations using visual caption data. In Stage 2, LLaVA-Mini is trained to perform various visual tasks based on minimal vision tokens with one model. During this stage, we use visual SFT data to train LLaVA-Mini (with generic compression and modality pre-fusion) to handle a range of visual tasks. The entire pretraining-then-SFT process is similar to the setup in LLaVA-v1.5.

Finally, only one model is needed to perform all visual tasks in our experiments. When handling different tasks, the compression module adaptively extracts key visual information from the input image and compresses it into a single vision token and the modality pre-fusion module fuses visual information into the text modality based on the input image and instruction (which may involve different tasks).

We hope this clarification helps provide a clearer understanding of the LLaVA-Mini training process.

 

Q2: Can we re-use the first few layers, which have high activation for vision tokens, as the fusion module? To further include the compression module, the first few layers can have two pathways, one pathway is to output the compressed tokens while the other pathway is to be gradually fused together with the text tokens.

A2: Thank you for your valuable suggestion. We indeed considered the approach you proposed during the initial design phase of our method. However, two key limitations led us to not adopt it:

• Vision representations after the early layers contain contextual information, which complicates the compression module: Once vision tokens are fed into the LLM, the early layers imbue the visual representations with contextual information. Applying query-based compression at this stage makes it difficult for the compression module to effectively distinguish between different vision tokens.

• Inter-layer operations within the LLM may not be compatible with existing acceleration frameworks: One of the primary motivations for placing the compression and pre-fusion modules outside the LLM backbone in LLaVA-Mini is to keep the LLM backbone unchanged. This design choice ensures compatibility with a wide range of existing LLM acceleration technologies and frameworks, thereby further boosting efficiency.

These considerations explain our architectural choices. According to your suggestion, we have also included a comparison between LLaVA-Mini and a version of LLaVA-Mini with compression applied at the 4th layer of the LLM. The results demonstrate that the current configuration of LLaVA-Mini offers more favorable performance. Following your suggestion, We have incorporated these result sand the rationale behind our architectural decisions into the Appendix E.4 (Line 1134-1163) in the new version of the paper.

Q3: If there’s any strategy (e.g, temporal embedding) to solve the discrepancy between training and inference (i.e., much longer videos)?

A3: The issue you raised is also of great interest to us. In LLaVA-Mini, the extension of training and inference capabilities, as well as the handling of longer video sequences, is achieved through the inherent context generalization capacity of the LLM itself, without the need for additional techniques. The ability of LLaVA-Mini to effectively generalize in this manner stems from representing each image as a single token during video processing. By using a single token per image, the position encoding implicitly facilitates the modeling of temporal information.

Moreover, due to the compression of vision tokens, the required generalization span for LLaVA-Mini is relatively small. For example, the training process only involves 60 seconds (60 frames), while inference can involve up to 1 hour (3600 frames). For LLaVA-Mini, the context only needs to generalize from 60 tokens to 3600 tokens, which is considerably easier than LLaVA-v1.5, which must generalize from 60×576 tokens to 3600×576 tokens.

Additionally, you mentioned that incorporating temporal embeddings could further enhance the model's explicit perception of temporal information. We plan to explore this in future work.

 

Thanks again for your careful and valuable comments to improve our work. If our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating.

Thanks for your careful and valuable comments, and we will refine the paper following your suggestions. Following, we will respond to your questions in detail.

 

Q1: May achieve inferior performance under scenarios demanding fine-grained visual details, e.g., text-oriented benchmarks including Text VQA.

A1: Indeed, to achieve significant efficiency improvements, LLaVA-Mini compresses visual information, which entails a trade-off between efficiency and visual comprehension quality. Based on our experimental results, we believe that the slight performance decline on TextVQA is acceptable when weighed against the substantial efficiency gains. We appreciate your suggestion and have included a discussion of this limitation and potential in the Appendix H (Line 1345-1407) of the new version.

• Flexible Adaptation to Different Visual Scenarios: LLaVA-Mini must balance efficiency (the number of vision tokens) and performance (visual understanding capability). It allows for flexibility by adjusting the hyperparameter C, which controls the number of vision tokens, offering the possibility of adapting to different scenarios. Results in Table 1 and Table 6 demonstrate that performance improves consistently as C increases. In practice, for scenarios requiring fine-grained visual details, users can set a larger C to capture more visual information. Conversely, for scenarios where efficiency is prioritized, users can set a smaller C, achieving significant efficiency gains while retaining substantial visual understanding capability.

• Comparison with Previous Token Reduction Methods: As you mentioned, most previous token reduction methods result in considerable loss of visual information. This serves as a core motivation for LLaVA-Mini, which seeks to maintain substantial visual comprehension while minimizing vision tokens through modality pre-fusion. The results in Table 1 further confirm that LLaVA-Mini outperforms prior approaches in terms of visual understanding.

Thank you for your insightful comments. We hope these clarifications address your concerns. We have also incorporated this limitation and its corresponding solutions into the Appendix H (Line 1345-1407) in the new version of the paper.

 

Q2: The baseline that keeps the visual tokens in early layers of LLMs and then compressing the visual token output on the L-th (4 in the paper) layer.

A2: Thank you for your valuable feedback. This approach was considered during the early stages of our work; however, two key limitations led us to not adopt it:

• Vision representations after the L-th layers contain contextual information, which hinders the compression module: After the vision tokens are fed into the LLM, the early layers cause the visual representations to carry contextual information. Applying query-based compression on top of these representations makes it difficult for the compression module to distinguish between different vision tokens.

• The inter-layer operations within the LLM may not be compatible with existing acceleration frameworks: One of the main motivations for placing the compression and pre-fusion modules outside the LLM backbone in LLaVA-Mini is to keep the LLM backbone unchanged. This design allows for compatibility with nearly all existing LLM acceleration technologies and frameworks, further enhancing efficiency.

The above points explain our architectural choices. Following your suggestion, we have also included a comparison between LLaVA-Mini and LLaVA-Mini (with compression at the 4th layer in the LLM). The results demonstrate that the configuration of LLaVA-Mini is more advantageous. Following your suggestion, We have incorporated these results and the architectural motivation into the Appendix E.4 (Line 1134-1163) in the new version of the paper.

We sincerely appreciate your valuable and insightful comments, which have significantly contributed to the improvement of our work. Hope that the additional experiments and responses we have provided dispel your concerns effectively and enhance the overall quality of the submission, potentially influencing your assessment rating of our work.

As the deadline for discussion (December 2nd) approaches, we would be grateful if you could let us know if there are any further questions and we are happy to respond as soon as possible.

Once again, we deeply appreciate the time and effort you have dedicated to the review and discussion process.

Q7: The presentation of "Modality Pre-fusion" and "Vision Token Compression" would benefit from a chronological reorganization to better reflect the processing pipeline (image encoder -> vision token compression -> modality pre-fusion).

A7: Thank you for your thoughtful suggestion regarding the writing sequence. In fact, the modality pre-fusion is also computed based on the vision representations output by the vision encoder. Logically, the compression module and the pre-fusion module in a parallel relationship, which is why we have organized the paper in this way. We appreciate your feedback, and in the next version, we will provide a more detailed explanation in the method overview part and consider adjusting the structure to help readers better understand the overall framework.

 

Q8: The training costs during the instruction tuning stage are not adequately addressed, making it difficult to assess the true efficiency gains during training.

A8: The motivation behind LLaVA-Mini is to improve the inference efficiency of LMMs by reducing the number of vision tokens input to the LLM backbone. Training efficiency is not the focus of our research, because LLaVA-Mini is not designed to improve training efficiency, and we do not claim that it can improve training efficiency. We apologize for any misunderstanding caused and will emphasize this point more clearly in the next version.

 

Q9: About the relation between decreased attention weight and high attention entropy? What causes high entropy in vision-related attention weights in the latter layers despite lower attention weights?

A9: We apologize for any confusion caused.

Firstly, we did not claim that there is high entropy in the vision-related attention weights in the later layers. As shown in Figure 3, we emphasize high entropy in the early layers (lines 186-188). In the later layers, however, no consistent entropy patterns in the later layers are observed across different models. For instance, LLaVA-v1.5-Vicuna-7B exhibits higher vision-related entropy in the later layers, while LLaVA-NeXT-Vicuna-7B shows lower entropy.

Additionally, attention weights and attention entropy are not directly correlated.

• Attention weight: The attention weight in Figure 2 measures how much attention a vision token receives from subsequent tokens. For example, when evaluating the attention of a response token towards previous tokens, the response token may attend to instruction tokens, vision tokens, and response tokens (prefix), with their attention weights summing to 1. Therefore, a decreased attention weight to vision tokens indicates a reduced importance of vision tokens in the later layers.

• Attention entropy: The attention entropy in Figure 3 measures the degree of focus that vision tokens (note that the attention received by vision tokens is normalized to adapt to the calculation of entropy) receive from subsequent tokens. For instance, if the response token equally attends to all vision tokens, the entropy is high. Conversely, if the response token focuses on only a few vision tokens, the entropy is low.

Thus, the observed decrease attention weight and attention entropy results is not contradictory. In the later layers, vision tokens receive less attention weight, and this attention could either be focused on a few vision tokens (low entropy) or spread across many vision tokens (high entropy).

We hope this clarifies your question regarding attention entropy.

 

Thanks again for your careful and valuable comments to improve our work. If our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating.

Q4: About the claim of "one vision token" and the computational efficiency gains are not solely from token reduction.

A4: Yes, the proposed LLaVA-Mini still processes all the projected vision tokens. The statement regarding "one vision token" refers to the number of vision tokens fed into the LLM, as the parameters and computational cost of the LLM backbone dominate the overall resource consumption in LMMs. Reducing the number of tokens fed into the LLM backbone can significantly enhance efficiency.

This is also the key innovation of LLaVA-Mini compared to previous works. Token reduction methods, such as PruMerge and MQT-LLaVA, focus on directly reducing the number of tokens output by the ViT. In contrast, LLaVA-Mini focuses on transmitting visual information to the LLM using the fewest possible vision tokens, thereby enhancing efficiency while maintaining strong visual understanding.

Thank you for your suggestion. In the new version, we have emphasized that "one vision token" refers specifically to the vision tokens fed into the LLM to avoid any potential misunderstanding.

 

Q5: About the ablation study of using only the pre-fusion module without compression.

A5: Thank you for your valuable suggestion. We have conducted the relevant ablation experiments in following table. As shown in table, when using only the pre-fusion module without compression, LLaVA-Mini achieves superior performance compared to LLaVA-v1.5 with both using 576 vision tokens, demonstrating the effectiveness of the pre-fusion module. We have included the experimental results in the Appendix E.3 (Line 1120-1133) in the new version of the paper.

 

Q6: About the computational overhead introduced by the fusion module, particularly regarding the processing of N^2×5 vision tokens plus l_q text tokens.

A6: We apologize for any confusion regarding the efficiency analysis in Section 5.3. The computational overhead reported in Figures 7 and 8 reflects the total computation required by the entire LLaVA-Mini model, including the vision encoder, compression module, pre-fusion module, and LLM backbone. Therefore, the computational overhead introduced by the fusion module is already accounted for in both Figures 7 and 8. In comparison, LLaVA-Mini demonstrates a clear efficiency advantage over LLaVA-v1.5.

Regarding the processing of N^2×5 vision tokens, the results in Figure 8 further highlight the efficiency of LLaVA-Mini. Specifically, LLaVA-v1.5 requires the LLM to process over 2000 vision tokens for high-resolution images, while LLaVA-Mini’s pre-fusion module consists of only 4 layers, offering a significant reduction in computational cost.

Additionally, we believe you may be interested in the proportion of computational load contributed by the pre-fusion module in LLaVA-Mini. To further study the proportion of computational load contributed by each component in LLaVA-Mini, we computed the FLOPs of each module, as shown in the table below. The proposed compression module and pre-fusion module incur minimal computational cost, while the computation required by the LLM backbone is significantly reduced. We have included the experimental results in the Appendix E.6 (Line 1182-1187) in the new version of the paper.

Thanks for your careful and valuable comments, and we will refine the paper following your suggestions. Following, we will respond to your questions in detail.

 

Q1: About the writing of detailed settings in attention weight analysis.

A1: We greatly appreciate your valuable suggestions. Due to space limitations, some specific details regarding the attention weight analysis were not included in the main body of the paper. Following your suggestion, we have provided a more detailed description of the analysis setup and formalized calculations in Appendix A (Line 918-954) .

• How to handle multiple attention heads: The attention weights are averaged across all attention heads.

• How attention scores are averaged among instruction, vision, and response tokens: Taking the attention scores from response to vision as an example, we compute the sum of attention weights from each response token to all vision tokens. Then, the attention from the response tokens to the vision tokens is averaged across all response tokens to obtain the average attention of response-type tokens to vision-type tokens. We have formalized this process and added the relevant details in Appendix A (Line 918-954).

• How to handle LLaVA 1.5 and LLaVA-NeXT with different resolutions: For LLaVA-v1.5 (pad) and LLaVA-NeXT (anyres), which may involve vision inputs at different resolutions, we use the original settings as specified. In our analysis, we do not distinguish between different types of vision tokens but treat them collectively as vision tokens, measuring their overall importance.

Thanks for your suggestion, we have added a detailed description of the experimental setup for the attention analysis in Appendix A (Line 918-954). Additionally, after the paper is de-anonymized, we will release the attention analysis scripts to ensure the reproducibility of our analysis. Hope the added formal details and future open source code can dispel your concerns about the reproducibility of analysis.

 

Q2: About the discrepancy between the visualization results and claims of attention to vision tokens decreases in later layers?

A2: We apologize for any confusion caused.

Figure 2 illustrates the attention weights received by vision tokens at each layer, which remain at very low levels in the later layers. The visualization in Figure 4 may have caused some misunderstanding, as it appears that instruction tokens still maintain some attention to vision tokens (appearing brighter) in the later layers. In reality, Figure 4 was designed to more clearly demonstrate the variation of attention distributions across different layers. Due to the overall sequence length, the attention scores for each position are quite small. Furthermore, the first token in the upper-left corner always receives 100% attention weight, which results in a visualization where only the upper-left position is bright, while the others appear extremely dark (since their values are very small). To make the visualization more readable and provide a clearer impression to the readers, the color bar in Figure 4 is not evenly distributed but follows a logarithmic scale. Thus, although some vision tokens appear a little bit bright in the later layers of the visualization, when considered alongside the color bar, their attention weights are indeed very low. Meanwhile, in the later layers of the visualization, the number of vision tokens receiving attention decreases significantly (with some columns of vision tokens appearing very dark), resulting in the overall attention weights for vision tokens remaining very low in these layers.

Thank you for your reminder. In the new version of the paper, we have highlighted the uneven distribution of the color bar in the visualization (Line 196).

 

Q3: Lacks sufficient details about the pre-fusion transformer architecture.

A3: Thank you for your insightful feedback. The pre-fusion module applies the same decoder-only architecture as the LLM, including both the structure and hyperparameters. The motivation behind this setting is to ensure flexible adaptation to existing LLM frameworks and other acceleration techniques. Based on your suggestion, we have included a more detailed explanation of pre-fusion transformer in the Appendix B (Line 958-967) of the new version.

Q4: Have the authors done any ablation on using SOTA token pooling methods for the vision token pooling and compare against them (other than average pooling)?

A4: Actually, by examining both Table 1 and Table 6 in the paper together, we can compare the query-based compression method proposed by LLaVA-Mini with state-of-the-art (SOTA) token pooling methods. We appreciate your suggestion and have included an ablation study in the tables that compares LLaVA-Mini with various token pooling methods. To ensure a fair comparison of token compression performance, we have removed the modality pre-fusion module from LLaVA-Mini for the comparison with SOTA token merging methods, including PruMerge, PruMerge++, and MQT-LLaVA. Specifically, PruMerge applies the widely-used token merge (ToMe) technique on ViT, PruMerge++ improves upon PruMerge by uniformly sampling additional vision tokens, and MQT-LLaVA employs Matryoshka representation learning to compress vision tokens.

As shown in the table, LLaVA-Mini's compression module outperforms PruMerge, PruMerge++, and MQT-LLaVA at the same compression rate, showing the advantages of query-based compression. Following your valuable suggestion, we have added this detailed comparison between query-based compression and SOTA token pooling methods in the Appendix E.2 (Line 1094-1119) of the new version.

 

Q5: Can the authors highlight results on more vision centric benchmark and show the impact of different number of vision tokens on these e.g., CV-Bench?

A5: Thank you for your valuable suggestion. Evaluating LLaVA-Mini on vision-centric benchmarks would further demonstrate its understanding capabilities of visual information. CV-Bench is an excellent benchmark for assessing vision-centric capabilities, encompassing both 2D and 3D visual understanding. In the table below, we compare LLaVA-Mini with LLaVA-v1.5, where LLaVA-Mini demonstrates superior vision-centric understanding with fewer vision tokens on CV-Bench.

Following your suggestion, we have included these experimental results in the Appendix E.1 (Line 1077-1092) of the new version.

 

Thanks again for your careful and valuable comments to improve our work. If our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating.

Thanks for your careful and valuable comments, and we have refined the paper following your suggestions. Below, we will respond to your questions in detail.

 

Q1: About the complexity of plugging this compression method to existing multi-modal pipelines, and the open source implementation of the code.

A1: Thank you for your suggestion. The compression method of LLaVA-Mini can be easily plugged into existing multi-modal pipelines, as it only requires the addition of two extra modules (the compression module and the modality pre-fusion module) before the LLM, while the other components (such as the vision encoder, the LLM, and the training loss) remain unchanged. Following your advice, we have includes a description of how to integrate the compression method into existing multi-modal pipelines in the Appendix B (Line 958-967) in the new version.

More importantly, we have provided a source code of LLaVA-Mini in the supplementary materials and plan to open-source the GitHub repository after the paper is anonymized, to ensure the reproducibility of LLaVA-Mini.

Hope the additional description and code in the supplementary materials can dispel your concerns about reproducibility.

 

Q2: About the limitation of the compression technique and potential solutions for addressing them?

A2: Thank you for your valuable suggestion. We have explained the limitation of LLaVA-Mini in the Appendix H (Line 1345-1407) of the new version. The limitation of LLaVA-Mini lies in the trade-off between the number of vision tokens (efficiency) and performance. As LLaVA-Mini uses a single vision token, it compresses visual information, which may leave room for improvement in image understanding tasks that involve complex visual content. A potential solution is to slightly increase the number of vision tokens, such as using 16 vision tokens, to achieve a better balance between efficiency and performance. Query-based compression offers flexibility in adjusting the number of vision tokens, as we can simply modify the parameter C to control the number of tokens. The results in Table 6 and Table 8 demonstrate that increasing the number of vision tokens leads to continuous performance improvements in LLaVA-Mini.

In practice, the number of vision tokens can be adjusted based on the specific efficiency requirements of different scenarios, allowing for a trade-off between efficiency and performance. We appreciate your feedback, and have incorporated the limitation and potential solutions of LLaVA-Mini into the Appendix H (Line 1345-1407) in the new version of the paper.

 

Q3: How model efficiency scales across different hardware platforms?

A3: Thank you for your reminder. The efficiency improvements brought by LLaVA-Mini stem from reduced computational load (FLOPs), which is consistent across different hardware platforms. To demonstrate the scalability of model efficiency across different hardware platforms, we have added experiments that compute the inference latency of LLaVA-Mini on three hardware configurations: RTX 3090, A100, and A800, as presented in the table below. The results indicate that the efficiency improvements brought by LLaVA-Mini are scalable across these platforms.

Following your recommendation, we have included the this experiment in Appendix E.5 (Line 1165-1180) of the new version.

We sincerely appreciate your valuable and insightful comments, which have significantly contributed to the improvement of our work. Hope that the additional experiments and responses we have provided dispel your concerns effectively and enhance the overall quality of the submission, potentially influencing your assessment rating of our work.

As the deadline for discussion (December 2nd) approaches, we would be grateful if you could let us know if there are any further questions and we are happy to respond as soon as possible.

Once again, we deeply appreciate the time and effort you have dedicated to the review and discussion process.

LLaVA-Mini and FastV differ entirely in method and results.

Liang Chen pointed out that a part of our preliminary experiments reached the same observation as their study. However, this similarity is limited to observational findings and does not pertain to the core methodology or innovations of our work.

The primary contribution and innovation of LLaVA-Mini lie in its ability to compress vision tokens fed into the LLM backbone into a single token through the proposed learnable compression module and modality pre-fusion module. We had not read their paper beforehand and independently arrived at a similar observation under unknown circumstances. Moreover, this observation constitutes only a minor part of LLaVA-Mini and is not central to our main contribution, and we have never claimed that we are the first to discover it. Therefore, we have appropriately cited their work in this observation, while the accusation of plagiarism is both unwarranted and excessive.

It is highly groundless and inappropriate to make accusations of plagiarism based solely on the use of a similar color scheme in attention visualization.

Liang Chen’s concerns stem from the situation that our attention visualizations use the same color scheme as theirs, which led to the claim that “features the identical token nomenclature and even the same color scheme, suggests we use their codebase.” It is inconceivable that a similar color scheme in visualization could serve as evidence for accusing us of using their code.

Using this as a basis to falsely accuse us of plagiarizing the FastV codebase is entirely unfounded. The token nomenclature in our work follows the naming conventions established in the original LLaVA paper, along with the corresponding token positions. As for the color scheme, we simply used cmap='viridis', a most widely used setting for attention visualization. Attention visualization is one of the most fundamental analytical techniques, and therefore, the explicit speculation that we used FastV’s codebase is entirely unacceptable.

In principle, we are not obligated to prove the originality of our code, as the allegation itself is baseless and purely speculative. However, since ICLR comments are publicly visible, such unfounded accusations have caused serious reputational harm to our work. Demonstrating the exact process by which code is created is not always straightforward. Fortunately, in this case, the complete visualization code was generated entirely from scratch using ChatGPT via textual prompts, with a recorded timestamp of August 30, 2024 (https://chatgpt.com/share/67c817bb-0af0-8013-887c-e022c5c04532). This directly invalidates the premise of Liang Chen’s claim that we used their codebase.

 

We appreciate ICLR for providing a platform for open academic exchange. We believe that it is unreasonable to make overly subjective accusations of plagiarism based solely on an attention visualization image with a similar color scheme, especially when the paper’s writing, model, training, and experimental results are entirely different. While OpenReview is an open platform for academic discussion, this does not allow unwarranted and exaggerated accusations, which run counter to the ICLR community's core values of fairness, mutual respect, and constructive exchange. Such speculation has unfairly impacted our work, and we request a clarification.

We sincerely appreciate the Program Chair’s time and effort on this matter.

Dear ICLR 2025 Program Chair, Liang Chen

Upon receiving his comments and email, we were surprised, as it was completely unexpected.

First, we would like to state that, prior to receiving an email from Liang Chen on March 2, 2025, we were unaware of the existence of FastV. This is because FastV was not mentioned in any of our highly-relevant baselines (studies focused on compressing the number of vision tokens fed into the LLM backbone, such as LLaVA-PruMerge, VoCo-LLaMA, MQT-LLaVA, and TokenPacker).

On March 2, 2025, we received the first email from Liang Chen, in which he pointed out that one of the three observations in our preliminary analysis—"Vision Tokens Are More Important in Early Layers"—aligned with their findings, and he suggested that we cite FastV. Upon reviewing this, we promptly included the citation in the latest version and replied to his email accordingly.

On March 3, 2025, Liang Chen responded with additional requests, asking us to:\

1. Remove the claim "To this end, we begin by exploring a foundational question: How does the LMM (particularly the LLaVA architecture) understand vision tokens?" Replace it with "Inspired by FastV, we conduct the same experiments..."\
2. Acknowledge FastV's prior work at the beginning of this section.\
3. Include a comparison with FastV's methods.\
4. Explicitly discuss the differences between FastV and your work.

From our perspective, these requests were inappropriate and exceed reasonable boundaries. Since we had not read their paper before, the demands to include statements such as "Inspired by FastV..." and "acknowledgment..." were factually inaccurate. Furthermore, LLaVA-Mini differs fundamentally from FastV in both methodology and conclusions. LLaVA-Mini introduces a novel architecture and training paradigm, whereas FastV is an inference method. LLaVA-Mini processes vision tokens extracted by the vision encoder, whereas FastV reduces the number of vision tokens at the intermediate layers of the LLM. The two works were evaluated on almost different benchmarks, and there are basically no identical experimental results. Given these fundamental differences on method and innovation, a direct comparison with FastV exceeds reasonable requests, just as none of the related baselines we compared against had referenced FastV. Since these requests were beyond a reasonable scope, we chose not to address them further. Subsequently, we received the email that Liang Chen posted on the OpenReview platform.

Dear ICLR 2025 Program Chair, Shaolei and All,

I must point out that claiming "prior to receiving an email from Liang Chen on March 2, 2025, we were unaware of the existence of FastV" is not a valid excuse for several reasons:

• Unawareness itself is not verifiable. In fact we have evidence indicating LLaVA-Mini team was familiar with FastV as early as March 2024 (1 year ago). We can show the evidence if it is required by PC.
• Failing to acknowledge highly related work constitutes a significant oversight. One of the baselines you cited (https://openreview.net/forum?id=gZnBI7WS1K) was rejected by ICLR 2025 specifically for "unawareness of FastV." according to the reviewers. It is inequitable that the same oversight is treated differently in this case.
• Even by the author's own description of LLaVAMini ("LLaVA-Mini processes vision tokens extracted by the vision encoder, whereas FastV reduces the number of vision tokens at the intermediate layers of the LLM"), the close relationship between the two approaches is so evident that your claim "LLaVA-Mini and FastV differ entirely in method and results" seems highly questionable and difficult to reconcile with the technical details.

Thank you for your attention to this serious matter.