LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture

Thank you for your insightful and helpful review. We undertake a series of systematic optimizations aimed at scaling Multi-modal LLMs to more than 1K Images efficiently. The model demonstrates impressive performance in potential application domains such as medicine and remote sensing. Below, we will provide detailed responses to each of your questions.

Q1: Absence of performance analysis among hybrid architecture models, pure Transformer models, and Mamba models

We appreciate your constructive feedback. In the original version of the paper, Section 5.1 provided an analysis of the performance across various architectures, encapsulating factors such as prefill time, throughput, and memory usage.

We have now incorporated an additional metric: the maximum throughput as suggested. This essentially measures the increase in batch size to the maximum permissible limit within memory constraints. Furthermore, in response to Reviewer RAbZ (the first reviewer)'s suggestion 1, we have also included an enhanced Mamba baseline in our study. Please refer to lines 384, 393-396 in the revised version.

Below is a brief summary of the key findings from our extended analysis.

Model	Arch.	Active Param.	Prefill (s)	TP (tokens/s)	Mem.(GB)	Max TP (token/s)
Cobra	Mamba	3B	10.2	42.7	29.9	192.1
Falcon-mamba[1]	Mamba	7B	24.3	32.4	48.8	90.7
LLaVA-1.6	Transformer	13B	34.0	14.7	79.4	14.7
LongLLaVA-9B	Hybrid	9B	16.5	62.1	38.7	155.2
LongLLaVA-A13B	Hybrid	13B	25.5	37.6	79.1	37.6

[1] Falcon Mamba: The First Competitive Attention-free 7B Language Model

Q2: Lack of emphasis on the novel contributions compared to existing hybrid architectures, potentially reducing the perceived novelty of the work.

We appreciate the opportunity to clarify the novel contributions of our work. Our contributions can be viewed from two key perspectives: the impact on the hybrid architecture community and the multimodal community.

• Hybrid Architecture Community: The hybrid architecture community has been seeking a compelling application to highlight its potential to the larger research audience. Our work leverages the long-context capabilities and efficient inference required by visual multimodal tasks, which align well with the strengths of hybrid architectures. We are the first to apply hybrid architectures to visual multimodal tasks, demonstrating the feasibility of modality expansion. This provides new insights and confidence for future optimization efforts within the community.

• Multimodal Community: At the time of our work, there was no existing research enabling reasoning over more than 1,000 images using the LLaVA architecture, particularly with efficient real-time deployment. The community lacked guidance on several critical elements, such as designing training strategies for optimal performance, sourcing and processing training data, and establishing appropriate data protocols. Our work, alongside contemporaneous efforts, addresses these gaps. We have conducted one of the few systematic optimization studies focused on processing 1,000 images efficiently. Unlike other methods with inference times spanning tens of minutes to hours, our approach processes 1,000 images in under 3 minutes, facilitating practical deployment and enhancing the model's utility for the community.

Section 6 of the paper outlines the application of LongLLaVA in domains such as medical and remote sensing imagery. The model achieves near state-of-the-art performance with only a few hours of training, and also excels in long-tail scenarios through a many-shot in-context learning (ICL) approach without fine-tuning. This highlights the robustness and efficiency of our approach as a foundational model, driving advancements in related fields and optimizing task-specific performance.

Thank you for your insightful and helpful review. We undertake a series of systematic optimizations aimed at scaling Multi-modal LLMs to more than 1K Images efficiently. The model demonstrates impressive performance in potential application domains such as medicine and remote sensing. Below, we will provide detailed responses to each of your questions.

Q1: Lack of an ablation study for the 3-stage versus 1-stage fine-tuning process

• Clarification of the Training Stages: To provide a comprehensive understanding of the training stages, we summarize the divisions used in related works.

  Model	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
  LLaVA	Multimodal Alignment Pre-training	Visual Instruction Tuning
  InternVL	Contrastive Pre-training	Generative Pre-training	Visual Instruction Tuning
  Qwen-VL	Multimodal Alignment Pre-training	Multi-task Pre-training	Visual Instruction Tuning
  VILA	Multimodal Alignment Pre-training	Multimodal Alignment Pre-training (interleaved)	Visual Instruction Tuning
  LongVA	Context Extension for LLMs	Aligning Long LM with Short Vision Data
  LongVILA	Multimodal Alignment Pre-training	Multimodal Alignment Pre-training (interleaved)	Visual Instruction-tuning (short context)	Context Extension for LLMs	Visual Instruction Tuning (long video)
  LongLLaVA	Multimodal Alignment Pre-training	Visual Instruction Tuning (single-image)	Visual Instruction Tuning (multi-image)

  Although different works adopt varying stage divisions, they all follow the intuitive progression from simpler to more complex tasks. LongLLaVA adheres to this principle by training first on single images and then on multiple images, aligning with curriculum learning insights. While single-stage training offers a simpler protocol and continuous learning rates, multi-stage training allows for step-by-step verification of the model's capabilities, making the process more controlled.

• Supplementary Ablation Experiments: In response to your suggestion, we conducted an ablation study to evaluate the necessity of separating training stages, using LongLLaVA-9B. The results, detailed in the accompanying table, show that multi-stage training achieves better performance on multi-image tasks while maintaining comparable results on single-image tasks. This finding aligns with insights from Section 3.3 of Cambrian-1[1], which also adopted a multi-stage training approach. The relevant results and discussion are presented in lines 365, 370-373 of the revised version of the paper.

  Stage	GQA	MMMU	ScienceQA	SeedBench	Milebench	VideoMME
  Stage1, 2, 3	58.4	34.4	69.9	67.9	46.5	43.7
  Stage1, 2&3	57.6	33.2	70.2	68.4	44.2	42.3
  Stage1&2&3	56.9	32.8	67.2	66.9	42.2	40.1
  & refers to the combination of the stages.

[1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Q2: Lack of qualitative case studies and specificity on optimal subtasks for the proposed method.

Thank you for this valuable feedback to enhance the paper's readability. We provide a detailed explanation below:

• Qualitative Examples: Due to regulatory constraints, we included two examples in Section 6, which demonstrates the applications of LongLLaVA. Please refer to lines 498-509 of the revised version.

• Well-Suited Subtasks: Among all models, LongLLaVA excels in multimodal long-context tasks, particularly in long-context retrieval tasks. As shown in Table 3, it significantly outperforms both proprietary and open-source models on retrieval-related tasks.

Q3: Missing analysis on tasks affected by pooling.

• Performance Impact: As indicated in Table 4, the model's performance on GQA, ScienceQA, SEEDBench, and MileBench degrades when using 2D Pooling (row 4) compared to the non-pooling baseline (row 2).

• Patching Strategy: The degradation caused by pooling can be mitigated using our image patching strategy. For instance, we assessed the model's ability to identify small objects within large images using the V* Bench in Section 5.2. As Figure 5 illustrates, directly resizing large images into smaller ones for understanding limits the model due to the pooling strategy, making it challenging to locate small objects (Sub-image Count = 1). However, by slicing large images and treating them as multi-image tasks, we significantly improved performance by 18.9% (Sub-image Count = 97). For illustrative examples, please also refer to Figure 7 in Section 6.2 showing how slicing enhances the model’s ability to capture finer details.

Q4: Question on whether this new architecture show improved many-shot ICL performance as opposed to finetuning.

Thank you for this insightful suggestion, which prompted us to further explore the model's long-context capabilities. In response, we expanded the evaluation of ICL capability in Section 5.1 of the original paper and detailed in Section 6.3 in the revised version as an application. Since LLMs fine-tuning can be costly and time-consuming, especially when data is scarce or frequent updates are needed. In contrast, Many-shot ICL allows models to use more task-specific examples during inference without retraining. We compared LongLLaVA-9B performance with different shot counts to fine-tuning on the same number of samples.

	VL-ICL 1 Shot	VL-ICL 5 Shot	VL-ICL 10 Shot	VL-ICL 50 Shot	VL-ICL 100 Shot	VL-ICL 1000 Shot
LongLLaVA-9B	51.6	60.2	62.6	66.5	69.2	70.4
After Training	50.1	53.1	53.5	60.2	68.9	71.3

As shown in table above, LongLLaVA-9B outperforms ICL up to 100 shots. Beyond 1000 shots, the benefit of adding more shots diminishes, and fine-tuning becomes more effective. This suggests that ICL is preferable with fewer than 100 samples, while fine-tuning is more beneficial with around 1000 samples.

Dear Reviewer UWnW,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10001 Authors

Are there any concerns or issues that are causing you to remain weak rejection instead of giving positive one? If not, please responsibly reconsider your score as it is important to us. If you have any other questions or need further clarification, please feel free to let us know. Your insights are highly valued.

From the perspective of community progress, weak rejection without any reason is not a responsible behavior in the community, which will result in both parties and the public being unable to benefit from communication.

We would greatly appreciate it if you could express your concerns or questions clearly.

Do you have any thoughtful suggestions for rebuttal itself? If you have any other questions or need further clarification, please feel free to let us know. Your insights are highly valued. We would greatly appreciate it if you could express your concerns or questions clearly.

Thank you for your insightful and helpful review. We undertake a series of systematic optimizations aimed at scaling Multi-modal LLMs to more than 1K Images efficiently. The model demonstrates impressive performance in potential application domains such as medicine and remote sensing. Below, we will provide detailed responses to each of your questions.

Q1: The selection of Expert-0 from the MoE layers for LongLLaVA-9B lacks sufficient justification.

Thank you for highlighting this concern. We chose Expert-0 based on preliminary experiments indicating minimal differences in model performance (measured by MMLU [1] and BBH [2] accuracy) after Pure-text Instruction Tuning, regardless of whether numerical averaging, spherical averaging, or random expert selection was employed. For simplicity, we opted to use Expert-0. As this experiment was not central to the main focus of the paper, detailed descriptions were initially omitted. The relevant experimental results are shown below, and in Appendix A and line 215 of the revised version.

Downcycling Strategy	Arithmetic Mean	Spherical Mean	Expert-0	Expert-5	Expert-12	Expert-15
MMLU	52.7	53.2	53.2	51.9	52.6	52.2
MMLU Aft. Train	53.8	54.3	54.3	53.3	53.8	53.3
BBH	36.7	36.7	37.2	36.7	37.4	36.3
BBH Aft. Train	37.8	37.9	38.4	38.9	38.9	37.9

[1] Measuring Massive Multitask Language Understanding

[2] Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Q2: Lack of LongVA Baseline Results and Discussion About Performance

• Lack of LongVA Baseline:

  We apologize for any inconvenience this may have caused. As noted in the LongVA paper, the LongVA model was trained on synthetic multi-image datasets where images are divided into sub-images, leading to its initial exclusion from our performance comparisons. Following your suggestion, we have now included LongVA in our evaluation to provide a clearer view of its performance relative to other models. Key results are presented in the table below.

• Discussion About Performance:

  As you mentioned, model performance is significantly influenced by factors such as training time, training data, inference FLOPs, and inference precision. LongLLaVA-9B uses the same training data and INT8 inference precision as LongLLaVA-A13B to ensure alignment. The multimodal adaptation training time for LongLLaVA-9B is 46h × 8 × A800-80G, which is 54.8% of LongVA's (84h × 8 × A100-80G) and significantly lower than LongVILA's. Additionally, the FLOPs required for inference are only 3.8% of those needed for the other two models.

  However, as reviewer RAbZ pointed out, this presentation format limits the scope and accessibility of the results for the community. To address this, we have taken two steps: first, we provide the models' FP-16 precision inference results for reference. Second, we introduce a variant, LongLLaVA-9B-MoreData, trained on more data (82h × 8 × A800-80G). Key results are summarized in the following table, demonstrating that the LongLLaVA series achieves comparable performance with high efficiency. Please refer to Table 2 for details.

  Model	Precision	PFLOPs	VideoMME-Short	VideoMME-Medium	VideoMME-Long	VideoMME-Avg.
  LongVILA	FP16	3.90	61.8	49.7	39.7	50.5
  LongVA	FP16	4.90	61.1	50.4	46.2	52.6
  LongLLaVA-9B	INT8	0.15	52.4	42.2	36.4	43.7
  LongLLaVA-A13B	INT8	0.22	60.9	49.7	44.1	51.6
  LongLLaVA-9B (MoreData)	FP16	0.15	58.4	48.3	41.7	49.5
  LongLLaVA-A13B	FP16	0.22	62.9	52.2	46.4	53.8

Q3: How can we ensure the hybrid architecture maintains VLM performance given the skewed comparisons due to differences in baseline performance and VLM training data?

• Clarification on LLaVA-1.5-13B + Jamba in Table 4: This refers to replacing LLaVA-1.5-13B's LLM (Vicuna-1.5) with Jamba (after Pure-text Instruction Tuning), with both trained using the LLaVA-1.5 recipe.

• We understand your concerns, and in our ablation study, we analyzed the impact of the LLM model using the same training recipe. However, as you noted, aligning the initial performance of the LLMs prior to VLM adaptation is crucial. Therefore, we conducted an additional ablation experiment by replacing the LLM base in LLaVA-1.5-13B with Jamba-9B (after pure-text instruction tuning) while following the LLaVA-1.5 training recipe. The initial performance comparison of the large models before VLM adaptation is presented below, showing that the two models exhibit comparable performance.

  	MMLU	BBH
  Vicuna 13B	55.3	40.5
  Jamba-9B	54.3	38.4

• The key experimental results of the related ablation are summarized below. As shown in the table, given the comparable initial performance of the LLMs and using the same training data combination, the hybrid architecture achieves competitive results. Additionally, the hybrid architecture requires fewer FLOPs for inference.

  Model	GQA	MMMU	ScienceQA	SeedBench	MileBench
  LLaVA-1.5-13B	63.3	34.4	71.6	68.2	27.6
  Jamba-9B (same data and training strategy with LLaVA1.5)	62.3	36.2	71.9	70.1	28.2
  Difference	-1.0	+1.8	+0.3	+1.9	+0.6

The relevant experimental details and discussion can be found in lines 372-373 of the revised version and Appendix E.2.

Q4: A lack of ablation studies on the training recipes, particularly regarding the Replay data

• On Merging or Separating Training Stages: Detailed experimental results on whether the various stages should be merged or separated are provided in our response to Reviewer RAbZ (the first reviewer)'s Q3. To enhance your reading experience, we will focus here on the discussion of Replay data.

• Impact of Replay Data:

  • Comparison With and Without Replay Data: As requested, we conducted experiments comparing models trained with and without Replay Data. To eliminate the influence of increased training data introduced by Replay Data, we also conducted an ablation study replacing Replay Data in the original training recipe with equivalent multi-image data. The results, presented in the table below, demonstrate that Replay Data is crucial for maintaining the model's original single-image understanding and text-following capabilities.

    	MMLU	BBH	GQA	MMMU	ScienceQA	SeedBench	MileBench
    LongLLaVA-9B	53.9	38.8	58.4	34.4	69.9	67.9	46.5
    w/o Replay Data	52.3	36.2	57.5	31.2	53.5	64.3	46.8
    replace Replay with Multi-Image	52.6	35.9	57.2	29.8	52.6	63.8	47.2

  • Replay Data Quantity Ablation: Based on your suggestion, we also explored the impact of the quantity of Replay Data.

    • Text: The supplementary experiment examining the impact of data quantity shows that adding text replay data enhances the model's text-following ability, although the improvement eventually saturates.

    	MMLU	BBH
    LongLLaVA-9B (w/o Replay Data)	52.3	36.2
    with 10K Text Replay Data	52.9	37.3
    with 20K Text Replay Data	53.4	38.1
    with 50K Text Replay Data	53.9	38.8
    with 100K Text Replay Data	53.9	39.2

    • Single-Image: The experiment indicates that the model's single-image capability continues to improve with increased data volume and has not yet reached saturation. However, the improvement in multi-image tasks is limited.

      	GQA	MMMU	ScienceQA	SeedBench	MileBench
      LongLLAVA-9B (w/o Replay Data)	57.5	31.2	53.5	64.3	46.8
      with 50K Single-Image Replay Data	57.9	32.3	58.2	66.2	46.5
      with 100K Single-Image Replay Data	57.9	33.5	62.7	67.1	46.5
      with 200K Single-Image Replay Data	58.2	34.5	67.1	67.9	46.8
      with 400K Single-Image Replay Data	58.5	35.2	73.2	68.2	46.4

The relevant experimental details and discussion can be found in lines 372-373 of the revised version and Appendix E.3.

Q5: What does 1D Pooling in Table 4 specifically refer to?

1D pooling involves flattening the image pixels into a sequence and applying pooling operations directly to four consecutive pixels, unlike 2D pooling, which preserves the relative positional information of the image space by pooling in units of 2×2 squares.

Here are examples of 1D and 2D pooling:

• 1D Pooling Example:

  For the following 2D array, 1D pooling applies a max pooling operation on every 4 consecutive pixels in each row:

  Original Array:
  [[ 1,  2,  3,  4]
   [ 5,  6,  7,  8]
   [ 9, 10, 11, 12]
   [13, 14, 15, 16]]
  

  1D Pooling Result (Max pooling across each row with 4 values):

  [[ 4,  8, 12, 16]]
  

• 2D Pooling Example:

  For the same array, 2D pooling applies a max pooling operation on 2x2 blocks:

  Original Array:
  [[ 1,  2,  3,  4]
   [ 5,  6,  7,  8]
   [ 9, 10, 11, 12]
   [13, 14, 15, 16]]
  

  2D Pooling Result (Max pooling with 2x2 blocks):

  [[ 6,  8]
   [14, 16]]
  

In summary, 1D pooling pools along rows of the flattened sequence, while 2D pooling operates on 2x2 blocks, maintaining spatial information.

Dear Reviewer jB9q,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10001 Authors

Dear Reviewer jB9q,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10001 Authors

Dear Reviewer jB9q,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10001 Authors

Q4: Other Issues and Factual Clarifications

• Lack of Needle in a Haystack Benchmark: The relevant results are provided in Appendix C of the original paper and referenced in lines 372–373. In the revised version, we increased the size of the corresponding image (1080 lines) and referenced in lines 348-350 of the text.

• Model Initialization for VLM Training (Figure 2): The model is initialized following pure-text instruction tuning.

• Ring-Attention and Sequence Parallelism: Ring-attention is a type of sequence parallelism method, with implementations similar to DeepSpeed Ulysses [1]. We will clarify this distinction in the revised paper.

• Extra Overhead from Ring-Attention or Sequence Parallelism: Both sequence parallel methods introduce additional communication overhead.

• About the Model Name "LongLLaVA": LongLLaVA (Long-Context Large Language and Vision Assistant) is a data-efficient multimodal adaptation approach, renowned for its multi-stage training methodology. We utilize efficient acceleration techniques to adapt it to long-context scenarios, and we express our respect and tribute to this outstanding work.

[1] DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

Thank you for your insightful and helpful review. We undertake a series of systematic optimizations aimed at scaling Multi-modal LLMs to more than 1K Images efficiently. The model demonstrates impressive performance in potential application domains such as medicine and remote sensing. Below, we will provide detailed responses to each of your questions.

Q1: The lack of evidence or literature supports that ICL is a weakness within the Mamba architecture. And the comparison in Table 5 about this is not fair.

We address this question from two perspectives: literature support and additional experimental evidence.

• Literature Support on the Weakness of Mamba Architecture for ICL Tasks:

  Researches indicate that Mamba architectures perform poorly in some ICL tasks, as noted in [1] and [2]. Specifically, [1] highlights Mamba's limitations in tasks involving non-standard retrieval functionality and introduces hybrid architectures to address these issues. Study [2] suggests that ICL capabilities are linked to specific attention heads that emerge during training.

  We appreciate your inclusion of the training data dimension in the ICL discussion. As mentioned, [1] highlights that while Mamba can learn to replicate simple ICL tasks through explicit training, this approach significantly constrains the full potential of the model’s parameters and training data.

• Literature Support for Hybrid Architectures:

  Interestingly, the authors of the Mamba series have also embraced hybrid architectures, highlighting their potential and performance advantages through comparative experiments (see Section 9.2.3 of Mamba-2 [3]). In a recent paper [4], they further support the distillation and acceleration of hybrid architectures, validating their performance advantages through extensive experiments.

• Additional Experiments:

  We acknowledge concerns regarding the fairness of comparing different architectures in Section 5.1. To address this, we compared our model against the largest available multimodal Mamba architecture at the time. Recently, thanks to the open-source community, we found a 7.3B Mamba language model [5] and trained and evaluated it using the same settings as our model. Due to the difficulty in flexibly adjusting the parameter count of MLLMs, we cannot fully align the number of parameters.

  Model	Arch.	Active Param.	VL-ICL 1 Shot	VL-ICL 2 Shot	VL-ICL 4 shot	VL-ICL 5-shot
  Falcon-mamba	Mamba	7B	49.0	51.9	52.4	53.2
  LongLLaVA-9B	Hybrid	9B	51.6	57.8	58.4	60.2
  LLaVA-1.6	Transformer	13B	50.0	52.3	54.6	58.9

  The key results demonstrate that, as the shot number increased, the hybrid architecture model can effectively utilize the shots compared to Mamba architecture model, especially under the 5-shot setting.

The above discussion and experimental results have been incorporated into the revised version of the paper, presented in a manner that aligns with the overall structure of the article. For further details, please refer to lines 121-137, 386, and 393-396 of the revised version.

 

[1] Can mamba learn how to learn? a comparative study on in-context learning task

[2] In-context Learning and Induction Heads

[3] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

[4] The Mamba in the Llama: Distilling and Accelerating Hybrid Models

[5] Falcon Mamba: The First Competitive Attention-free 7B Language Model

Q2:The model performance is unsatisfactory, particularly for LongLLaVA-9B. Additionally, the baseline and evaluation are neither comprehensive nor adequate.

• Clarification: LongLLaVA-9B is a dense model, not an MoE model.

We apologize for any confusion caused by unclear writing in the original paper. To clarify, LongLLaVA-9B is a dense model, as indicated in lines 213–214 of the original version and lines 195-196 of the revised version.

• Comprehensive Evaluation:

In the original paper, we included seven standard VLM benchmarks, as detailed in Appendix B and discussed in lines 298–299. In the revised version, we have improved the presentation for better readability, with the related details now included in Appendix C and discussed in lines 303. Additionally, following your suggestion, we have added DocVQA and ChartQA to the existing benchmarks and enhanced the presentation logic to avoid further confusion. These updates are covered in lines 1041-1052 of the revised version.

• Addition of LongVA Baseline:

  • Performance Concerns:

    • LongLLaVA-9B uses the same training data and INT8 inference precision as LongLLaVA-A13B. The model's performance is influenced by training time, data, inference FLOPs, and inference precision. The multimodal adaptation training time for LongLLaVA-9B is 46 hours × 8 × A800-80G, which is 54.8% of LongVA's (84 hours × 8 × A100-80G) and significantly lower than LongVILA's. Additionally, the FLOPs required for inference are only 3.8% of those needed for the other two models.

      To address presentation limitations, we now provide the models' FP-16 precision inference results for reference. Additionally, we introduce LongLLaVA-9B-MoreData, trained on the dataset for around two epochs (82 hours × 8 × A800-80G). The key results, summarized in the table below, demonstrate that the LongLLaVA series achieves comparable performance with high efficiency. Please refer to Table 2 for details.

      Model	Precision	PFLOPs	VideoMME-Short	VideoMME-Medium	VideoMME-Long	VideoMME-Avg.
      LongVILA	FP16	3.90	61.8	49.7	39.7	50.5
      LongVA	FP16	4.90	61.1	50.4	46.2	52.6
      LongLLaVA-9B	INT8	0.15	52.4	42.2	36.4	43.7
      LongLLaVA-A13B	INT8	0.22	60.9	49.7	44.1	51.6
      LongLLaVA-9B (More Data)	FP16	0.15	58.4	48.3	41.7	49.5
      LongLLaVA-A13B	FP16	0.22	62.9	52.2	46.4	53.8

  • Lack of LongVA Baseline: We apologize for any inconvenience caused by the omission. The LongVA model was initially excluded from the performance comparison because it was not trained on real multi-image datasets, as noted in the LongVA paper. However, following your suggestion, we have now included LongVA in the evaluation to provide a clearer view of the progress in community model performance. Key results are presented in the table above.

Q3:A lack of ablation studies on the multi-stage training recipes and insufficient clarification of the motivation behind model architecture design details.

• Motivation for Architecture Design Details:

In the hybrid architecture study by the authors of Mamba [1], various hybrid ratios (1:0, 7:1, 3:1, 1:1) were evaluated. The most significant benefit was observed in the transition from 1:0 to 7:1, with diminishing returns beyond that point. To optimize efficiency, we selected the 7:1 hybrid ratio. This decision aligns with findings in Section 6.1 of [2], where the performance difference between 7:1 and 3:1 was negligible, despite increased inference FLOPs. While we acknowledge the trade-offs involved with MoE, future investigations will explore these aspects further.

• Ablation Studies on Multi-stage Training Recipes:

  • Clarification on "Multi-stage" Training:

    We summarize the training stage divisions presented in related works, which generally progress from simpler to more complex tasks.

    Model	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
    LLaVA	Multimodal Alignment Pre-training	Visual Instruction Tuning
    InternVL	Contrastive Pre-training	Generative Pre-training	Visual Instruction Tuning
    Qwen-VL	Multimodal Alignment Pre-training	Multi-task Pre-training	Visual Instruction Tuning
    VILA	Multimodal Alignment Pre-training	Multimodal Alignment Pre-training (interleaved)	Visual Instruction Tuning
    LongVA	Context Extension for LLMs	Aligning Long LM with Short Vision Data
    LongVILA	Multimodal Alignment Pre-training	Multimodal Alignment Pre-training (interleaved)	Visual Instruction-tuning (short context)	Context Extension for LLMs	Visual Instruction Tuning (long video)
    LongLLaVA	Multimodal Alignment Pre-training	Visual Instruction Tuning (single-image)	Visual Instruction Tuning (multi-image)

    Processing single images serves as the foundation for handling multiple images. LongLLaVA adheres to this curriculum-learning insight by first training on single images before advancing to multiple images. While single-stage training offers a simpler protocol and continuous learning rates, multi-stage training allows for step-by-step verification of the model's capabilities, making the process more controllable. As suggested, we recognize the need for an ablation study to provide further clarity.

  • Supplementary Ablation Experiments:

    • We conducted an ablation experiment using LongLLaVA-9B to assess the necessity of separating training stages. The results, detailed in the accompanying table, indicate that multi-stage training achieves better performance on multi-image tasks while maintaining comparable results on single-image tasks. This finding aligns with insights from Section 3.3 of Cambrian-1[3], which also employed a multi-stage training approach. The relevant results and discussion are presented in lines 365, 370-373 of the revised version of the paper.

      Stage	GQA	MMMU	ScienceQA	SeedBench	Milebench	VideoMME
      Stage1, 2, 3	58.4	34.4	69.9	67.9	46.5	43.7
      Stage1, 2&3	57.6	33.2	70.2	68.4	44.2	42.3
      Stage1&2&3	56.9	32.8	67.2	66.9	42.2	40.1
      & refers to the combination of the stages.

[1] The Mamba in the Llama: Distilling and Accelerating Hybrid Models

[2] Jamba: A Hybrid Transformer-Mamba Language Model

[3] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Dear Reviewer RAbZ,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10001 Authors

Dear Reviewer RAbZ,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10001 Authors

Dear Reviewer RAbZ,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 10001 Authors