LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

Continue training from image model or joint training.

Thank you for the valuable suggestion. We have conducted an ablation experiment to compare direct training with all data formats against continued training from a single-image pre-trained model, and we provide the detailed results below on single-image, multi-image, and video. In this table,

• The first row represents direct training with a combination of single and multi-image data
• The second row illustrates fine-tuning from single-image models (stage-i) using multi-image data and a subset of single-image data.
• The results clearly indicate that continual training yields superior performance.

Thank you for your thoughtful comments and acknowledgment of our work. We have thoroughly responded to your feedback to address your concerns.

Mixed interleaved data formats.

We apologize for not providing sufficient details about our use of mixed data formats.

• In our methodology, each data sample is randomly assigned to either an interleaved format or a front-loaded format, rather than being trained in both formats simultaneously. This approach ensures that the total number of training samples remains constant.
• The observed improvement in performance is not a result of increased data quantity but rather the enhanced diversity introduced by this mixed-format strategy.

In addition, we conducted an ablation study to evaluate the impact of data sufficiency on multi-image performance.

• Setting: We randomly sampled 25%, 50%, and 75% of the M4-Instruct dataset to fine-tune the LLaVA-NeXT-Image model.
• Results: As summarized in the table below, it demonstrates that as the data size increases, the model exhibits strong scaling performance, highlighting the high quality and effectiveness of our M4-Instruct dataset. We also observe that, the performance improvement from 0% to 50% is more substantial than the improvement from 50% to 100%.
• Observation: The results suggest that, for multi-image capabilities, the enhancement of data diversity (achieved in the 0% to 50% range) in M4-Instruct is more critical than merely increasing data quantity.

The authors have done additional experiments and addressed my concerns. I am standing on my previous rating.

Scope of evaluation metrics.

• We conducted qualitative evaluations to assess the emerging capabilities of our model, as presented in Tables 9–12 and 17 in our paper.
• Additionally, we have included new examples in the appendix of our revised submission (Tables 18–20) to further illustrate these capabilities. While a user study would provide additional insights, due to time constraints, we plan to include such studies in future research.

Scalability.

We scaled our model from 0.5B to 14B parameters, with the 14B model representing a robust, large-scale LLM. To illustrate the scaling law, we created a figure (Figure 6, appendix of the revised submission) that demonstrates the model's performance across multi-image evaluations for both in-domain and out-domain metrics, averaged across tasks. The results indicate that scaling up LLMs significantly enhances performance. However, due to limited time and computational resources, further scaling will be explored in future research.

5. Generalization to Unseen Tasks.

We highlighted emerging capabilities of our model through qualitative visualizations.

• For unseen tasks, we conducted evaluations on out-domain data, as shown in Table 1 of our paper. The results demonstrate that our model achieves state-of-the-art performance compared to prior models. However, defining "completely unseen" tasks is inherently challenging, and qualitative evaluations are limited by these ambiguities.
• For now, we demonstrate the model's capabilities on certain unseen tasks via visualization (Tables 9–12 and 17) and have included additional examples in the appendix of our revised submission (Tables 18–20). We will continue to expand this type of evaluation in future research.

Regarding unseen modalities, our current focus is solely on vision and language modalities. Research on other modalities is planned for future work.

Computational efficiency.

Our model training was conducted using 32 A100 GPUs.

For inference speed, we note that since the vision encoder is relatively small compared to the language model, the primary factors affecting performance are the size of the language model and the number of input tokens. The model's complexity scales with the number of input tokens, which includes both image and language tokens. Thus, inference speed and model complexity are more closely tied to token count than to other factors.

To evaluate the inference speed for multi-image scenarios, we based our analysis on the Qwen-7B model, using an average of six images as input. We compared this with two existing models: the single-image model LLaVA-Next-Image and the multi-image model Mantis. The inference speed results are provided below.

It is worth noting that LLaVA-Next-Image employs the AnyRes technique, which typically divides an image into four patches and combines them with the original image, effectively processing five images in total. Despite this, our results indicate that our inference speed is comparable to existing models. Moving forward, we aim to explore more efficient methods for processing multiple images to further optimize performance.

We genuinely appreciate your insightful comments and recognition of our work. We have carefully addressed your feedback with detailed responses and have updated the relevant sections in the revised manuscript to address your concerns.

Details on data curation, quality control, and potential bias.

We apologize for any missing details in the description of our data construction process. In the M4-Instruct dataset, we not only aggregated existing datasets but also created new ones through innovative methods, such as leveraging GPT-4 Vision (GPT-4-V) prompts. For instance, notable contributions is the construction of the MMChat_Twitter_Post, ScanQA, and MagicBrush dataset.

• For the MMChat_Twitter_Post dataset, we utilized images sourced from an existing social media dataset MMChat. These images were input into GPT-4-V, paired with a language prompt specifically designed to generate Twitter-like posts. This process allowed us to create realistic and contextually relevant data entries, enhancing the overall dataset quality.

• For the ScanQA dataset, since the original ScanQA only includes paired question-answer data with 3D point clouds, we sourced the previous ScanNet dataset and identified the mapping relationship between each point cloud and its corresponding video. From each video, we uniformly sampled 16 frames and prompt GPT-4-V to refine the aligned question-answer pairs in ScanQA, constructing multi-image instructions.

• For the MagicBrush dataset, we transformed its original image editing task into various conversational formats. In general, we concatenate the source and target image in the instruction as the condition and set the task as predicting the corresponding editing prompt. For cases where multiple edits are required between two images, we either structure the data into multi-turn conversations, enabling models to progressively interpret the differences, or prompt GPT-4-V to generate a consolidated editing instrution encapsulating several editing points.

It is important to note that we did not directly incorporate existing datasets into our new combined dataset. Instead, we implemented a rigorous data curation process to ensure quality and consistency, which included the following steps:

1. Task Description and Ground-Truth Conversion:

• Generated a task-specific prompt to describe each dataset.
• Converted ground-truth labels into natural language answers for consistency.

2. Quality Assessment of Candidate Datasets:

• Sampled approximately 100 entries from each candidate dataset.
• Evaluated the quality and relevance of the dataset for training.
• Rejected datasets deemed low quality or irrelevant.

3. Filtering of Selected Datasets:

• Removed samples that did not meet token length requirements (e.g., entries with fewer than two images or more than 15 images).

4. Dataset Splitting:

• Divided the newly collected dataset into training and validation subsets.

5. Integration and Performance Evaluation:

• Trained the new dataset alongside previously verified datasets.
• Evaluated its impact on performance for both existing and new tasks.
• Revisited data samples if the new dataset negatively affected performance or exhibited poor validation performance.

6. Conflict Resolution and Refinement:

• Resolved task/prompt conflicts between the new and existing datasets.
• Double-checked data quality during this process.

7. Rebalancing Dataset Sizes:

• Adjusted the representation of datasets to ensure proportional and meaningful contributions to the combined dataset.

Following these procedures, the final dataset was rigorously optimized to maintain performance on previous tasks while excelling on new tasks.

Regarding potential biases, all our data were collected from publicly available datasets, and we did not impose any preferences that might exacerbate biases. However, we recognize the importance of addressing biases in datasets and will investigate this issue further in future work.

We believe that this new dataset represents a valuable resource for future research, thanks to its enhanced quality, diversity, and comprehensive nature.

Technical innovation during training.

Thanks for this insightful question. We summarize the response as below.

1. Unified Training Formats:

• Our main contribution is to integrat diverse data formats, including multi-image, video, 3D, and single-image, into a single interleaved training framework.

2. Preservation of Model Architecture for Ease of Adaptation:

• We achieved the unification without modifying the existing model architecture, ensuring compatibility and adaptability across other models.

3. Simplicity as a Strength:

• We prioritized simplicity over unnecessary complexity, providing practical advantages in implementation and scalability.

4. Recognition of Novelty:

• The novelty and effectiveness of the unified framework were acknowledged by Reviewer jqfW (Strength 1).

Applying the M4-Instruct data to other 7B-sized single-image LMMs.

Thank you for this valuable question. Given that many multi-modal LLMs adopt a LLaVA-like architecture, we first conducted a comparison between Qwen and LLaMA within our model framework. Additionally, as some models you mentioned do not have open-source training code, we performed an ablation study on ShareGPT4V and SPHINX by training it with our M4-Instruct data. The results, as shown below, demonstrate that our constructed dataset provides substantial benefits and utility when applied to other models.

We sincerely appreciate your valuable comments and recognition of our work. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping to address your concerns.

Technical innovation in data.

We apologize for any lacking details about the technical innovation in our data. In the M4-Instruct dataset, we not only aggregated existing datasets but also created new ones through innovative methods, such as leveraging GPT-4 Vision (GPT-4-V) prompts. For instance, notable contributions is the construction of the MMChat_Twitter_Post, ScanQA, and MagicBrush dataset.

• For the MMChat_Twitter_Post dataset, we utilized images sourced from an existing social media dataset MMChat. These images were input into GPT-4-V, paired with a language prompt specifically designed to generate Twitter-like posts. This process allowed us to create realistic and contextually relevant data entries, enhancing the overall dataset quality.

• For the ScanQA dataset, since the original ScanQA only includes paired question-answer data with 3D point clouds, we sourced the previous ScanNet dataset and identified the mapping relationship between each point cloud and its corresponding video. From each video, we uniformly sampled 16 frames and prompt GPT-4-V to refine the aligned question-answer pairs in ScanQA, constructing multi-image instructions.

• For the MagicBrush dataset, we transformed its original image editing task into various conversational formats. In general, we concatenate the source and target image in the instruction as the condition and set the task as predicting the corresponding editing prompt. For cases where multiple edits are required between two images, we either structure the data into multi-turn conversations, enabling models to progressively interpret the differences, or prompt GPT-4-V to generate a consolidated editing instrution encapsulating several editing points.

It is important to note that we did not directly incorporate existing datasets into our new combined dataset. Instead, we implemented a rigorous data curation process to ensure quality and consistency, which included the following steps:

1. Task Description and Ground-Truth Conversion:

• Generated a task-specific prompt to describe each dataset.
• Converted ground-truth labels into natural language answers for consistency.

2. Quality Assessment of Candidate Datasets:

• Sampled approximately 100 entries from each candidate dataset.
• Evaluated the quality and relevance of the dataset for training.
• Rejected datasets deemed low quality or irrelevant.

3. Filtering of Selected Datasets:

• Removed samples that did not meet token length requirements (e.g., entries with fewer than two images or more than 15 images).

4. Dataset Splitting:

• Divided the newly collected dataset into training and validation subsets.

5. Integration and Performance Evaluation:

• Trained the new dataset alongside previously verified datasets.
• Evaluated its impact on performance for both existing and new tasks.
• Revisited data samples if the new dataset negatively affected performance or exhibited poor validation performance.

6. Conflict Resolution and Refinement:

• Resolved task/prompt conflicts between the new and existing datasets.
• Double-checked data quality during this process.

7. Rebalancing Dataset Sizes:

• Adjusted the representation of datasets to ensure proportional and meaningful contributions to the combined dataset.

Following these procedures, the final dataset was rigorously optimized to maintain performance on previous tasks while excelling on new tasks. Therefore, we believe the data curation itself is also innovative.