Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models

We deeply appreciate your comprehensive review and are grateful for your positive remarks about our novel dataset, benchmark, and the thoroughness of our supplementary materials. We acknowledge your concerns and would like to address them and your question in the following response.

---

1: Regarding Image Numbers and Dataset Diversity

As shown in Table 1, our dataset spans one to four images across one or two dialogue turns and is not limited to two images. Additionally, the model can generalize to scenarios with more than four images or two turns. Additionally, we include examples with four or five images in Appendix F of the revised manuscript to demonstrate this expanded capability. We acknowledge your point on diversifying conversation patterns and plan to introduce additional templates and user-initiated conversation structures to expand the range of dialogue scenarios.

2: Clarifying Model Contribution

Our contribution to the model is the open-source SparklesChat to support chats across multiple images and dialogue turns. This capability has been a frequent request in issue trackers that remain unresolved since the release of popular multimodal instruction-following models such as MiniGPT-4 and LLaVA (see issues 1, 2, and 3. We believe SparkelsChat is contributing to the community by pushing the boundaries of existing architectures to address novel challenges.

3: Scale of SparklesDialogue Dataset

The instructional capability of SparklesChat primarily stems from an instruction-tuned LLM (Vicuna). SparklesDialogue is designed to provide dialogue data to learn alignments involving multiple images and turns. With a total of 12K dialogue turn samples (4.5K x 2 + 2K x 2), our dataset is more than triple the size of MiniGPT-4's fine-tuning set (3.5K image-description pairs). We are committed to further expanding the dataset and enhancing its diversity to serve as a training and evaluation resource for multimodal dialogue models.

4: Broadening Baseline Models:

SparklesChat is built upon the MiniGPT-4 and adapted to accommodate multiple images due to its relevance and prominence in the field. We extended our approach to more advanced models such as LLaVA-v1.5to offer a broader view of our approach's effectiveness. LLaVA-v1.5 has improved its image input resolution from 224 pixels to 336 pixels. It is trained on a diverse dataset comprising about 665K instruction-following data. Additionally, LLaVA-v1.5* is fine-tuned on SparklesDialogue (about 6.5K) using the low-resource technique LoRA to conserve memory and time. The results are shown below and have updated in the revised manuscript in Section 5.1 and Table 2.

According to the results, despite LLaVA-v1.5's advantages of higher image resolution (336 vs. 224 pixels) and a larger training set (665K vs. 6.5K), SparklesChat significantly outperforms LLaVA-v1.5 in three evaluation sets involving multiple images. While LLaVA-v1.5 outperforms MiniGPT-4 on BISON and NLVR2, it shows weaker results on SparklesEval. This may be due to LLaVA-v1.5's training data primarily focusing on closed-set multimodal tasks such as VQA, TextCaps, and RefCOCO, while lacking in open-ended dialogue training. After fine-tuning with our SparklesDialogue, LLaVA-v1.5* not only significantly improved in open-ended dialogue tasks but enhanced traditional multimodal tasks. These results validate the adaptability of our method in unlocking chats across multiple images for multimodal instruction-following models with minimal additional training cost.

5: Choice of Evaluation Dataset such as BISON and NLVR2

NLVR and BISON were selected for their use of natural images and the feasibility of automatic evaluation with unambiguous answers (e.g., TRUE/FALSE, single-choice problems). Based on your feedback, we aim to include broader evaluation benchmarks, incorporating images from diverse domains (e.g., documents, fashion, cartoons) and a wider range of evaluation metrics beyond accuracy measures.

Thank you for your thorough review and the valuable insights! We appreciate your recognition of our new dataset and benchmark. In response to your concerns, we would like to provide the following clarifications and address the questions raised.

---

1. Comparison with GPT-4

SparklesChat underperforms compared to GPT-4, despite using richer image representations, because GPT-4 operates based on detailed image annotations and the evaluation set was generated by GPT-4. This gives GPT-4 an inherent advantage in image understanding tasks, as it does not face the same challenges in interpreting images.

2. Model Contribution

Thank you for pointing out that the dataset is our main contribution. Our contribution to the model is the open-source SparklesChat to support chats across multiple images and dialogue turns. This capability has been a frequent request in issue trackers that remain unresolved since the release of popular multimodal instruction-following models such as MiniGPT-4 and LLaVA (see issues 1, 2, and 3). We believe SparkelsChat is contributing to the community by pushing the boundaries of existing architectures to address novel challenges.

3. Adapt to More than Two-Turn Dialogue

We initially created two-turn dialogues to validate our ideas in unlocking chats across multiple images and turns, which is adaptable to more turns, as shown in Figure 8. Additionally, we acknowledge the practicality of longer sessions and plan to expand the dataset accordingly. This can be achieved by incorporating more turns of conversation based on the current two-turn dialogues or exploring new scenarios for images and text from diverse domains (e.g., documents, fashion, cartoons).

4. Extension to Other Approaches such as LLaVA

In light of your feedback, we extended our approach to more advanced models such as LLaVA-v1.5to offer a broader view of our approach's effectiveness. LLaVA-v1.5 has improved its image input resolution from 224 pixels to 336 pixels. It is trained on a diverse dataset comprising about 665K instruction-following data. Additionally, LLaVA-v1.5* is fine-tuned on SparklesDialogue (about 6.5K) using the low-resource technique LoRA to conserve memory and time. The results are shown below and have updated in the revised manuscript in Section 5.1 and Table 2.

According to the results, despite LLaVA-v1.5's advantages of higher image resolution (336 vs. 224 pixels) and a larger training set (665K vs. 6.5K), SparklesChat significantly outperforms LLaVA-v1.5 in three evaluation sets involving multiple images. While LLaVA-v1.5 outperforms MiniGPT-4 on BISON and NLVR2, it shows weaker results on SparklesEval. This may be due to LLaVA-v1.5's training data primarily focusing on closed-set multimodal tasks such as VQA, TextCaps, and RefCOCO, while lacking in open-ended dialogue training. After fine-tuning with our SparklesDialogue, LLaVA-v1.5* not only significantly improved in open-ended dialogue tasks but enhanced traditional multimodal tasks. These results validate the adaptability of our method in unlocking chats across multiple images for multimodal instruction-following models with minimal additional training cost.

5. Similarity of Sources in SparklesDialogueVG and SparklesEval

Although there are similarities in image sources between SparklesDialogueVG and SparklesEval, the test set features unique combinations (e.g., 2-2 image pairings not present in training). The test set is carefully curated to ensure diversity, allowing for a comprehensive evaluation of multi-image dialogue capabilities. We acknowledge the similarity in image sources as a concern and plan to introduce additional data sources to provide a more balanced assessment.

6. Ethics Concerns

Our dataset and evaluation process has involved initial human review, including manual initiation and selection of dialogue demonstrations and rigorous quality checks, as detailed in Section 4.1 of our paper. We acknowledge the importance of thorough human review to ensure the safety and ethical soundness of the data.

7. Correction Acknowledged

Thank you for pointing out the typo. We have corrected it.

Thank you for your insightful and constructive review of our work. We are encouraged by your recognition of the novelty and potential of our approach. We would like to address your concerns regarding the prioritization of single-image versus multi-image comprehension and other aspects related to our dataset and model capabilities.

---

1. Prioritization of Single vs. Multi-Image Comprehension

   • We acknowledge the importance of single-image understanding in visual comprehension. However, our focus on multi-image dialogues stems from real-world complexity, where individuals often encounter and discuss multiple images simultaneously.
   • Since the release of existing multimodal models such as MiniGPT-4 and LLaVA, there have been requests for multi-image capabilities in issue trackers that remain unresolved (see issues 1, 2, and 3). Our work aims to address these demands with our dataset and methodology.
   • What's more, advancing multi-image dialogue capabilities complements single-image comprehension. Tackling complex scenarios with multiple images helps uncover limitations in single-image understanding and drives the field forward.

2. Dialogue Demonstrations' Contribution to Data Quality and Diversity:

   • Thank you for your question! We add examples and analysis in Appendix K, to show that Dialogue Demonstrations act as contextual learning examples, guiding GPT-4 to produce well-formatted and diverse responses.
   • To demonstrate this effect, we modified the original prompt used for GPT-assisted Multiple Dialogues Generation, as detailed in Table 9, by removing content relating to demonstration dialogues. We then employed the same Candidate Image Descriptions as in Figure 18 to create a new prompt and generate a response.
   • The resulting response was inferior in quality, failing to meet the desired formatting criteria, such as assigning image IDs, specifying the number of images per dialogue turn, and incorporating new images in subsequent turns. Furthermore, the response lacked the diversity that its dialogues typically ask for more detailed descriptions of images but not specifying particular aspects. In conclusion, dialogue demonstrations are crucial not only for enhancing data quality by providing formatting guidelines but also for increasing diversity by conditioning different demonstrations.

3. Baseline with Strong Caption Model and Instruction-Tuned LM We appreciate your suggestion to consider a baseline using a strong caption model combined with an instruction-tuned language model. We analyze the advantages of multimodal models over this baseline lie in that (1) some images are challenging to describe solely through language, as visual elements can be more expressive than words, and (2) multimodal models can interpret and discuss visual elements in specific contexts. In light of your suggestion, we plan to experiment with this in future iterations to provide a more comprehensive comparison.

4. Capability with More than Two Images:

   • SparklesChat is capable of handling dialogues involving multiple images, not limited to two. We include examples with four or five images in Appendix F of the revised manuscript to demonstrate this expanded capability.
   • To analyze SparklesChat's ability to generalize in scenarios involving a larger number of images, we consolidated several questions from a dialogue into a single query. This was done to generate responses for queries involving four or five images, as shown in Figure 9 and Figure 10 in the updated manuscript, where the model effectively distinguishes between the images and adheres to complex queries. In the case of Figure 10, where three questions involving five images are concatenated into one query, the model tends to ignore the final question and only responds to the first two. We believe this limitation arises from the absence of similar patterns in the training data. A potential solution could involve incorporating multiple turns into each training dialogue to enhance the model's ability to handle such complex scenarios.

Thank you for your valuable feedback! We appreciate your recognition of the novelty of our scenario and our contributions to the field in introducing new training data and benchmarks. Below, we address your concerns regarding the data construction process and our experimental setup.

---

1. Data Construction Process:

   • Our approach to generating visual dialogue with multiple images extends beyond simple image-text pair linking or descriptions. We instruct GPT-4 to simulate realistic user-assistant interactions as introduced in Section 4.1, and not to directly describe images with the prompt "Please make sure not to reveal the content of the images or describe the images in the user messages" as shown in Table 8. Thus, the generated dialogues are not mere concatenations of descriptions but are contextually rich and coherent conversations.
   • Sometimes, a detailed image description is part of the image understanding and reasoning process. The case in Figure 3 is picked to show a response that clearly distinguishes between multiple images. Meanwhile, Figure 19 (original Figure 17) presents a more complex example of generating a promotional text inspired by several images but not describing them naively.
   • The statistics in Figure 4 further illustrate the diversity of SparklesDialogue, showcasing its range from generating text materials to seeking advice or discussing relationships between images. This involves reasoning about the content and context of the images, connecting different visual elements, and generating responses that reflect a deep understanding of the images about the user's queries.

2. Expanded Experiments and Benchmarks:

   Thanks for your constructive suggestion!

   Expanded Baselines. SparklesChat is built upon the MiniGPT-4 and adapted to accommodate multiple images due to its relevance and prominence in the field. We extended our approach to more advanced models such as LLaVA-v1.5 to offer a broader view of our approach's effectiveness. LLaVA-v1.5 has improved its image input resolution from 224 pixels to 336 pixels. It is trained on a diverse dataset comprising about 665K instruction-following data. Additionally, LLaVA-v1.5* is fine-tuned on SparklesDialogue (about 6.5K) using the low-resource technique LoRA to conserve memory and time. The results are shown below and have updated in the revised manuscript in Section 5.1 and Table 2.

   Model	Instruction Data	BISON	NLVR2	SparklesEval
   GPT-4	-	-	-	9.26
   MiniGPT-4	description	46.0%	51.3%	3.91
   MiniGPT-4*	description	51.3%	46.7%	3.50
   LLaVA-v1.5	Mixture (665K )	52.7%	53.3%	2.75
   LLaVA-v1.5*	+SparklesDialogue	65.3%	56.7%	7.93
   SparklesChat	description	52.0%	48.0%	3.06
   SparklesChat	reasoning	52.7%	54.0%	6.71
   SparklesChat	SparklesDialogue	56.7%	58.0%	8.56

   According to the results, despite LLaVA-v1.5's advantages of higher image resolution (336 vs. 224 pixels) and a larger training set (665K vs. 6.5K), SparklesChat significantly outperforms LLaVA-v1.5 in three evaluation sets involving multiple images. While LLaVA-v1.5 outperforms MiniGPT-4 on BISON and NLVR2, it shows weaker results on SparklesEval. This may be due to LLaVA-v1.5's training data primarily focusing on closed-set multimodal tasks such as VQA, TextCaps, and RefCOCO, while lacking in open-ended dialogue training. After fine-tuning with our SparklesDialogue, LLaVA-v1.5* not only significantly improved in open-ended dialogue tasks but enhanced traditional multimodal tasks. These results validate the adaptability of our method in unlocking chats across multiple images for multimodal instruction-following models with minimal additional training cost.

   Evaluation Benchmarks. Our SparklesEval benchmark is designed to evaluate open-ended dialogues involving multiple images and turns, using LLMs as judges. Additionally, NLVR and BISON were specifically chosen for their use of natural images and the feasibility of automatic evaluation with unambiguous answers, such as TRUE/FALSE or single-choice problems. In response to your suggestion, we are expanding our experimental framework to include broader evaluation benchmarks, incorporating images from diverse domains (e.g., documents, fashion, cartoons) and a wider range of evaluation metrics beyond accuracy measures. Additionally, we plan to diversify SparklesEval with different distributions of images and text to reduce potential biases.