Unified Generative and Discriminative Training for Multi-modal Large Language Models

Table B: Comparison with baseline on 11 visual-language benchmark with equitable data volume.

We also present the results in Table B of the Rebuttal PDF.

We sincerely thank you for your comprehensive comments and constructive advice. We will explain your concern as follows.

Q1: This paper did not discussed the impact of including interleaved image-text pairs in MLLM learning. For example, how will it affect the performance on basic visual-language benchmarks (Table 1) and image-text retrieval. Will there be any negative effects?

A1: Thank you for your insightful question. We conducted additional ablation experiments (Table F of the Rebuttal PDF) by gradually increasing the MMC4 sampling ratio in basic visual-language benchmarks (Table 1) and image-text retrieval. The results demonstrated the following effects of interleaved image-text pairs on MLLM learning.

(1.1) MMC4 does not directly contribute to performance improvement for basic visual-language benchmarks. This is because its long sequence undermines the model's interleaved ability (e.g., VQA dropped from 79.9 to 60.5 when MMC4 radio increased). Additionally, without Sugar, MMC4 is unable to assist in image-text retrieval.

(1.2) (a) Delightfully, using a synergistic framework in Sugar can significantly enhance the capability of interleaved text while maximizing the retention of basic visual-language capabilities as the MMC4 data increases. Moreover, it will facilitate certain tasks that require distinguishing between multiple images or texts, such as POPE (85.5 to 86.4）and multi-image VQA(from 38.4 to 39.5). (b) Only Sugar can simultaneously achieve image-text retrieval. With the increase in MMC4 data, it can significantly improve its performance in complex retrieval tasks (increasing from 14.4 to 23.2) while maintaining competitive performance in general retrieval tasks.

Please refer to Table F of the Rebuttal PDF for the Results. The table below is the same as in the PDF

Q2: Can Sugar better leverage the multi-modal in-context examples or better understand interleaved image-text content, is there any evaluation for that?

A2: Thank you for your constructive suggestions. Following your recommendations, we have conducted additional experiments on 5 benchmarks across 3 task types to further evaluate the effectiveness of our method in Table E.

To effectively answer multi-image QA, the model must clearly distinguish between different images and understand their common global information. In in-context interaction, the model must fully comprehend the preceding image-text dialogue to provide reasonable responses. For visual prompt comprehension, the model needs to meticulously identify visual cues in the images to answer questions. Our model achieved the best results across all three task types, especially in VIST-SIS, which involves 5 rounds of interleaved image-text dialogue.

Table E Comparison with baseline in 5 challenging tasks

We also present the results in Table E of the Rebuttal PDF.

Q3: What is exactly the amount of interleaved image-text sequences (from MMC4) and image-text pairs (from other datasets) used to train Sugar.

A3: Thank you for your question. We apologize if our manuscript gave the impression that the training recipe for Sugar was unclear. The specific data usage and comparisons are detailed in the Table A. We did not use any additional data beyond what baseline used.

We utilized a limited dataset in comparison to VILA and LLaVA-1.5. Nonetheless, we achieved competitive outcomes and acquired additional retrieval and retrieval-augmented capabilities. By employing an equivalent data volume and integrating the 300K SFT data used by VILA， we observed improvements in 12 multi-modal comprehension capabilities (Table B,C), showcasing the model's scalability.

Q4: What is the context window size of Sugar?

A4: Thank you for your insightful query! Due to the context length limitation of Vicuna-7B, which is 4096 tokens. So, Sugar cannot handle very long multimedia documents. However, we believe that the mutual enhancement of generative and discriminative capabilities demonstrated by Sugar, as well as the exploration of integrating retrieval and generation within a single model, are beneficial. Your question is valuable and will guide our future improvements and research directions.

Thank you once again for your recognition and constructive suggestions, which have been instrumental in enhancing the quality of our research!

Table F: Ablation study on the effects of MMC4. We evaluated basic visual-language benchmarks, including VQA and GQA. For image-text retrieval, we reported MSCOCO R@5 for both text-to-image(t2i) and image-to-text (i2t). Additionally, we included Mantis for multi-image VQA and VIST R@5 for complex retrieval tasks.

We also present the results in Table F of the Rebuttal PDF.

Table E: Comparison with baseline in 5 challenging tasks.

We also present the results in Table E of the Rebuttal PDF.

Dear Reviewer DDMF,

Thank you for the time and effort you have dedicated to reviewing our submission. We hope we have addressed the concerns raised in your initial reviews and eagerly await your thoughts and further guidance to refine our work. As the author-reviewer discussion period for NeurIPS 2024 will be over soon, please let us know if you require any additional information or clarification from our end. We are open to engage in further discussions to enhance our submission. Thank you!

Table E: Comparison with baseline in 5 challenging tasks.

We also present the results in Table E of the Rebuttal PDF.

Table D: Knowledge-based VQA.

We also present the results in Table D of the Rebuttal PDF.

Table C: DEMON benchmark with equitable data volume.

We also present the results in Table C of the Rebuttal PDF.

Table B: Comparison with baseline on 11 visual-language benchmark with equitable data volume.

We also present the results in Table B of the Rebuttal PDF.

We sincerely thank you for the valuable comments and we will explain your concern as follows.

Q1: While the paper shows impressive results, there is limited discussion on the potential limitations and areas where the model might underperform.

A1: Thank you very much for your valuable question! Our potential limitations are as follows:

(1.1) Firstly, the context length limitation of Vicuna, capped at 4096 tokens, poses a significant challenge. This restricts Sugar's ability to handle very long multimedia documents.

(1.2) Moreover, real-world applications often present varied and dynamic data, which might not align perfectly with the model's training data. This discrepancy could lead to suboptimal performance in diverse and unforeseen contexts.

(1.3) The limitation of our model lies in its reliance on the existing Vicuna and CLIP encoder, which is suboptimal and somewhat limits the comprehension ability [Karamcheti et al., 2024]. For example, there are some hallucinations.

[Karamcheti et al., 2024] Prismatic VLMS: Investigating the Design Space of Visually-Conditioned Language Models. ICML, 2024.

Q2: The paper primarily focuses on specific benchmarks. It would be beneficial to discuss how well the proposed method generalizes to other types of vision-language tasks not covered in the experiments.

A2: Thank you for your suggestion. In the original manuscript, we primarily used the single-image QA benchmarks from LLaVA-1.5 to evaluate our model's multi-modal comprehension capabilities. To provide a more comprehensive evaluation of our model, we have supplemented our method with 5 benchmarks in Table E of the Rebuttal PDF.

To answer multi-image QA, the model must clearly distinguish between different images and understand their common global information. In in-context interaction, the model must fully comprehend the preceding image-text dialogue to provide reasonable responses (Visual Dialog and VIST-SIS, involve 10 and 5 rounds of dialogue respectively). For visual prompt comprehension, the model must meticulously identify visual cues in the images to answer questions. Our model achieved the best results across all three task types.

Q3: Can you provide more detailed ablation studies to understand the contribution of each component of the proposed method, such as the dynamic sequence alignment and the GAK?

A3: To test the effectiveness of our method, we introduced two variants of GAK: GMin and GMax, and examined the impact of the hyperparameter $\delta$. To avoid interference from additional data, we explored the impact of different parameters without using any extra task-specific data. We tested two variants of the kernel function:

GMin kernel: $K_{GMin}(\mathbf{x}, \mathbf{y}) =\min_{\pi \in \mathcal{A}(\mathbf{x}, \mathbf{y})} \prod_{i=1}^{|\pi|} e^{-\phi_\sigma} \in (0, 1]$,

GMax kernel$: K_{GMax}(\mathbf{x}, \mathbf{y}) =\max_{\pi \in \mathcal{A}(\mathbf{x}, \mathbf{y})} \prod_{i=1}^{|\pi|} e^{-\phi_\sigma} \in (0, 1]$.

Based on the experimental results in Table G, we found that:

1. GAK is relatively stable and not very sensitive to the hyperparameter δ.
2. GAK generally performs better compared to GMin and GMax.
3. Tasks requiring global information, such as VIST and KGQA, and those needing detailed semantics, like Winground, are the most sensitive to changes in the method.

Q4: Considering the connection with real-world scenarios is necessary. The authors need to discuss: what requirements might the tasks introduced in the paper have in real-world scenarios?

A4: Thank you for your insightful question. Below, we outline two valuable real-world applications:

(4.1) Complex information retrieval. Real-world information retrieval requires models to search through interleaved multi-modal sequences , as articles, websites, and books are often composed of extensive interwoven image and text sequences. Current retrievers like CLIP and Sentence BERT are designed to handle single images or texts and struggle with complex interleaved retrieval tasks. In Section 4.4, we tested complex retrieval scenarios and achieved promising results. Adapting to more complex scenarios is an important direction for future research.

(4.2) Complex comprehension tasks, such as the 5 new tasks mentioned in Q2, which respectively require capturing global semantics and detailed semantic differentiation.

(4.3) Knowledge retrieval for answering questions. In reality, many questions require external knowledge for accurate answers. For tasks such as FVQA and WebQA, traditional methods first require the retriever to find relevant knowledge in a knowledge base, and then the generator to formulate the answer. The final performance, therefore, depends on both the retriever and generator models, highlighting compatibility issues and sub-optimal issues between them. Based on the Table D, we can observe the following:

(4.3.1) MLLM can answer a small portion of FVQA questions using internal knowledge, but it still requires some knowledge from a retriever for accurate responses.

(4.3.2) The impact of retrieval strategies on the results is inconsistent. For instance, text-based retrieval strategies often outperform image-based ones in FVQA, whereas in WebQA, image-based retrieval is more effective.

(4.3.3) There are also compatibility issues between retriever and models. For example, in WebQA, VILA is more sensitive to retrieval strategy, exhibiting fluctuations 3.4 times greater than those of LLaVA-1.5.

(4.3.4) Our integrated retriever and generator model does not require additional retrievers, thereby avoiding the optimization and selection issues mentioned above.

We hope this clarifies your concerns. We are committed to thoroughly incorporating your suggestions in the next version of the paper. Thank you once again for your excellent feedback.

Dear Reviewer Lcdg,

Thank you once again for reviewing our submission. We are deeply grateful for your recognition of the value of our work, particularly our proposal of the unified approach that combines generative and discriminative training paradigms to address the challenges of global semantic understanding and detailed semantic capture in existing VLMs. In response to your thoughtful suggestions regarding performance across various vision-language task types and real-world applications, we have conducted additional experiments on 5 more challenging benchmarks, as well as a knowledge-based VQA task of practical significance. These efforts further demonstrate the effectiveness and practical value of our method.

Over the past few days, we have carefully addressed the concerns of other reviewers. They have kindly raised their scores and agreed that our approach not only addresses the limitations of VLMs and enhances comprehension capabilities, but also offers emergent insights into complex retrieval tasks and the unified nature of generative and retrieval abilities, which could inspire future work.

As the author-reviewer discussion period for NeurIPS 2024 is drawing to a close, please feel free to let us know if you have any remaining concerns or questions. We greatly appreciate your feedback and look forward to any further insights you may have.

Table B: Comparison with baseline on 11 visual-language benchmark with equitable data volume.

We also present the results in Table B of the Rebuttal PDF.

We sincerely appreciate your constructive and insightful comments. We will explain your concerns point by point.

First, we must clarify that we only used a very limited amount of data (see Table A of the Rebuttal PDF). Despite this, we have effectively integrated the generative and discriminative paradigms, alleviating the challenges remain in generative paradigms and collaterally enabling our model to tackle many other tasks. Such as complex retrieval tasks (Section 4.4), multi-modal retrieval-augmented generation (Section 4.5), and knowledge-based VQA in one model.

Q1: It's unclear what is dynamic time warping framework.

A1: Thank you for your question! We apologize for not clearly explaining our method and the dynamic time warping framework. Below, we will elaborate on DTW and our contributions.

(1.1) The Dynamic Time Warping (DTW) framework is a dynamic programming-based method for calculating the similarity of unequal length sequences, commonly used in text information retrieval [Müller M., 2007] [Cuturi M, Blondel M., 2017] and sequence matching problems [Cao et al., 2020].

(1.2) To enable the current separated generative and discriminative paradigms to achieve synergistic gains, we primarily face two challenges: (a) comprehensively capturing the global semantics for interleaved sequences of unequal length; (b) keenly differentiating the detailed semantics.

(1.3) We explicitly designed two corresponding modules in Sugar to tackle these challenges: (a) formulate the relationship between interleaved sequences as a dynamic sequence alignment problem within the DTW framework; (b) integrate a novel kernel into the framework to leverage the strengths of various discriminative pre-trained models. The ablation experiments validate the effectiveness of our approach (Section 4.5 and Table G).

[Müller, 2007] Dynamic Time Warping. Information Retrieval for Music and Motion.

[Cuturi & Blondel, 2017] Soft-DTW: A Differentiable Loss Function for Time-Series. ICML 2017.

[Cao et al., 2020] Few-Shot Video Classification via Temporal Alignment. CVPR 2020.

Q2: The performance improvement of the proposed method Sugar is not significant. Compared with some baselines, such as VILA and LLaVA-1.5, Sugar performs worse than them on many tasks, as shown in Table 1. This raises concerns about the effectiveness of the proposed method.

A2: Thank you for raising an important concern. We will address your question from the following three aspects:

(2.1) We used a very limited amount of data compared to VILA. Despite this, we achieved competitive results and gained additional retrieval and retrieval-augmented capabilities. By using an equitable data volume (incorporating 300K SFT data used by VILA), we observed enhancements in 11 comprehension benchmarks of Table 1 (Table B,C), demonstrating our model's scalability.

(2.2) In the original manuscript Table 1, we primarily used the single-image QA benchmarks. To provide a more comprehensive assessment of our model, we have supplemented our method with 5 new, more challenging benchmarks, showcasing our superiority (see Table E). To answer multi-image QA, the model must clearly distinguish between different images and understand their shared global information. For in-context interaction, it must fully comprehend the preceding multiple image-text dialogue to provide reasonable responses. For visual prompt comprehension, the model needs to accurately identify visual cues in the images to answer questions. Our model achieved the best results across all three task types.

Q3: This could be meaningless to align a visual token and a text token in an MLLM model since the LLM is trained with next-token prediction instead ... (omitted)

A3: We apologize if our manuscript gave the impression that the method of Sugar was unclear.

(3.1) Simply using contrastive learning methods like CLIP can only calculate the similarity between single images and texts. However, we consider the interleaved image-text sequence as the general format of input samples. This means that the number of tokens in different sequences is of unequal length. If we simply use contrastive learning like CLIP, the similarity of interleaved sequences would lose global information.

(3.2) Hence, we select the final token of a sequence to represent this input interleaved sequence. As you mentioned, 'the current token is conditioned on previous tokens' , the final token can integrate global information from the preceding content. In this way, we can modulate the hidden states of the MLLM by leveraging the semantic relationships between interleaved input sequences, thereby ensuring consistency between hidden states and encouraging the MLLM to capture the global semantics. Additionally, we introduced a novel kernel to provide more detailed signals, enhancing the ability to differentiate fine-grained semantics.

(3.3) The effectiveness of our method is demonstrated through experiments on 11 comprehension benchmarks and others VILA can't do (Table B,C,D,E).

Q4: What is the evaluation protocol? Does Sugar train on each benchmark first then evaluate or directly zero-shot evaluation? In the former case, will Sugar lose generative ability after training with discriminative task data?

A4: All the experiments in our paper are zero-shot. We have merely introduced a structure-induced training strategy within the generative paradigm and subsequently conducted zero-shot evaluations for each task.

Q5: What is the training recipe of the proposed method?

A5: We apologize if our manuscript suggested that Sugar's training method was unclear. The specific data we used is detailed in Table A of the Rebuttal PDF.

We hope the above discussion addresses your concerns. The discussion remains open, and we always welcome your review.

Table E: Comparison with baseline in 5 challenging tasks.

We also present the results in Table E of the Rebuttal PDF.

Table D: Knowledge-based VQA.

We also present the results in Table D of the Rebuttal PDF.

Dear Reviewer 4VP3,

Thank you for the time and effort you have dedicated to reviewing our submission. We hope we have addressed the concerns raised in your initial reviews and eagerly await your thoughts and further guidance to refine our work. As the author-reviewer discussion period for NeurIPS 2024 is half past, please let us know if you require any additional information or clarification from our end. We are open to engage in further discussions to enhance our submission.

Dear Reviewer 4VP3,

Thank you again for reviewing our submission. As the author-reviewer discussion period for NeurIPS 2024 is nearly over, please let us know if any further information or clarification is needed. We are ready to engage in any further discussions with you！ Looking forward to your further feedback!