NExT-GPT: Any-to-Any Multimodal LLM

Thank you so much for highly acknowledging the strengths of our work, such as 'lightweight alignment learning techniques', 'high-quality modality switching instruction-tuning dataset', and 'good and clear formulation', which means a lot to us. Following, we provide our feedback to your concerns and questions.

---

Q1: The model relies on different pretrained models to understand different types of information, like text, images, and audio. The quality of pretraining may directly impact how well the model performs its tasks.
A: Here, we want to clarify that we actually did not utilize many different pre-trained modes to encode each individual modality. In our practice, we take advantage of a unified solution, i.e., ImageBind, to encode and project different types of information. With ImageBind, we are spared from managing many numbers of heterogeneous modal encoders. Furthermore, ImageBind is featured as a SoTA model to effectively understand six modalities. Finally, the model architect presented in this paper facilitates easy and cost-effective extension to powerful pre-trained models, thereby enhancing the understanding capabilities of different modalities.

---

Q2: What is the size of parameters when tuning the model for each modality?
A: Actually, we have clearly stated the information about the size of tunable parameters for each modality in Table 1 of the paper. Specifically, the parameter size for the input alignment module is 4M, while the parameter sizes for the output alignment modules corresponding to image, audio, and video modalities are 31M, 31M, and 32M, respectively.

---

Q3: The work in [1] may be related, can the authors provide a comparison? (not included in rating)
A: Thanks for your suggestion. Truth is, we have checked the reference work [1], and as of now, the checkpoint has not been released. Thus, unfortunately, we are unable to provide a comparison at this moment. We are, however, actively monitoring the progress of that work, and once the checkpoint is released, we will supplement our paper with the subsequent comparative results. Meanwhile, we have included additional experiments, as detailed in reviewer #ijPL's rebuttal, where our results consistently demonstrate comparability or even superiority.

(2-2)

To support the above claim, during the rebuttal period, we conducted experiments to compare with such a pipeline-style baseline. We consider two types of comparisons: 1) Visual-ChatGPT and HuggingGPT, which are existing systems that have free open access; 2) NExT-GPT variant with captions as the messenger (which we mark as NExT-GPT-caption). To implement NExT-GPT-caption, the captions directly generated by LLM will be fed into the following generation models, instead of using the soft representations of the signal tokens. As Visual-ChatGPT only supports image generation, we here consider the evaluation on the Text-to-Text&Image setting.

To evaluate if the system really or how well understands the input and generates output content (response text + image), we perform the human evaluation. For constructing the testing data, we first leverage GPT-4 to synthesize 1) 100 simple instructions (e.g., involving short and simple semantic content) that can explicitly prompt MM-LLMs to generate images, and 2) 100 complex instructions (e.g., involving intricate and semantically-rich scenes) that require implicit reasoning ability to generate image content. Then, the synthesized instructions are fed into the models to generate the response text + image content. Subsequently, five unbiased volunteers evaluate the generated results under three aspects, 1) Instruction following, identifying, among the four models, which of the generated text+image accurately responded to the input instructions, 2) Rationality, determining which of the generated images adhered to the input instructions, 3) Quality, evaluating which of the generated images exhibited the highest quality.

The evaluation results are shown in Table 6&7, where we can notice the interesting observation. On the simple instructions, mostly these four models perform at similar levels. This means the impacts could be quite limited whether we take a pipeline modeling or end-to-end system on the comparatively simple user inputs. But on complex instructions, ours performs significantly better than two existing systems and NExT-GPT-caption in terms of the instruction-following capability and image generation quality. Notably, a notable degradation in the quality of generated images is observed when captions are utilized as messengers compared to the instruction-following performances. This highlights the inadequacy of captions in conveying the necessary information for generating complex images.

Table 6: Human Evaluation (1-100 scale, results are on average) of NExT-GPT on simple instructions in comparison with pipeline baselines that directly generate captions for downstream generation models. The results are.

Table 7: The results on complex instructions.

Thank you sincerely for your valuable time and comments. We've conducted additional experiments and tried every effort to address your concerns. We hope you'll reconsider your evaluation if you find our responses are effective.

---

(2-1)

Q: The major concern I have regarding this paper is the training object during alignment, which is to align the semantic tokens with outputs from text encoders of diffusion models. This seems reasonable at first, but if the objective is to "match the semantics token with textual captions' representations from the text encoders of diffusion models", why not just directly use the diffusion model's text encoder to encode the textual captions? More specifically, why not just let the LLM output a caption according to some fixed format, and extract the caption, then feed it to the diffusion model for generation? I think it would be a more direct approach and probably will enable better performance. Unfortunately, I do not find comparison with this simple alternative in the paper.

A: Thanks for your comment. Actually, the approach you mentioned here that LLM outputs textual captions and feeds to the follow-up diffusion models for generation, is one type of prior existing methods for reaching the goal of (nearly) any-to-any MM-LLM systems, where LLMs are straightfowardly combined with external tools by connecting the LLMs with other modules with explicit textual instructions (e.g., captions or descriptions of content to generate) as the messenger. They essentially have a pipeline architecture. Representative models include Visual-ChatGPT and HuggingGPT. However, as we emphasized in the paper, there can be at least two limitations in such an approach.

Firstly, the information transfer between different modules is entirely based on discrete texts produced by the LLM, where the cascade process inevitably introduces noise and propagates errors. This is a problem rooted in the core of different modalities, i.e., there is an inherent gap that persists between language and other modalities that cannot be eliminated. Utilizing textual captions as intermediate representations runs the risk of overlooking modality-specific features when expressing non-linguistic modalities solely through language. In other words, relying on textual captions as an intermediate representation may create a bottleneck, especially when handling complex or implicit input instructions. For example, prompting the pipeline system to generate a picture of a dog might result in a simple caption like ‘a dog running in a park’ from the LLM. However, we expect the model to generate images with much more detailed and intricate scenes more than just the ‘dog’ and a ‘park’ only, but also include vivid background details like ‘a playground with green grass, with many people walking by’, and the dog itself ‘holding a frisbee in the mouth’. All these can highlight the potential limitations of simply using textual captions as an intermediate step.

Secondly, the entire system only leverages existing pre-trained tools for inference only, limiting its capabilities of content understanding and multimodal generation, especially in interpreting intricate and implicit user instructions. With the continual representations of special signal tokens as the messenger between different modules, we end-to-end unified MM-LLM system in the true sense. The full system can be updated throughout the whole system, which thus helps to have a much stronger capability in better understanding complex user instructions and generating content more accurately and in high quality.

Dear Reviewer,

We would like to thank you again for your efforts and feedback. We have further explained about your concerns. We kindly hope that you can take some time to check our response and re-evaluate our paper based on our replies. If you have any further concerns or questions, please do not hesitate to let us know. We will be happy to address them promptly.

Best

(3-3)

---

Q4: Additionally, in a multimodal language model, text generation is crucial. It would be interesting to know how the system performs on recent benchmarks such as MME [1], MMBench [2], and SEEDBench [3].
A: During the rebuttal period, we have done the experiments as required here. Following we can present the evaluation results for MME, MMBench, and SEEDBench in Table 1, 2 and 3.

Table 1: Evaluation results (%) on MME.

Table 2: CircularEval results on MMBench test set (L-2 abilities), including Logical Reasoning (LR), Attribute Reasoning (AR), Relation Reasoning (RR), Fine-grained Perception (Cross Instance) (FP-C), Fine-grained Perception (Single Instance) (FP-S), and Coarse Perception (CP).

Table 3: SEEDBench evaluation results with accuracy (%) as the primary metric for each task.

Observing the results, our model mostly achieves better text generation performance than the comparing baseline MM-LLMs the reviewer suggested here. But, it is worth highlighting that these benchmarks are quite limited to text generation, whereas our model, pioneering any-to-any generation, extends its capabilities beyond text. In addition to excelling in text responses, our model significantly contributes by proficiently generating diverse content, including image, video, and audio. This broader spectrum of generation capabilities sets our model apart from the benchmarks mentioned. As our next step, we will extend the MM-LLM leaderboard of any-to-any generation based on our MosIT data.

---

Q5: The qualitative comparison provided lacks thoroughness. The paper only presents a few demonstrations and fails to provide comparisons with other MLLMs, including InstructBLIP [4], LLaVA [5], mPLUG-Owl [6] for text generation, and DreamLLM [7] and EMU [8] for conditioned image generation.
A: For this question, we also provide the experimental results, by comparing NExT-GPT with other MLLMs on NoCaps, Flicker 30K, and COCO datasets (under the fair zero-shot setting), and also conditioned image generation results compared with DreamLLM and EMU. We report the CIDEr score for text generation and FID for image generation. From the results, we see that the conclusions drawn in the paper still hold water.

Table 4: Zero-shot evaluation of image-to-text generation with CIDEr score (the higher the better). Results marked with * are copied from the original paper.

Table 5: Zero-shot evaluation of Text-to-Image Generation (COCO) with FID (the lower the better) score. Models pre-trained with LION (2B) data are denoted with *.

(3-2)

---

Q3: What would be the results if the model was trained on a mixture of the proposed MosIT dataset and benchmark datasets, and then evaluated on the benchmarks?

A: To answer your question, we first would like to show how our system is trained. NExT-GPT roughly considers four stages of the training/learning process:

• Stage-1: Encoding-size Alignment Learning. The input projection layer is trained by adopting an X-to-Text generation task on Webvid-2M, CC3M, and AudioCaps dataset.
• Stage-2: Decoding-side Alignment Learning. The three output projection layers are optimized on Webvid-2M, CC3M, and AudioCaps.
• Stage-3: End-to-end Instruction-Tuning. We optimized the whole NExT-GPT using instruction-following datasets, including Text+X → Text Dataset, Text → Text+X Dataset, and MosIT dataset. This step is to enhance the ability of LLM to understand the user input’s instructions.
• Stage-4: In-domain Fine-tuning on Task-specific Training Dataset. NExT-GPT is fine-tuned on the specific in-domain dataset of a specific task, after which the system will be evaluated on the testing set.

It is worth noting that Stage-3, involving instructive tuning with MosIT data, is designed to enhance the model’s proficiency in performing complex, multi-turn, modality-switching interactions. However, in Stage-4, when evaluating on traditional benchmarks, modality alignment without intricate instruction understanding is required essentially. Without instructive tuning with MosIT, the expected impact on performance during traditional benchmark evaluation could be limited or minimal.

With above discussion, for your question about the results if the model was trained on a mixture of the proposed MosIT dataset and benchmark datasets, as the objectives of the benchmark datasets and MosIT dataset are quite distinct, mixing them probably introduces the risk of interference of feature patterns, where the information learned from in-domain task-specific dataset might be disrupted by instructive tuning data. Hence, theoretically and practically, it is likely that such a mixture of training data may lead to a degradation in performance.

We sincerely appreciate your valuable constructive feedback, which will surely enhance our work. We have conducted additional experiments and made every effort to address your concerns. Whenever you find we have successfully addressed your worries, please reconsider your evaluation, thanks.

(3-1)

---

Q1: The quality of the generation output is primarily dependent on the pre-trained generation modules. If these modules are flawed or produce errors, the system cannot rectify these issues. For instance, if in image generation, stable diffusion struggles with accurately rendering certain elements (e.g., the number of human fingers), NExT-GPT would not be able to produce an accurate output, irrespective of its understanding of the instruction.

A: We would like to clarify three aspects.

Firstly, the core target or original intention of this work is to build a more unified general-purpose AI agent. That is, we do not intend to deliberately address the specific issues that individual SoTA modules (e.g., diffusion models) may face, which may quite be beyond our scope. That being said, we would like to emphasize that, to ensure obtaining an optimal system, we carefully selected all the backbone modules that are the current SoTA systems of the kind in the community, such as stable diffusion for image generation, zerocope for video generation, or AudioLDM for audio generation.

Secondly, to our best knowledge, the most commonly adopted approach for building big LLM systems is to integrate well-established strong-performing systems. Leveraging existing pre-trained generation modules is both an effective (lowest cost) and efficient (rapid construction) method in developing unified MM-LLMs capable of delivering content in any modality, avoiding training a system from scratch, especially when the system is very huge with a big cost.

Thirdly, the model architecture presented in this work facilitates easy and cost-effective extension to powerful incoming generation modules, potentially enabling more high-quality synthesis of image, video, or audio and beyond.

---

Q2: The evaluation strategy appears questionable. It seems that the NExT-GPT model used in the evaluation was fine-tuned on individual datasets, which may not accurately reflect the effectiveness of the proposed method.

A: Our system introduces a novel any-to-any task format, for which currently no dedicated benchmark is designed for evaluation. Under such a circumstance, when expecting to quantify the effectiveness of our model, there is no better way only can we conduct evaluations on existing benchmark datasets of the traditional individual cross-model or multimodal tasks (e.g., text-to-image, video-to-text), comparing with baselines. It is worth noticing that we strictly adhere to the experimental settings used in the baselines of each data to ensure a fair comparison, including data splitting and fine-tuning/zero-shot setups. While lacking a specific any-to-any multimodal benchmark (evaluation set), we are planning to further leverage our proposed MosIT dataset for this, e.g., splitting it into training and testing sets, to assess the performance of any-to-any models.

Our model achieved the best performance on the majority of tasks compared to baselines, including Audio-to-text generation, Image-to-text generation, Video-to-text generation, and Text-conditioned audio editing, and also secured performances on par with SoTA baselines in other rest tasks such as Text-to-image generation, Text-to-audio generation, and Text-conditioned image/video editing. Notably, even in a zero-shot scenario, particularly in the Text-to-video generation task, our model surpasses the SoTA baseline. It is very important to emphasize a factual point that, as our goal is to build a general-purpose MM-LLM, we run on the downstream in-domain datasets to exhibit that our system indeed can perform well on existing tasks. Thus we didn’t do much task-specific optimizations and fine-tuning of our system the way the baselines do on these in-domain datasets, which actually leaves very ample room for further huge improvement to those benchmarks. In what follows, we will carefully fine-tune our system on those datasets, and it is quite promising to see our system outperforming all the existing SoTA baselines. By the way, we also plan to expand our evaluation to more benchmarks encompassing a broader spectrum of tasks, further validating our system’s advance.

Dear Reviewer,

We would like to thank you again for your efforts and feedback. We have explained in detail about the concerns you raised before. We kindly hope that you can take some time to check our response and re-evaluate our paper based on our replies. If you have any further concerns or questions, please do not hesitate to let us know. We will be happy to address them promptly.

Best

Thank you for your valuable feedback, especially for the recognition of our strengths, such as 'important research', 'valuable datasets', and 'well-structured and easy to follow'. Below, we show you the point-to-point responses, which we hope can address your concerns.

---

Q1: The proposed alignment learning technique ... different modalities.

A: Thanks for your insights; it’s a good point. But here, we would like to clarify a few points.

Firstly, our proposed alignment learning techniques are featured as lightweight and cost-effective, facilitating rapid iteration of computational-centric model training.

Secondly, we have indeed considered the challenges introduced by the ”any-to-any modality” paradigm more or less, including the performance balance issue. For example, utilizing a unified encoder (i.e., ImageBind) to perceive the non-language modalities, instead of separate encodings, helps mitigate imbalances between models. Additionally, we have designed three independent output projection layers to align LLM with downstream generation systems, alleviating the interference between modalities and maintaining a balance.

Thirdly, in contrast to balancing the performance across different modalities, we emphasize that it can be much more prioritized to maximize the performance of individual existing encoder and decoder used in our system, achieving fully optimal performance of each modality.

Admittedly, there is ample room for improvement of more advanced alignment learning, and we appreciate your suggestions and we will explore further in the future.

---

Q2: Although introducing ... as expected?

A: Thanks. Here, first of all, we would like to clarify a misunderstanding that, our paper never (or intended to) claimed that introducing other modalities enhances the performance of the specific modality. Our focus always lies in constructing a unified agent, i.e., NExT-GPT, to mimic the complementary nature of different modalities in real-world scenarios. The proposed alignment tuning is designed to project different modalities into a shared semantic space, achieving such a unified agent where different modalities are complementary, rather than enhancing individual modality performance. We would like to note that, the performance mutual enhancement of multimodal systems is mostly observed and expected in tasks involving multimodal inputs that different modalities refer to shared semantic meaning, which however is not the primary focus of our system.

In addition, regarding the concern about leads to worse performance on benchmarking datasets, experimental results yet indicate that our system outperforms the baselines in tasks ranging from text-to-video to audio-to-text, image-to-text, and video-to-text generation. For the remaining evaluation tasks, our model still secured performances on par with the best-performing baselines, as shown in the tables. We kindly note that, most SoTA baseline systems are deliberately created for addressing that particular benchmarks and tasks which are well fine-tuned on the datasets for achieving a very high score, while our systems focus more on the overall (general-purpose) any-to-any capabilities. We want to claim a fact that, we didn’t do much task-specific optimizations and fine-tuning of our system as the baselines do, which actually leaves ample room for further huge improvement of benchmarks.

---

Q3: Will the pretrained model and dataset be released to the public?

A: Yes, all our pre-trained model (also will be consistently updated with new versions) and datasets will be released to the public to facilitate the follow-up research.

---

Q4: Ethics concerns about the content generated by the model or the dataset content

A: Thanks for bringing this attention to us. In addressing ethical considerations related to the content generated by Next-GPT, several key measures are implemented.

In the model aspect, Next-GPT is built upon existing well-established models, where we take full advantage of their inherent safety mechanisms to ensure the quality of generation content. Then, we emphasize user awareness by advising adherence to usage guidelines (e.g., interpreting the generated content with caution) and strictly limiting applications to academic research rather than commercial purposes.

In the data aspect, concerning data creation, we take meticulous measures to protect privacy, e.g., by removing or obfuscating personally identifiable information. We remain vigilant in mitigating bias during dataset collection, striving for representativeness to avoid favoring or disadvantaging any specific group or perspective. It is important to note that we possess the necessary licenses to collect data. And for data distribution to researchers and individuals, adherence to the stipulated license is mandatory.