Making Multimodal Generation Easier: When Diffusion Models Meet LLMS

Thanks for your comments!

1. The proposed EasyGen simply marries the LLM with the UnifDiffuser. Meanwhile, in order to perform the text generation and image generation tasks, LLM should be plugged into different positions and retrained, which is redundant and loses the conciseness of large models.

In order to make LLMs have the ability to understand image, current MLLMs such as LLaVA and MiniGPT-4 usually leverage instruction tuning method to fine-tune large models. This method can enhance the capability of multimodal interaction while maintaining the LLM capability. So our primary task is to employ an instruction tuning method to enable the model to generate image captions. Suitable instruction datasets could include PhotoChat, TextBind, and others.

2. It seems that the claimed “data efficient training” property of EasyGen comes from its UniDiffuser component, which can be regarded as a well-established multimodal interactor. Therefore, it is unfair to compare the data efficiency of EasyGen with other MLLM like BLIP, MiniGPT-4 and LLaVA (Table 4), which all start training from raw LLM and vision backbone.

We acknowledge the comparision is unfair. So we do the following experiments. To examine the efficiency of BiDiffuser, we combine its output with the image encoded by image CLIP ViT-L/14 and fine-tune the parameters of the LLM and projection layers. Also, w/o BiDiffuser means we directly align LLM with CLIP and using the same dataset from EasyGen. We fine-tune EasyGen using training and validation splits from VQAv2, Text Captions, AOK-VQA, and TextVQA datasets. The table below showcases the results of our method compared to leading models. To validate the effectiveness of the BiDiffuser module, we also incorporate it into LLaVA Vicuna-7B and use the same mixed datasets for instruction tuning. Given that BiDiffuser can convert images into text vectors, it can be directly transferred from EasyGen Vicuna-7B to the LLaVA Vicuna-7B model. In EasyGen, the model's performance significantly improves when using the same alignment and instruction tuning data with the BiDiffuser module.

3. A main weakness of existing diffusion models lies in that they cannot effectively comprehend long textual context due to its weak text encoder. Therefore, it is prospective to combine the LLM and diffusion models for stronger generation ability. Does the EasyGen possess such the ability?

Since EasyGen is solely trained to produce image captions that can be comprehended by the diffusion model, we do not modify the diffusion model's text encoder. Integrating LLM as a text encoder for the diffusion model can enhance the alignment of the two models. However, this necessitates retraining the diffusion model, which is a resource-intensive process.

Thanks for your comments!

1. For multimodal dialogue, when considering long context conditional image generation or image conditional (multi-images or single-image) image generation like image edition/translation (it is, however, what UniDiffuser initially can do), only generating text inputs as diffusion modal conditions is limited. For the evaluations, the current results only include traditional short QA or image captioning tasks. However, one of the advantages of LLMs is that they are mighty at world knowledge and long-context reasoning. It would be better if results on modern benchmarks like MM-Vet and MMBench were provided. Also, the most commonly used VQAv2 results are not provided.

We acknowledge that converting images to text may result in loss of information, hence we only apply EasyGen for image-captioning and zero-shot VQA tasks. To fully utilize the image's information while considering the significant cost of fine-tuning the diffusion model for the VQA task, we choose to combine the output of BiDiffuser with the image encoded by image CLIP ViT-L/14. We then fine-tune the parameters of the LLM and projection layers. EasyGen is fine-tuned using training and validation splits from VQAv2, Text Captions, AOK-VQA, and TextVQA datasets. The table below illustrates the results of our method compared to leading models. To confirm the effectiveness of the BiDiffuser module, we also incorporate it into LLaVA Vicuna-7B and use the same mixed datasets for instruction tuning. Given that BiDiffuser can convert images into text vectors, it can be directly transferred from EasyGen Vicuna-7B to the LLaVA Vicuna-7B model. The results indicate that BiDiffuser aids the model in understanding images.

2. What about COCO captioning results on test split?

In our study, we utilize the COCO Karpathy test split as the COCO captioning test split.

3. It is somewhat overclaimed to criticize the large data requirement of CLIP/ImageBind models since UniDiffuser is also trained on large-scale corpora using large amounts of resources that cannot be affordable by most labs.

We acknowledge the comparision is unfair. So we do the experiments. However, when multimodal language models use them as a visual module, CLIP/ImageBind still requires large-scale corpora to achieve alignment. In our experiment, the volume of data needed for fine-tuning and aligning the UniDiffuser process is significantly reduced.

4. Missing citations and discussions. The method of first converting images to texts and then feeding them to LLMs is the same as Img2LLM, but it is not cited or discussed. Besides, several closely related concurrent works should be discussed.

Thanks for your advice, we will add the missing citations and discussions to our related works in the revision

Thanks for your comments!

1. The claim of data efficiency, a major highlighted merit, appears somewhat tenuous, given the possibility of the BiDiffuser being a fine-tuned iteration of a pre-trained UniDiffuser.

We recognize that the comparison might not be completely fair. So we do the following experiments. To evaluate the effectiveness of BiDiffuser, we incorporate its output with the image encoded by image CLIP ViT-L/14 and fine-tune the parameters of the LLM and projection layers. We conduct fine-tuning on the training and validation splits of VQAv2, Text Captions, AOK-VQA, and TextVQA datasets. The table presented here showcases the results of our approach compared to state-of-the-art models. Additionally, we integrate the BiDiffuser module into LLaVA Vicuna-7B and perform instruction tuning using the same mixed datasets. Since BiDiffuser can map images to text vectors, the BiDiffuser in EasyGen Vicuna-7B can be directly transferred to the LLaVA Vicuna-7B model. In EasyGen, with the same alignment and instruction tuning data, the model's performance significantly improves when the BiDiffuser module is incorporated.

2. The paper could further benefit from more details of instruction-tuning. Incorporating concrete examples instead of mere templates in Section 3.2.2 would be much helpful for comprehension. For a more streamlined reader experience, it would be advantageous to highlight the foremost results in Tables 4 and 6.

Thanks for you advice! We will add concrete examples Section 3.2.2 and highlight the foremost results in Tables 4 and 6. Also revising "RELATED WORK" to "RELATED WORKS" .

Thanks for your comments!

1. Each modality employs the same forward process to corrupt its representations, following the same schedule. The distinction lies in the addition of 4X64X64 Gaussian noise to the image (encoded by AutoencoderKL) and 77X768 Gaussian noise to the text (encoded by CLIP text encoder). As UniDiffuser can integrate different modalities, we can use the text representation as input to the LLM to aid in image comprehension.

2. BiDiffuser utilizes an iterative denoising method, specifically employing DPM-Solver for sampling with a step number of 50. In our experiments, we tested our model's performance on an A100(80G). We used 1000 image-caption pairs for inference. On average, EasyGen took 2.95 seconds for the caption generation task (with the diffusion module taking about 2.41 seconds), and approximately 4.96 seconds to generate an image.

3. In our tasks, we updated all the parameters of the Language Models (LMs). We also attempted efficient instruction-tuning using LoRA:

4. We are very sorry, as our experiments were conducted in a lab setting, we only studied our method on 7B LLMs.

5. Custom data. In line with current works such as LLaVA and InstructBLIP, we positioned <query> after <image>.

6. We believe the primary reason for the inferior performance of EasyGeN with Vicuna-7B on OK-VQA compared to LLaVA is the instruction tuning data. As demonstrated in the table, to verify the effectiveness of the BiDiffuser module, we also added this module to LLaVA Vicuna-7B and used the same mixed datasets for instruction tuning. Since BiDiffuser can convert images into text vectors, it can be directly transferred from EasyGen Vicuna-7B to the LLaVA Vicuna-7B model. The results indicate that BiDiffuser aids the model in understanding images. Both LLaVA and our model use the same instruction tuning datasets. It's worth noting that LLaVA has used 595K pretraining data from the CC-3M dataset to align CLIP and LLM. Our model does not pre-align CLIP and LLM, and only uses instruction-tuning data for training.

7. In our alignment and VQA fine-tuning stages, this table provides a description of the datasets we utilized. It is important to mention that during the alignment process, we employed 5,000 images from the VQAv2 dataset and an additional 5,000 images from the VisDial dataset.

8. Since Table 7 displays the results of the ablation study and OK-VQA has a smaller size compared to the GQA test split, we only evaluate these models on the OK-VQA test split.

9. To compare our models with other methods in the benchmarks, we included MMbench. The results can be found in the table above.

10. The bidirectional conditional diffusion model can narrow the gap between language models (LMs) by utilizing text embeddings to represent images. While diffusion model is not necessary for reducing the gap, BLIP2, for example, utilizes Q-Former to enable learnable inputs to capture image information. However, the bidirectional conditional diffusion model can also learn to generate images.

11. Unlike CLIP, our model leverage BiDiffuser's text embedding as image's representation and our projection layer aligns the text features of BiDiffuser with LMs, making it easy to train the projection layer and achieve alignment. Just using the embedding of image usually need a large amount of image-text pairs in the alignment stage. Additionally, we took into consideration that converting images to texts may result in missing information. Therefore, we chose to concatenate the output of BiDiffuser with the image encoded by image CLIP ViT-L/14 and fine-tune the parameters of the LLM and projection layers. The experiments' results can be found in Q3's table.