Modality-Specialized Synergizers for Interleaved Vision-Language Generalists

We thank Reviewer ccUt for the constructive comments and valuable insights to improve this work. The responses to the comments are as follows:

W1: Comparison with more baselines

While we appreciate the reviewer for suggesting the additional baselines, we argue that we have tried our best to cover most of the mentioned baselines.

(1) We did not include MM-Interleaved in the baselines because the authors only released the pre-trained checkpoint without supervised fine-tuning or instruction tuning. The code and data for supervised fine-tuning reported in their paper are not publicly available. Thus, we cannot reproduce their method.

(2) OpenLEAF is a pipeline-based model where GPT-4 is cascaded with an image generation model (SDXL). It first applies GPT-4 to generate text and visual prompts and then uses SDXL to generate images using the visual prompts. This pipeline is very similar to the pipeline-based baselines (i.e., Gemini1.5+SDXL and GPT-4o+DALLE3) reported in our paper. Also, OpenLEAF is not publicly released and there are no implementation details to replicate the model.

(3) Since the Chameleon checkpoint released by Meta does not include the part of the weights required for image generation, Anole introduces an efficient training method to train those missing weights, thereby enabling Chameleon's multimodal generation capability. As mentioned in the paper (Lines 361–363), we utilize the model and checkpoints provided by Anole as the implementation of Chameleon. Consequently, the reported results for Chameleon in our work are actually derived from the Anole model. We will clarify this detail in the revised version of the paper.

W2: Illustrations of discrete-based VLGs in Figure 1

Thanks for pointing this out. For VLGs based on discrete tokens, we also observed the limitations illustrated in Figure 1, including inferior text and image quality and weak instruction-following capability. We are preparing the updated version of our paper by considering all the reviewers' comments including adding the example of VLGs based on discrete tokens. We will upload the new version of the paper soon.

Q1: Why fine-tuning unified models cannot capture modality-specific inductive biases

Several recent studies [1,2] have demonstrated that the plain transformer architecture lacks vision-specific inductive biases, even after extensive fine-tuning on large-scale datasets. This is because the architecture of multi-head self-attentions is more suitable for modeling long-range global dependency and less effective at modeling local priors due to its weak inductive bias [4,5]. Through the lens of Fourier analysis [1,4,5], i.e., analyzing the amplitude of Fourier-transformed image features generated by either purely transformer-based vision models or convolution-based vision models, previous studies show that transformer-based models reduce the high-frequency signals in images, acting as a low-pass filter, and conversely, convolution-based encoders amplify high-frequency components, acting as a high-pass filter. Thus, relying solely on multi-head self-attentions can cause the VLGs to miss important local visual information, and two structures can be combined to capture richer information from images. In our work, we harmonize two structures by integrating convolutional LoRA into the multi-head self-attention layers for effectively modeling both global and local dependency of image features in image generation.

Q2: Whether discrete tokens or continuous embeddings is better

Our work primarily focuses on enhancing pre-trained multimodal models with specialized architectures for parameter-efficient interleaved visual instruction tuning. We demonstrate that MOSS is broadly applicable to both continuous and discrete image tokens. While an in-depth discussion on the merits of discrete tokens versus continuous embeddings is beyond the scope of our work, we can provide some insights from a recent study [3], which found that continuous tokens generally outperform discrete tokens in terms of performance.

Reference

[1] Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model. Zhong et al., ICLR 2024.

[2] Vision Transformer Adapter for Dense Predictions. Chen et al., ICLR 2023.

[3] Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens. Fan et al., 2024.

[4] Inception Transformer. Si et al., NeurIPS 2022.

[5] How Do Vision Transformers Work? Park et al., ICLR 2022.

We thank the reviewer for their constructive comments and valuable insights to improve this work. The responses to your comments are as follows:

W1: Performance drop in text quality on Chameleon-MoSS

The reason why adding MoSS in Chameleon can cause a slight performance drop in text quality is that the original Chameleon usually generates long and verbose text responses but with no image output. On the contrary, as the text responses in our LeafInstruct dataset are more concise to allow for including more images, after interleaved instruction tuning, our model learns to generate more concise text responses. Specifically, the average generated word length of the original Chameleon is 653, whereas that of Chameleon-MoSS is 166. The verbose responses from the original Chameleon are preferred by the LLM judge due to their verbosity bias [1,2], leading to a slight drop in text quality of Chameleon-MoSS in LLM-based evaluation.

To better support this analysis, we further conduct a human evaluation of the text quality of the two models by randomly sampling 100 instances from InterleavedBench. We ask a human annotator to select the preferred text responses given the system outputs from two models. We report the Win-Tie-Loss results in Table A. Win means our Chameleon-MoSS is better than the original Chameleon, Tie means the quality of two responses is equally good, and Loss means the original Chameleon is better.

Table A: Human evaluation results on text quality of the original Chameleon and our Chameleon-MoSS. "Win" indicates our Chameleon-MoSS's responses are preferred by humans.

From the results, the text quality of our Chameleon-MoSS is actually better than the original Chameleon. One issue we frequently observed in the original Chameleon is the text responses are overly verbose and sometimes even severely repetitive. In our evaluation protocol of text quality adopted from InterleavedEval [3], such verbosity and repetitiveness are not penalized, making the automatic evaluation results heavily biased towards the longer responses from the original Chameleon. In addition, we also provide a qualitative comparison between the original Chameleon and Chameleon-MOSS, which also shows that applying MoSS will not hurt text quality and can alleviate the repetition issue.

Example:

Instruction: Describe the new features of Trek's expanded Slash lineup, highlighting the availability of carbon and aluminum models and the introduction of a more affordable top-of-the-line option. Input Context: Trek expands its award-winning Slash lineup in Waterloo, WI.

Original Chameleon Response: The new features include the introduction of the Alpha Platinum Aluminum model, a lower-priced carbon version, and upgrades across all models. The current price range for the new models is $5,000 to $20,000. The features of the new full-carbon Slash 9.9 mountain bike include carbon wheels and Rapid Drive hub. The price range for the new full-carbon Slash 9.9 mountain bike is $20,000 to $25,000. The price range for the new aluminum Slash 9.9 mountain bike is $10,000 to $15,000……

Chameleon-MoSS Response (ours): There are several options in Trek's Slash lineup. For example, you can now purchase the new top-of-the-line Slash 9.9 Alpha Platinum Aluminum, which comes equipped with top-notch components such as an SRAM X01 Eagle drivetrain and a Race Face Turbine handlebar, making it an affordable yet high-performance option for those looking to save money without sacrificing quality.

Reference

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Zheng et al., NeurIPS 2023 Datasets and Benchmarks Track.

[2] Verbosity Bias in Preference Labeling by Large Language Models. Saito et al., 2023.

[3] Holistic Evaluation for Interleaved Text-and-Image Generation. Liu et al., EMNLP 2024.

W2: Attribution of performance improvement

We have an isolated experiment in Table 2 to show a direct comparison between our MoSS and other PEFT methods (i.e., LoRA and MoE-LoRA). All the PEFT methods are trained on our LeafInstruct dataset, and the only difference among the models is the PERT architecture. From Table 2, using MoSS with the same training set achieves significant improvement, which shows the effectiveness of our proposed MoSS framework.

We clarify that the results for the four middle rows in Table 1 reflect the model's original performance without further training on LeafInstruct. We also have the results for full-parameter fine-tuning using LeafInstruct in Table B:

Table B: Comparison between full finetuning and parameter-efficient tuning with MoSS.

The first row represents the results of fully fine-tuning EMU2 on our proposed LeafInstruct dataset. The second row shows the results of fine-tuning EMU2 using our proposed MoSS framework. As observed, while full fine-tuning allows EMU2 to achieve better performance on text generation, the model demonstrates inferior performance on image generation due to its lack of inductive bias. In contrast, tuning with MoSS, which incorporates ConvLoRA, significantly improves image generation performance, even though the number of trained parameters in full fine-tuning is substantially larger than that of MoSS. These results clearly highlight the advantages of integrating ConvLoRA into the transformer architecture for processing visual information.

W3: Computation cost of ConvLoRA

We did a comparison of the computational cost between using linear LoRA and our ConvLoRA, respectively. We compute the inference time for generating 1,000 images for each model. The total inference times for linear LoRA and ConvLoRA are 4,380 seconds and 5,910 seconds, respectively. The difference between the two models is around 1.5 seconds per image, indicating the computational cost increased by ConvLoRA is not significant.

Dear Reviewer YfET,

We sincerely appreciate your thoughtful evaluation and your explicit intention to raise the score to 7, acknowledging our responses and additional experiments have effectively addressed your previous concerns.

Given that the system only allows scores of 6 or 8, we would like to respectfully note that the score of 8 (“Accept, good paper”) in ICLR’s scale closely aligns with a score of 7 in other top-tier conferences like NeurIPS, which defines 7 as “Accept: Technically solid paper with high impact on at least one sub-area.” We bring this up in hopes to make sure your score is aligned with your intention and this perspective might be helpful as you finalize your assessment within the system’s constraints.

Thank you again for your careful consideration and detailed feedback throughout this process.

Best regards,

Authors

We thank Reviewer pC3r for the constructive comments and valuable insights to improve this work. The responses to the comments are as follows:

W1: Novelty of two LoRA types.

We would like to clarify and justify the key novelty and contributions of this work as follows: Our work is the first to integrate convolution LoRA into autoregressive generative models backed by large language models (LLMs). This requires new design strategies to reconcile the discrepancy between the spatial operation in convolution and the sequential autoregressive generation process of language models. For example, during inference, we introduce an on-the-fly partial convolutional mechanism tailored for autoregressive generation (as shown in the right part of Figure 2), where the convolution kernel operated on each image patch only covers its neighboring patches on the left and top, as the patches on the right and bottom have not been generated yet. In addition, while previous works such as mixture-of-LoRA [3,4] propose to leverage separate linear LoRAs with the same architecture for different tasks or different modalities, we are the first to integrate two distinct LoRA architectures within a single model for processing images and text, respectively. We will add more detailed explanations and discussion regarding the contributions in the revised version.

W2: The necessity of convolutional LoRA.

Several recent studies [1,2] have demonstrated that a plain ViT-based image encoder lacks vision-specific inductive bias for dense predictions. Specifically, the quantitative analysis of [5,6] proves that multi-head self-attentions are more capable of modeling global shapes and structures of a scene in an image, while convolution layers are more capable of capturing local information such as edges and textures due to their strong inductive bias such as the local spatial operation. Through the lens of Fourier analysis [1,5,6], i.e., analyzing the amplitude of Fourier-transformed image features generated by either purely transformer-based vision models or convolution-based vision models, previous studies show that transformer-based models reduce the high-frequency signals in images, acting as a low-pass filter, and conversely, convolution-based models amplify high-frequency components, acting as a high-pass filter. Thus, relying solely on multi-head self-attentions can cause the VLGs to miss important local visual information, and two structures can be combined to capture richer information from images. In our work, we harmonize two structures by integrating convolutional LoRA into the multi-head self-attention layers for effectively modeling both global and local dependency of image features in image generation. In Table 2, we also empirically proved that applying convolutional LoRA significantly outperforms the baselines using a mixture of linear LoRA, which further justified the necessity of convolutional LoRA.

W3: Clarification for “same framework for both text and image”.

We would like to clarify that while VLGs typically use a separate image encoder to convert images into continuous vectors or discrete image tokens, by “same framework” we mean existing methods using the identical architecture within the LLM component of VLGs for processing both images and text. Given that the LLM is responsible for reasoning over multimodal inputs and generating multimodal outputs, we argue that it should have dedicated parameters to mitigate modality conflicts. The effectiveness and necessity of using separate sets of LoRA within LLMs have been demonstrated in prior studies [3,4] in the multimodal domain.

W4: Applying MoSS to more powerful VLGs.

Emu2 and Chameleon were two state-of-the-art VLGs at the time of our paper submission. More advanced models, such as Emu3, were published after the abstract deadline and are therefore concurrent with our work. Given that Emu3 employs transformer layers as its building blocks and uses discrete tokens to represent images, which are similar to Chameleon, we argue that our method can be similarly adapted to Emu3. Given the limited time and high demand for computational resources, we will not be able to get the results during the rebuttal period but will include the results in the final version.

Reference

[1] Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model. Zhong et al., ICLR 2024.

[2] Vision Transformer Adapter for Dense Predictions. Chen et al., ICLR 2023.

[3] Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning. Gou et al., 2023.

[4] Multimodal Instruction Tuning with Conditional Mixture of LoRA. Shen et al., ACL 2024.

[5] Inception Transformer. Si et al., NeurIPS 2022.

[6] How Do Vision Transformers Work? Park et al., ICLR 2022.

Dear Reviewer pC3r,

We sincerely appreciate the time and effort you've devoted to reviewing our work. We understand that your schedule may be quite busy, and we are truly grateful for your valuable feedback. As the Author-Reviewer discussion phase is ending soon, we would greatly value the opportunity to engage in further discussion with you. Our aim is to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss.

We look forward to the opportunity for further discussion with you. Thank you for your thoughtful consideration.

Best regards,
Authors

We would like to extend our sincerest gratitude for the time and effort you have made in reviewing and providing valuable feedback that is crucial for improving our work.

We greatly value each comment and suggestion from your review. As there is still time remaining before the conclusion of the discussion phase, we would be truly grateful if you could let us know whether there remain any unresolved concerns in your view. In our continuous effort to enhance the quality and impact of our research, we are glad to address any remaining issues during the discussion phase. Thank you a lot!

We thank the reviewer for their constructive comments and valuable insights to improve this work. The responses to your comments are as follows:

W1: Experimental comparison with two ConvLoRA. To show the benefits of our modified ConvLoRA architecture compared to the ConvLoRA proposed in [3] denoted as SAM-ConvLoRA, we replace the ConvLoRA in MoSS with SAM-ConvLoRA. Specifically, we set the rank of project-down and project-up matrices in SAM-ConvLoRA to 256 which is the same number of ranks in our proposed MoSS-ConvLoRA, and adopt the multi-scale convolution kernels to the size of 2x2 and 4x4.

Table A: Comparison of two types of ConvLoRA.

As shown in Table A, Our MoSS-ConvLoRA consistently outperforms the previous SAM-ConvLoRA on all other evaluation aspects, which demonstrates the superiority of our proposed ConvLoRA architecture. Particularly, our MoSS-ConvLoRA achieves notably better visual qualities, including perceptual quality and image coherence, thanks to our novel design that our convolution operation is applied to the full-rank original image features instead of low-rank image features as in SAM-ConvLoRA.

W2: Adding more evaluation benchmarks.

We would like to first clarify that the main focus of our work is interleaved generation. The main evaluation benchmark we used, i.e., InterleavedBench, has already covered a broad array of widely adopted interleaved generation tasks collected from existing well-established academic datasets, including multimodal script generation from WikiHow (Yang et al., 2021), visual storytelling from VIST (Huang et al., 2016), multi-concept image composition from CustomDiffusion (Kumari et al., 2023), activity generation from ActivityNet (Krishna et al., 2017), image editing from MagicBrush (Zhang et al., 2023a), and so on. Given that InterleavedBench is currently the most comprehensive and well-established benchmark for interleaved generation, we believe our evaluation coverage is already comprehensive and sufficient.

Nevertheless, we sincerely appreciate the reviewer’s suggestion to include more benchmarks to measure the performance from different perspectives. We are currently working on these additional experiments and will report the results as soon as possible.

W3: More details on the proposed dataset.

Due to the space constraint, we elaborated on the dataset construction details in Appendix B, where we present the detailed process of how we construct LeafInstruct from existing data resources.

In Line 299, “We include the details on dataset construction in Table 3 in Appendix B.”, the reference “in Table 3” is a typo, and the corrected version should be “We include the details on dataset construction in Appendix B.” We thank the reviewer for spotting this typo for us and we will correct this in the revision.

W4: The writing and organization of the paper.

We thank the reviewer for the suggestion. We will correct the specified reference errors in the revised paper.

W5: Limited improvement of MoSS.

First, we would like to justify that our approach can significantly improve the original VLG by a large margin (e.g., 97.76% on the average of 5 aspects), which effectively demonstrates the effectiveness of our approach. It is also worth noting that among the 5 evaluation aspects, helpfulness is the most important aspect as it holistically measures if the generated content is useful for the task. From Table 1, our Emu2+MoSS model significantly outperformed all the open-sourced VLG baselines on helpfulness, which also shows that our framework can effectively improve the model’s overall capabilities and instruction-following ability.

Second, the reason why adding MoSS in Chameleon can cause a slight performance drop in text quality is that the original Chameleon usually generates long and verbose text responses but with no image output. On the contrary, as the text responses in our LeafInstruct dataset are more concise to allow for including more images, after interleaved instruction tuning, our model learns to generate more concise text responses. Specifically, the average generated word length of the original Chameleon is 653, whereas that of Chameleon-MoSS is 166. The verbose responses from the original Chameleon are preferred by the LLM judge due to their verbosity bias [1,2], leading to a slight drop in the text quality of Chameleon-MoSS in LLM-based evaluation.

To better support this analysis, we further conduct a human evaluation of the text quality of the two models by randomly sampling 100 instances from InterleavedBench. We ask a human annotator to select the preferred text responses given the system outputs from two models. We report the Win-Tie-Loss results in Table B. Win means our Chameleon-MoSS is better than the original Chameleon, Tie means the quality of two responses is equally good, and Loss means the original Chameleon is better.

Table B: Human evaluation results on text quality of the original Chameleon and our Chameleon-MoSS. "Win" indicates our Chameleon-MoSS's responses are preferred by humans.

From Table B, the text quality of our Chameleon-MoSS is actually better than the original Chameleon. One issue we frequently observed in the original Chameleon is the text responses are overly verbose and sometimes even severely repetitive. In our evaluation protocol of text quality adopted from InterleavedEval [3], such verbosity and repetitiveness are not penalized, making the automatic evaluation results heavily biased towards the longer responses from the original Chameleon. In addition, we also provide a qualitative comparison between the original Chameleon and Chameleon-MOSS, which also shows that applying MoSS will not hurt text quality and can alleviate the repetition issue.

Example:

Instruction: Describe the new features of Trek's expanded Slash lineup, highlighting the availability of carbon and aluminum models and the introduction of a more affordable top-of-the-line option. Input Context: Trek expands its award-winning Slash lineup in Waterloo, WI.

Original Chameleon Response: The new features include the introduction of the Alpha Platinum Aluminum model, a lower-priced carbon version, and upgrades across all models. The current price range for the new models is $5,000 to $20,000. The features of the new full-carbon Slash 9.9 mountain bike include carbon wheels and Rapid Drive hub. The price range for the new full-carbon Slash 9.9 mountain bike is $20,000 to $25,000. The price range for the new aluminum Slash 9.9 mountain bike is $10,000 to $15,000……

Chameleon-MoSS Response (ours): There are several options in Trek's Slash lineup. For example, you can now purchase the new top-of-the-line Slash 9.9 Alpha Platinum Aluminum, which comes equipped with top-notch components such as an SRAM X01 Eagle drivetrain and a Race Face Turbine handlebar, making it an affordable yet high-performance option for those looking to save money without sacrificing quality.

W6: Qualitative results of Chameleon-MoSS.

Thanks for pointing this out. We are preparing the updated version of our paper by considering all the reviewers' comments including adding the qualitative results of Chameleon-MoSS. We will upload the new version of the paper soon.

Thank you for your response. We added the results of Chameleon on the additional evaluation benchmarks in Table E. From the results, our MoSS (based on Emu2) outperforms the original Chameleon baseline on 5 out of 7 benchmarks by a significant margin, demonstrating the strong capabilities and generalizability of our approach.

Table E: Comparison between Chameleon and our MoSS on additional evaluation benchmarks.

For GILL, we found it very difficult to generalize to many of your requested benchmarks, as GILL is highly specialized in text-and-image generation tasks without considering much multimodal understanding capabilities. For example, GILL is solely trained on Conceptual Captions (CC3M) to generate interleaved images and captions, but it lacks training on multimodal comprehension or instruction following data. In addition, Chameleon (released in May 2024) is a more recent and advanced model compared with GILL (released in May 2023). Therefore, we believe comparing Chameleon with our model is sufficient to demonstrate the effectiveness and superiority of our approach. Given the time costs of these additional experiments are high and the remaining time of the discussion period is limited, we will include more baselines on these additional benchmarks for reference in the final version.

Please let us know if this fully addresses your question and if you still have any remaining concerns. Thank you again for your time and engagement in the discussion.

W2: Adding more evaluation benchmarks. (continued)

We have finished all the additional experiments requested by the reviewer. The detailed implementation details and results are as follows. To show that our MoSS framework can also excel on tasks requiring single modality outputs i.e., the output only contains text or an image, we evaluate its performance on widely adopted image understanding benchmarks including MMBench, MME, MMMU, Pope, and MM-Vet, and text-to-image generation benchmarks including MSCOCO 30K [7], and GenEval [9].

Since LeafInstruct mainly targets tasks with interleaved outputs, we augmented it with 500,000 instances from Vision-Flan [5], a popular visual-instruction tuning dataset targeting image understanding, and 500,000 instances from LAION-COCO [6], a standard training dataset for text-to-image generation. We finetune Emu2 with LoRA, MoE-LoRA, and MoSS on the mixed dataset. We report their performance of multimodal understanding tasks in Table C, and text-to-image generation tasks in Table D.

Table C: Results on widely adopted multimodal understanding benchmarks.

From Table C, our MoSS outperforms previous LoRA and MoE-LoRA on most of the multimodal understanding benchmarks by a notable margin, which demonstrates that MoSS can be well generalized to diverse multimodal comprehension tasks. Note that for all multimodal understanding tasks, we use their official implementation for evaluation.

Table D: Results on widely adopted text-to-image generation benchmarks. Note that the FID metric is the lower the better.

For MSCOCO-30K, following the previous evaluation protocol [8], we randomly sample 30,000 captions from the validation set of MSCOCO and generate 30,000 images. We report the FID between the 30,000 generated images and real images from the validation set of MSCOCO (Note for FID, the lower the better). For GenEval, we adopt their official implementation of the evaluation. From Table D, our MoSS achieves better performance on both benchmarks, showing the effectiveness and generalizability of our approach. Notably, our MoSS achieves significantly better FID on MSCOCO-30K, which validates that our ConvLoRA can effectively improve the quality of generated images.

Please let us know if our responses and additional experiments have addressed your concerns and if you still have remaining questions. We are looking forward to your reply and sincerely hope you can reconsider your evaluation. Thank you again for your time and feedback in improving this work.

Reference

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Zheng et al., NeurIPS 2023 Datasets and Benchmarks Track.

[2] Verbosity Bias in Preference Labeling by Large Language Models. Saito et al., 2023.

[3] Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model. Zhong et al., 2024

[4] Holistic Evaluation for Interleaved Text-and-Image Generation. Liu et al., EMNLP 2024.

[5] Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning. Xu et al., ACL 2024

[6] https://huggingface.co/datasets/laion/laion-coco

[7] Microsoft COCO: Common Objects in Context. Lin., et al.

[8] Generative multimodal models are in-context learners. Sun., et al., CVPR 2024

[9] GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. Ghosh et al., CVPR 2023.