ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

Thanks for your time reviewing and providing the helpful suggestions for our paper. We address open questions and remarks below:

Clarify our contributions

We appreciate your recognition of our work as an exceptional engineering project with superior performance and good reproducibility. In the last paragraph of the introduction section, we outline the contributions of our work, including: 1) a highly extensible model with scalable-friendly architecture and modality-agnostic tasks that have the potential to expand to unlimited modalities; 2) the first representation model to achieve new SOTAs across a wide range of uni-modal and cross-modal tasks; 3) strong emergent retrieval capabilities for aligning unpaired modalities, eliminating the need for paired data collection across different modalities.

Here we elaborate on the first point to emphasize the technical contributions of ONE-PEACE:

1. Scalable-friendly architecture. ONE-PEACE processes different modalities through shared attention layers and separate Adapters/FFNs. When introducing a new modality, it only requires the addition of new adapters and FFNs. While previous works like VLMO [1] and BEIT-3 [2] have also explored this design, they primarily concentrate on enhancing model performance rather than highlighting the utilization for modality expansion.

2. Modality-agnostic tasks. We combine two modality-agnostic pretraining objectives: cross-modal contrast and intra-modal denoising contrast, enabling the model to achieve outstanding zero-shot retrieval performance and fine-tuning performance simultaneously. We also analyzed the impact of different denoising losses in Table 11. We found that denoising contrastive loss is more compatible with cross-modal contrastive loss than other losses, it can further improve the model's performance. These experiments have not been explored well in previous works.

3. With the help of scalable-friendly architecture and modality-agnostic tasks, we design a two-stage training method. It allows us to add new modalities (audio) into ONE-PEACE without affecting its performance on existing modalities (vision and language), and even enables the achievement of SOTA results in tasks related to the new modalities. This modality extension method is a significant contribution of our paper, and to the best of our knowledge, it has not been explored in previous works.

Experiment on more heterogeneous modalities

The design of our architecture and tasks can effectively handle heterogeneous modalities with a lack of paired data.

1. Modality adapters are solely responsible for transforming the raw input into vectors, allowing us the flexibility to choose its architecture, such as CNN, RNN, and MLP, guided by previous works (e.g., HighMMT, ImageBind [3]). Therefore, this design is very friendly to heterogeneous modalities. In addition, the training curves in Figure 4 indicate that the model architecture of ONE-PEACE helps accelerate the convergence speed of the model and achieve better results.

2. Denoising contrastive loss is a modality-agnostic task that can be used on any modality, and it does not rely on paired data. The results in Table 10 show that combining denoising contrast and cross-modal contrast pre-training tasks can achieve better modality alignment effects, making this design suitable for modalities that lack paired data.

We agree that experiments on more heterogeneous modalities can further demonstrate the potential of ONE-PEACE, but due to constraints during the response period, we have included a discussion in the Appendix. Additionally, we have also discussed related works, such as HighMMT, in this section.

[1] Bao H, Wang W, Dong L, et al. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts[J]. Advances in Neural Information Processing Systems, 2022, 35: 32897-32912. 
[2] Wang W, Bao H, Dong L, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks[J]. arXiv preprint arXiv:2208.10442, 2022.
[3] Girdhar R, El-Nouby A, Liu Z, et al. Imagebind: One embedding space to bind them all[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15180-15190.

Thanks for pointing this out. We have re-collected the AVQA dataset, conducted experiments, and updated the results in Table 6. As shown in Table 6, ONE-PEACE also achieves excellent results in this task. Thus, we have validated the ability of ONE-PEACE in all combinations of modalities, including vision-language (visual grounding, image-text retrieval), vision-audio (video-audio classification), audio-language (audio-text retrieval, audio question answering), and vision-audio-language (AVQA).

[1] Fan C, Zhang X, Zhang S, et al. Heterogeneous memory enhanced multimodal attention model for video question answering[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 1999-2007.
[2] Li X, Song J, Gao L, et al. Beyond rnns: Positional self-attention with co-attention for video question answering[C]//Proceedings of the AAAI conference on artificial intelligence. 2019, 33(01): 8658-8665.
[3] Li X, Gao L, Wang X, et al. Learnable aggregating net with diversity learning for video question answering[C]//Proceedings of the 27th ACM international conference on multimedia. 2019: 1166-1174.
[4] Zhang J, Shao J, Cao R, et al. Action-centric relation transformer network for video question answering[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 32(1): 63-74.
[5] Jiang P, Han Y. Reasoning with heterogeneous graph alignment for video question answering[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 11109-11116.
[6] Le T M, Le V, Venkatesh S, et al. Hierarchical conditional relation networks for video question answering[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 9972-9981.
[7] Li G, Hou W, Hu D. Progressive Spatio-temporal Perception for Audio-Visual Question Answering[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 7808-7816.
[8] Yang P, Wang X, Duan X, et al. Avqa: A dataset for audio-visual question answering on videos[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 3480-3491.

Q1: About the two-stage pre-training

We designed the two-stage pre-training intentionally to demonstrate that ONE-PEACE has the potential to expand to unlimited modalities. In the first pre-training stage (VL pre-training), vision and language representation are aligned from paired VL data. In the second pre-training stage (AL pre-training), we want the new modality audio to align with language and not influence the VL-aligned feature so we freeze the text-branch parameters. This approach proves that ONE-PEACE can introduce new modalities without influencing the performance of the existing modalities, and even achieve excellent performance in the new modality. If we directly conduct pre-training for the visual, textual, and audio modalities simultaneously, we would not be able to demonstrate the scalability of ONE-PEACE to new modalities.

Q2: About using ONE-PEACE

We appreciate your experience with ONE-PEACE and the feedback you provided. The following points should be considered when using ONE-PEACE:

• ONE-PEACE has a maximum audio duration limit of 15 seconds. Any audio longer than this will be cut off. Therefore, you should ensure that the input audio is not too lengthy and that its crucial information is within the first 15 seconds.
• The audio training data for ONE-PEACE solely consists of environmental sound datasets (such as bird chirping, alarm clock ringing, etc.), the accuracy of the other types of audio may be somewhat compromised.
• If you use our demo, please note that the demo only uses 60,000 images as the candidate set (ImageNet val set + MSCOCO val set), so there may be some audio clips without corresponding images in the candidate set.

Thanks for your time reviewing and providing helpful suggestions for our paper. First of all, we would like to clarify that, to the best of our knowledge, ONE-PEACE is still the SOTA in a lot of datasets, such as ADE20K, RefCOCO+, VGGSound, FSD50K, etc.

We address open questions and remarks below:

W1: Add more results to backup the visual-acoustic feature alignment quality

Some of the experiments done in the paper, as well as our newly added experiments, can address your concerns:

1. We conducted experiments on the VGGsound dataset, which requires the model to predict the category of samples based on the video and audio information. The results are listed in Table 6. It can be seen that ONE-PEACE achieved SOTA results on this task, even outperforming previous video pretraining models.

2. Following the suggestion of Reviewer DHvY30, we conducted experiments on the AVQA dataset. This dataset consists of video, audio, and text modalities, and it requires the model to answer questions based on the video and audio information. We have updated the results in Table 6. ONE-PEACE still achieves leading results on the dataset.

In addition to the above experiments, we also provide the multi-modal retrieval cases in Figure 3 and Figures 7-9 to demonstrate the visual-acoustic feature alignment quality of ONE-PEACE.

W2: Lacks insights on the comparison against other LMMs

In fact, the baselines compared in the paper cover various model architectures (encoder-only, dual-encoder, encoder-decoder, etc.) and pre-training tasks (contrastive loss, masked prediction, etc.). However, since these works utilize different pre-training datasets and techniques, we focus our analysis on the ablation study to ensure a fair comparison between different architectures and pre-training tasks.

1. In Section 5.4, we conduct a comparative analysis of the ONE-PEACE architecture against alternative frameworks. We hypothesize that the shared attention layer facilitates the effective integration of multimodal data, while the separated FFNs enhance the model's ability to extract modality-specific information. This hypothesis is substantiated by the findings presented in Table 9. The training curves in Figure 4 also indicate that the design of the shared attention layer and separate FFN layer can lead to faster convergence speed and better convergence performance.

2. We further analyze the impact of various pre-training tasks. In comparison to exclusively utilizing cross-modal contrastive learning, the inclusion of denoising loss enables the model to learn fine-grained information within modalities, thereby leading to better fine-tuning and zero-shot retrieval performance. The results in Table 10 confirm this. Table 11 also provides a further comparison of different forms of denoising loss and validates that the denoising contrastive loss used in this paper is more compatible with cross-modal contrastive loss.

Moreover, we also compared with OFA[5], a generative model based on the encoder-decoder structure in Figure 5. It can be seen that ONE-PEACE exhibits better visual grounding ability on out-of-domain pictures.

[1] Gong Y, Rouditchenko A, Liu A H, et al. Contrastive audio-visual masked autoencoder[C]//The Eleventh International Conference on Learning Representations. 2022.
[2] Zhu W, Doshi K, Yi J, et al. Multiscale Multimodal Transformer for Multimodal Action Recognition[J]. 2022
[3] Huang P Y, Sharma V, Xu H, et al. MAViL: Masked Audio-Video Learners[J]. arXiv preprint arXiv:2212.08071, 2022.
[4] Yang P, Wang X, Duan X, et al. Avqa: A dataset for audio-visual question answering on videos[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 3480-3491.
[5] Wang P, Yang A, Men R, et al. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework[C]//International Conference on Machine Learning. PMLR, 2022: 23318-23340.

Thanks for your time reviewing and providing helpful suggestions for our paper. We address open questions and remarks below:

Discuss and compare with related works

Some of the works have been discussed in the related work section (ImageBind) or compared in the experiments section (CLAP). Due to the page limit, we added a new discussion section in the appendix to discuss the other works. Here, we briefly discuss the differences between ONE-PEACE and these works:

• ImageBind aligns multiple separated models with a pre-trained visual model to integrate different modalities into a unified semantic space. In contrast, ONE-PEACE uses a single model to align multiple modalities and allows them to interact through the self-attention mechanism, thus being more effective in handling fine-grained multimodal tasks like visual grounding. We updated the zero-shot results of ImageBind on ESC-50 in Table 6, and it can be seen that ONE-PEACE achieves better performance on this dataset.

• CLAP uses contrastive learning to align audio and language. We compare ONE-PEACE and CLAP in Table 5 and Table 6, ONE-PEACE achieves better performance in all the audio-related tasks.

• ULIP introduces the 3D modality through contrastive learning. However, since ONE-PEACE has not incorporated the 3D modality, we cannot directly compare it with ULIP. We discuss the related works in the newly added discussion section.

• Chatbridge provides multimodal capabilities for LLM by incorporating representation models. Our paper mainly compares ONE-PEACE with representation models, where generative models that combine LLMs and representation models have not been included in the baselines yet. As a general representation model, ONE-PEACE can also be integrated into LLM to provide multimodal capabilities. We discuss this in the discussion section.

Integrating and examining more modalities

it is noteworthy that ONE-PEACE has demonstrated remarkable performance on various video-related datasets, including K400, VGGsound, and AVQA. These results validates its capability across four modalities: image, audio, language, and video. We agree incorporating additional modalities such as 3D data and IMU can further demonstrate the potential of ONE-PEACE. However, due to time constraints during the response period, we added a discussion in the discussion section and left this for future research.

Thank you again for taking the time to review our paper. We appreciate your feedback and hope that we were able to address your concerns in our response. As the deadline is nearing, please let us know if you have any further questions before the discussion period ends. We are glad to address your concerns.

Thank you very much for the reply. We would like to ensure that we have addressed your concerns. If you have no further questions, would you kindly consider raising the score?