Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

Thanks to the reviewer for recognizing the significant contributions of our work, including the PANGEA model, the PANGEAINS dataset, and the PANGEABENCH evaluation suite. We are glad the reviewer appreciates the comprehensiveness of our datasets, benchmarks, and evaluation results, as well as the practical value they offer to the multilingual research community.

The paper lacks some crucial details about the model and training process.

Thank you for pointing this out! We added more details on the training specifics in the revised version in Section 4.1. Specifically:

• We used the LLaVA-NeXT framework, with Qwen2-7B-Instruct as the language backbone and clip-vit-large-patch14-336 as the vision encoder.
• The training consists of two stages:
  1. Pretraining the connector on the LLaVA pretraining dataset, which took around 4 hours with 8 × H100 GPUs (32 GPU hours).
  2. Finetuning for 1 epoch on PANGEAINS, which took approximately 168 hours with 8 × H100 GPUs (1344 GPU hours).

Some evaluation setup details are missing, such as the number of shots for each dataset. For instance, the results for Qwen2-7B-Instruct are significantly worse than those reported in the official release (https://qwenlm.github.io/blog/qwen2/#multilingual-capability-of-instruction-tuned-models). What is the reason?

Thank you for pointing this out! We added more details of the evaluation setup in Section 3.1 in the revised version. For all the evaluation, we used a zero-shot setting for all evaluation benchmarks.

In the Qwen official release, results were demonstrated for MMMLU and MGSM. Key differences:

• For MMMLU, our reported performance is higher because we used the OpenAI version of MMMLU (link), which was not available during the Qwen official release.
• For MGSM, we were unable to reproduce their results under the zero-shot setting. However, under the 5-shot setting, our performance closely aligns with theirs:

We evaluated Qwen2-7B-Instruct's performance under the 5-shot setting:

• English: 79.2
• Multilingual: 50.7
• Overall: 53.3 (close to their reported overall performance of 57.0).

It's unclear which specific languages have shown improvement and to what extent compared to the backbone model.

In the text-only benchmarks, we evaluated the performance of both Qwen2-7B-Instruct (the backbone model) and Pangea. In the appendix, we included a language-wise breakdown of results in Tables 16, 17, and 18. Specifically, Pangea shows improvements over Qwen2-7B-Instruct in the following languages:

• Korean
• Swahili
• Hindi
• German
• Japanese
• Thai

In Figure 7, what might explain the model's unexpected poor performance on English OCR?

It is important to note that this is a preliminary exploration of constructing OCR instruction tuning data, which aims to provide insights for future research on OCR training. For testing the model's OCR capabilities, we reserved a small portion of the OCR instruction tuning dataset curated from multilingual web screenshots. Upon manual inspection, we found that the English training data for web OCR tasks had more complex structures and contexts compared to data in other languages. This increased complexity in English OCR data likely explains the model's unexpected poor performance on English OCR tasks.

We appreciate the reviewer’s feedback! We would like to address the following concerns:

There exists a potential imbalance in both the quantity and quality of data for specific languages. What strategies can be implemented to ensure a more balanced representation of low-resource languages within the PANGEAINS dataset?

We acknowledge that there might be an imbalance in the quality and quantity of data for lower-resource languages. When curating PangeaIns, we aimed for balanced sampling, and we believe that abilities gained from training on high-resource languages can be transferred to lower-resource languages. For example, we achieved similar or higher accuracy on two low-resource languages, Swahili and Tamil, on MaRVL, as shown in Table 9 in the paper.

How can we establish rigorous quality assessment protocols to evaluate the accuracy and relevance of data for each language in the dataset?

To ensure the high quality of the training data, we performed:

1. Careful selection of existing datasets and comprehensive filtering to ensure only high-quality data is included.
2. Manual quality checks for the images and texts of each existing corpus before incorporation into PangeaIns.
3. Input from multilingual experts to inspect the quality of multilingual data.

For translations:

• We examined various open-source translation models, such as NLLB-3B, but found through manual inspections that they struggled with complex instruction-following and context-switching tasks, especially in specialized domains like code generation and mathematical reasoning.
• Therefore, we opted for the proprietary Gemini 1.5 Pro model, which demonstrated slightly better performance in small-scale human evaluations compared to GPT4o.

For culturally relevant data:

• We manually inspected a small subset of outputs in all 39 languages to ensure high quality.
• Once satisfied with the general quality, we curated the culturally relevant subset of PangeaIns and filtered out lower-quality data using a large language model (Llama-3.1-8B-Instruct).

In evaluating cultural understanding, the discussion regarding whether the PANGEAINS dataset adequately reflects diverse cultural backgrounds is somewhat lacking. What methodologies can be employed to enrich the dataset with culturally relevant examples and ensure that cultural nuances are adequately captured across all languages?

In PangeaBench, we evaluated Pangea on two benchmarks:

1. CVQA
2. MaRVL

These benchmarks test model performance in reasoning tasks involving culturally relevant imagery and concepts across multiple languages. We believe that high scores on these benchmarks reflect:

• Good cultural understanding capabilities of Pangea.
• The effectiveness of PangeaIns in providing diverse and culturally rich training data, which enables the model to perform well in cultural understanding tasks.

From the perspective of experimental design, there is a notable lack of in-depth analysis concerning specific tasks or languages. For example, in tasks such as multilingual visual question answering (VQA) or multi-subject reasoning, a detailed exploration of the factors influencing model performance is missing.

In Section 5 and Figure 6, we performed an in-depth analysis on the impact of the proportion of training samples in a language on downstream performance. Additionally, we explored factors influencing model performance on multimodal chat in Section 5 and Appendix E.

What role can data augmentation techniques play in enhancing the dataset, particularly for underrepresented languages, to improve model performance?

We utilized data augmentation techniques to enhance data in lower-resource languages such as Telugu and Swahili. Data augmentation is particularly valuable in cases where data for specific languages is limited. For instance:

• We used machine translation and culturally relevant curation to address resource insufficiencies.
• These efforts led to improved performance in underrepresented languages. Table 9 demonstrates an example of such improvements.

Thanks to the reviewer for highlighting the significant contributions of our work, including the PangeaIns dataset, PangeaBench evaluation suite, and the Pangea model, which together provide robust multilingual multimodal resources and achieve superior performance across diverse tasks!

In Section 2.1, the paper mentions a post-processing pipeline for noisy translations; however, details are missing.

Thank you for pointing this out! We will release all the code for curating PangeaIns, including that for post-processing translations. For example, in the post-processing step, we checked whether the number of turns in conversations remains the same post-translation, and we verified whether the ratio between the length of pre- and post-translation is reasonable.

In Section 4.1, it has been mentioned that the model is based on LLaVA-NeXT architecture, but this needs to be detailed.

Thank you for pointing this out! We added more details on the training specifics in the revised version in Section 4.1. Specifically:

• We used the LLaVA-NeXT framework, with Qwen2-7B-Instruct as the language backbone and clip-vit-large-patch14-336 as the vision encoder.
• The training consists of two stages:
  1. Pretraining the connector on the LLaVA pretraining dataset, which took around 4 hours with 8 × H100 GPUs (32 GPU hours).
  2. Finetuning for 1 epoch on PANGEAINS, which took approximately 168 hours with 8 × H100 GPUs (1344 GPU hours).

For text-only performance, how is the chat performance compared to other models? Some evaluation benchmarks such as MTBench, ArenaHardAuto, or AlpacaEval can be used for this comparison.

In PangeaBench, we didn’t include text-only chat evaluation benchmarks. Instead, we included two multilingual multimodal chat benchmarks:

• Multilingual LLaVA Bench
• XChat

We believe these benchmarks provide good indicators of Pangea’s chat capabilities. Additionally, we evaluated Pangea and LLaVA-NeXT on MT-Bench. The results are as follows:

Have you checked if there is any data leakage from the benchmarks that have been tested?

Yes! We ensured that no data from PangeaBench was included in PangeaIns. However, it's worth noting that PangeaIns does include the training splits of NLVR2 and GQA-ru datasets. In PangeaBench:

• We evaluated model performance on the evaluation splits of these datasets:
  • NLVR2's evaluation split was used for the English performance of the multilingual-only dataset MaRVL.
  • GQA-ru's evaluation split was used for GQA-ru.

It is important to note that this is a common practice in instruction tuning, as seen in [1, 2].

[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. URL
[2] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. ArXiv preprint, abs/2406.16860, 2024. URL

Thanks to the reviewer for the positive feedback on our paper addressing multilingual multimodal learning, highlighting the well-detailed PANGEAINS dataset, PANGEABENCH evaluation suite, and the comprehensive experiments demonstrating strong model performance.

It could be helpful to add a section to discuss related multilingual multimodal datasets (e.g., M3IT, etc.), and show how PANGEA compares to them.

In the revised version, we added a table comparing other multimodal instruction tuning datasets to PANGEAINS in Table 4 and included more discussion of other multilingual multimodal datasets in the Appendix A Related Works section.

Although the authors highlighted the introduction of PANGEA, the discussion on the modeling aspect of PANGEA is light.

Thank you for pointing this out! We added more details on the training specifics in the revised version in Section 4.1. Specifically:

• We used the LLaVA-NeXT framework, with Qwen2-7B-Instruct as the language backbone and clip-vit-large-patch14-336 as the vision encoder.
• The training consists of two stages:
  1. Pretraining the connector on the LLaVA pretraining dataset, which took around 4 hours with 8 × H100 GPUs (32 GPU hours).
  2. Finetuning for 1 epoch on PANGEAINS, which took approximately 168 hours with 8 × H100 GPUs (1344 GPU hours).

The proposed method (model and data) is only evaluated on one eval set, which is the proposed PANGEABENCH. It could be helpful to evaluate on additional evaluation suites.

PANGEABENCH is not a single eval set. Instead, it consists of 14 unique evaluation datasets, covering a wide range of languages and tasks, including:

• Multimodal chat
• Image captioning
• Cultural understanding
• Multilingual visual question answering
• Multi-subject reasoning
• Text-only evaluation

Among the evaluation datasets in PANGEABENCH, 12 out of 14 datasets are out-of-domain. Thus, we believe PANGEABENCH provides a comprehensive evaluation of PANGEA's capabilities.

Would the other open-source models used as the baseline in this paper also benefit from training on the PANGEAINS data?

Yes! We believe that training on PANGEAINS, a high-quality instruction tuning multimodal dataset spanning 39 languages, could improve the performance of other models. We chose LLaVA-NeXT as our model backbone due to its simplicity, effectiveness, and widespread adoption.

How to validate if there is potential overfitting of training on PANGEAINS and evaluating on PANGEABENCH?

It's important to note that PANGEAINS was curated with a primary focus on providing high-quality multilingual and multimodal instructions, rather than being tailored specifically for performance on PANGEABENCH. In fact:

• Only 2.2% of the samples in PANGEAINS are in-domain training examples related to PANGEABENCH.
• We also conducted a data contamination check before training the model. The strong performance observed on PANGEABENCH reflects the robust capabilities developed through training on the diverse and comprehensive data in PANGEAINS, rather than overfitting to the benchmark dataset.