Parrot: Multilingual Visual Instruction Tuning

Thank you for your kind comments and constructive feedback on our paper.

Q1: Motivation of data efficiency.

A1: While large-scale translated multilingual data may seem abundant, its quality (especially for low-resource languages) is often critically compromised due to translation errors, cultural mismatches, and noisy artifacts. Even with massive data volumes, low-resource languages remain underrepresented in practice because high-resource languages inevitably dominate training distributions (e.g., 90%+ of tokens in typical datasets). Forcing higher proportions of low-resource data risks triggering the curse of multilingualism, where over-parameterized models sacrifice high-resource language performance to accommodate low-resource languages, as observed in prior work [1-3]. Our approach strategically prioritizes high-quality alignment signals over raw data quantity, avoiding both the pitfalls of noisy translation and the imbalance inherent to brute-force multilingual scaling. This ensures stable performance across all languages without compromising high-resource capabilities. We will further discuss and refine the motivation, taking this issue into careful consideration in the final version.

[1] Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020.
[2] When Is Multilinguality a Curse? Language Modeling for 252 High- and Low-Resource Languages. ICLR 2024.
[3] Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models. EMNLP 2024.

Q2: Translate-train/test baselines.

A2: We appreciate the insightful suggestion to include translate-train and translate-test baselines. While these are valid approaches, they face critical limitations:

1. Translate-Train:
   • Data Quality: Machine-translated data often contains semantic distortions, syntactic errors, and cultural mismatches (e.g., idioms, region-specific references), especially for low-resource languages. For example, as shown in the table below, our experiments showed that training with 70K translated multilingual samples for each language achieved limited improvement.
   • Imbalanced Optimization: Merging large-scale translated data amplifies the dominance of high-resource languages (e.g., English/Chinese), as models tend to overfit to their syntactic patterns, further marginalizing low-resource languages. This phenomenon is called the curse of multilingualism.

2. Translate-Test:
   • Latency Overhead: Translating inputs introduces 2× latency (translation + inference), making real-time applications impractical. For instance, translating a 100-token Arabic query to English adds ~500ms latency (Google Translate API), which is prohibitive for interactive systems.
   • Limited Gains: As shown in Table 14 in Appendix, our preliminary tests with LLaVA on MMMB showed that the translate-test improved Arabic accuracy by only 3.8% (vs. Parrot’s +19.2% gain), while degrading Russian performance by 0.2% due to translation errors. Further details are provided in Response A8 of the reply to Reviewer Gwu1.

These results align with prior findings [4-5], where translate-train/test paradigms underperform dedicated multilingual architectures in both efficiency and robustness. Parrot circumvents these issues by directly aligning cross-lingual semantics without relying on noisy intermediate translations.

[4] Lost in Translation: Analysis of Information Loss During Machine Translation Between Polysynthetic and Fusional Languages. ACL 2018.
[5] Why do LLaVA Vision-Language Models Reply to Images in English? EMNLP2024.

Q3: Template formatting issue.

A3: We sincerely appreciate the reviewer’s attention to detail. Upon re-examination, we confirm that the manuscript was prepared using ICML’s official template. However, subtle discrepancies in visual formatting emerged during PDF compilation due to technical nuances in rendering tools. Regardless of the compilation method, the page limits and other requirements are fully in line with ICML’s guidelines. While we cannot directly update the PDF during the rebuttal phase, these unintended inconsistencies have now been fully resolved, with corrections to be incorporated into the final version.

We sincerely appreciate the reviewer’s thoughtful and candid feedback.

Q1: Prior work and drawback.

A1: In section B in Appendix, we have discussed prior multilingual MLLM methods like mCLIP, VisCPM, and M3IT. Most prior work relies on large-scale multilingual multimodal data (e.g., M3IT uses 2.5M+ samples), which is impractical for low-resource languages (more details are referred to in Response A1 of the reply to Reviewer QoYJ). Other multilingual CLIP methods improve performance for specific languages but sacrifice generalizability across diverse languages due to fixed encoders. For more clarity, we will expand this section in the revision to explicitly outline their key drawbacks.

Q2: Benchmark and method co-design.

A2: MMMB is not specifically designed for Parrot. In contrast, MMMB and Parrot are co-designed to address two limitations in multilingual multimodal research: (1) existing benchmarks (e.g., M3Exam, LLaVA-Bench) not only lack standardized linguistic coverage, typically limited to 2-3 high-resource languages with inconsistent annotation protocols, but also exhibit systemic limitations that we have discussed in Section 2.1; (2) conventional models trained on imbalanced SFT data exhibit performance erosion across languages, yet lack evaluation frameworks to diagnose such failures. Additionally, we conducted experiments not only on our designed benchmark but also validated results on other multilingual benchmarks, such as MMBench (Table 1) and LLaVA-Bench (Table 4 in the Appendix).

Q3: Using multilingual CLIP and LLM.

A3: In our preliminary experiments, we explored the use of multilingual CLIP as a baseline. However, this approach exhibited critical limitations: (1) It failed to balance performance across all target languages, as reliance on the inherent multilingual capacity of CLIP led to inconsistent generalization. In other words, it is hard to generalize beyond typologically similar languages covered in multilingual CLIP's pretraining. (2) Multilingual CLIP showed inferior visual perception capabilities compared to the original OpenAI-CLIP, degrading performance on general visual tasks. As shown in the table below, while LLaVA equipped with multilingual CLIP shows improvement in Chinese (+1.5), Arabic (+2.7), and Turkish (+1.1), performance in other languages declines (e.g., English -1.7).

In contrast, Parrot leverages a single OpenAI-CLIP alongside a multilingual LLM backbone (Qwen-LM) and achieves balanced, state-of-the-art performance across all 6 languages in two benchmarks. This demonstrates that our MoE-driven alignment paradigm effectively resolves language bias without requiring language-specific or multilingual visual encoders, while preserving strong general multimodal perception capabilities.

Q4: Lack of comparison to the latest MLLM.

A4.1: Current MLLMs are primarily data-driven or architecture-driven. Our work aims to achieve data-efficient multilingual adaptation by enhancing multilingual capabilities with minimal training data while preserving general ability, rather than developing a model with exceptionally strong general capabilities. Many leading methods do not release their data. Therefore, direct comparisons of general capabilities with strong models risk unfairness due to vast data disparities. Despite this, we also compare Parrot to general-purpose MLLMs (e.g., Qwen2-VL (2024.09) and LLaVA-OV (2024.08)) on the MMMB and multilingual MMBench dataset in the table below and Table 12 in Appendix. Although Qwen2-VL and LLaVA-OV are trained with over 10x the amount of data used by our model, Parrot still outperforms them on the multilingual benchmark. This comparison spans two datasets and six languages, demonstrating Parrot's superior performance over these recent approaches.

A4.2: For the radar chart in Figure 7, we have compared Parrot with the SOTA in 2024 (e.g., Mini-Gemini and LLaVA-Next), which shows Parrot’s competitiveness even with limited data. Crucially, Parrot’s goal is not to surpass general-purpose MLLMs but to enhance multilingual ability with minimal data.

We sincerely thank the reviewer for their thorough and constructive feedback, as well as their endorsement of our work.

Q1: MoE vs. Simpler Methods (LoRRA).

A1: We explored the LoRRA-based (abbreviated as L-based) adaptation shown in the table below but found it insufficient for two reasons.[1]

1. L-based method introduces fixed language-specific parameters, which struggle to handle multilingual prompts dynamically (e.g., mixed-language queries).
2. Its low-rank updates are less effective for aligning diverse languages, especially when training data is sparse.

In contrast, MoE dynamically routes visual tokens to language-specific experts based on textual guidance, enabling flexible adaptation with minimal parameters. Additionally, due to the small number of parameters in MoE (<0.5% of the parameters in the LLM), it is not an expensive method.

[1] Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020.

Q2: Whether each language expert indeed handles its language-specific embeddings.

A2: In Figure 7c, the router activates the Chinese expert most strongly, but experts are not strictly language-exclusive. Instead, they learn cross-lingual synergies (e.g., English/German share morphological features). To further present the activation using other language prompts, we will add t-SNE visualizations of expert activations across languages and include expert distributions of MoE for each language in the final version.

Q3: Evaluation fairness.

A3: To validate Parrot's effectiveness, we conduct an ablation study by expanding LLaVA with the same multilingual data used in Parrot. Both models are evaluated on MMMB, with results in Table 13 in Appendix. While LLaVA shows slight improvement, the gains are limited. In contrast, Parrot achieves substantial improvements, highlighting that simply adding multilingual data is insufficient to bridge the gap. Moreover, the findings from the ablation study in Figure 6a further support this conclusion, reinforcing the validity of our design.

Q4: Ablation studies.

A4: In the previous response (A1), we provided comparisons between Parrot and L-based alternatives. Due to time and word constraints during the rebuttal phase, we will include expanded ablation studies on expert counts, sizes, and frozen vs. trainable configurations in the final version.

Q5: Suggestions regarding the figure revisions.

A5: We thank your valuable feedback. We are committed to implementing these revisions and will incorporate them into the final version.

Q6: Non-English data scarcity.

A6: While non-English web data is abundant, high-quality multimodal data remains scarce. 1) Most non-English alt-text is noisy or misaligned (e.g., social media images with irrelevant captions). 2) Parrot’s semi-automatic curation (Figure 3) ensures linguistic and cultural precision, which raw web data lacks. Further details are provided in Response A1 of the reply to Reviewer QoYJ.

Q7: Mitigating multilingual erosion via pre-training.

A7: Incorporating more multilingual data during pre-training could help mitigate multilingual erosion, but it may not fully resolve the issue. The main goal of multimodal pre-training is to learn a projector for cross-modal alignment, which often remains biased toward dominant languages (e.g., English) due to data imbalances. Without explicit mechanisms like Parrot's MoE-based language-specific alignment, subsequent SFT stages would still be affected by English-centric bias. Due to word constraints, we will include experiments with multilingual pre-training data in the final version.

Q8: Translation baseline explanation.

A8: The translation baseline uses Google Translation API to translate non-English queries to English, feeds them to LLaVA, then translates responses back.

1. Translated prompts often lack cultural-specific context. Image: A traditional Chinese red envelope (红包) with handwritten characters "福到" (upside-down "福", symbolizing "fortune arrives") and "岁岁平安" ("peace every year"). The translated-based method cannot perform Glyph-aware visual grounding (recognizing 福's inverted form as intentional)
2. When a Portuguese user asks about a Russian meme with the text "Почему программисты любят кофе? Потому что Java!" (where Java is a pun on both coffee culture and programming language), machine translation to English collapses the dual meaning into "coffee brands," stripping the programming-language humor.

We have presented examples (Figure 10) where translation easily fails due to ambiguity or cultural nuances.

We are deeply grateful for the reviewer’s thorough and thoughtful assessment of our work, as well as their recognition of Parrot’s contributions.

Q1: Scalability to many languages.

A1: Parrot’s MoE framework is inherently designed to support seamless integration of new languages. The addition of language-specific experts requires only minimal data (e.g., ~5k samples per language) and incurs a linear parameter overhead (one expert per language). Empirically, we observe robust knowledge transfer within language families. We will extend Parrot to encompass broader linguistic diversity in future work and present more case studies given an unseen language to the model.

Q2: MoE re-weighting parameter.

A2: The trade-off parameter $\alpha$ in Eq. 5 balances the preservation of original visual semantics ($\alpha=0$) against language-specific transformation strength ($\alpha=1$). Through grid search on MMMB validation data with $\alpha \in \{0.1, 0.3, 0.5, 0.7, 1.0, 1.5\}$, we found $\alpha=0.5$ optimally maintains visual fidelity while enabling robust multilingual alignment (>5% gain over $\alpha=0.1$), as excessive reliance on original English-biased features hindered language adaptation. Higher $\alpha$ values (>1.0) caused degradation in general visual tasks (e.g., -2.2% on MMBench object counting). We will include full parameter sensitivity curves in the final version to clarify this design choice.

Q3: MoE gating mechanism.

A3: As described in Eq. 4, we use soft gating (weighted combination of experts) rather than hard routing during inference. This ensures smooth transitions between languages, avoids overfitting to dominant experts, and dynamically handles multilingual prompts (e.g., mixed-language queries). We will add a discussion in §3.3 highlighting this design choice.

Q4: Base LLM impact.

A4: We ablated this by replacing Qwen1.5-7B with Vicuna1.5-7B (weaker multilingual support). As shown in the table below, Qwen1.5 outperforms Vicuna1.5 by +4.4% on Turkish and +8.5% on Arabic. This highlights that Parrot’s effectiveness is somewhat reliant on the base LLM’s multilingual ability. Notably, while using the weaker multilingual LLM, the performance gain compared to the baseline LLaVA-1.5 is remarkably large.

Q5: Inference efficiency.

A5: Parrot maintains inference efficiency comparable to LLaVA by executing cross-attention and MoE processing once per image during visual token projection, not per generated token. The lightweight MoE module adds <0.5% parameters to the LLM, resulting in almost no extra runtime latency increase versus LLaVA-1.5 under identical hardware. Frozen CLIP encoding and single-pass visual-language alignment ensure the computational overhead remains negligible despite enhanced multilingual capabilities.

Q6: High-resolution image handling.

A6: Thank you for your valuable suggestion. Parrot currently uses CLIP ViT-L/14-336 that is the maximum input size version. Crucially, our current implementation prioritizes fair comparison with LLaVA-1.5 baselines that share the same CLIP backbone. In future work, we plan to integrate dynamic-resolution approaches like NaViT's flexible patching or SigLIP's high-res pretraining to better handle high-resolution images while maintaining multilingual alignment capabilities.

Q7: Minor revisions.

A7: Thanks for the detailed review and helpful comments. We will incorporate these updates into the final version.

1. Figure 2 caption: We have clarified the caption to "Examples of suboptimal multilingual benchmark design in existing works, as evaluated against MMMB's principles."
2. Reference: we shall include a comprehensive comparative analysis of multilingual MLLM's recent advancements in the final version.