MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost

Dear reviewer fvJg,

Thanks so much for your constructive comments and support for acceptance. We hope our responses can address your concerns.

Q1: Lack of as-is baseline performances from English-only T2I backbone models.

A1: We thank the reviewer for emphasizing the importance of “as-is” baseline performances. In response, we evaluated Stable Diffusion v1.5 and PixArt-α by directly inputting prompts in 11 languages from the XM12 dataset, using InternVL-LLaMA to compute CLIP scores. These as-is results were then compared with our MuLan-adapted models.

We found that MuLan adapters consistently improved performance across all languages. Notably, even for languages close to English—such as French, Spanish, and German—our method still achieved clear gains over the as-is baseline, demonstrating its broad effectiveness.

Q2: Confusion over the CLIPScore specification.

A2: While the industry-standard CLIP-ViT models are indeed widely used, they are primarily trained on English data and thus provide unreliable similarity scores for non-English inputs. In contrast, InternVL-LLaMA has been trained on multilingual image-text pairs and possesses better understanding of non-English languages. Therefore, we chose InternVL-LLaMA as the surrogate model for computing CLIPScore in our paper.

To further ensure the objectivity of our CLIPScore evaluation, we additionally used the multilingual CLIP-ViT model released by LAION to compute similarity scores. The results, shown in the table below, demonstrate that our model still performs strongly and that the observed trends are consistent with those reported in Table 3 of the paper.

Q3: Limitations of CLIPScore.

A3: The prompts in XM12 predominantly feature compositional text prompts. While CLIPScore is useful for evaluating whether the main subject in the image aligns with the prompt, it falls short in assessing object-level details and spatial relationships. As you suggested, we additionally evaluated our model using VQAScore to better measure fine-grained text-image alignment.

Since the default VQAScore evaluation model (clip-flant5-xxl) does not support multilingual prompts adequately, we adopted GPT-4o as our evaluation model to enable multilingual VQAScore assessment. Results show that our model maintains strong performance, significantly outperforming GlueGen and AltDiffusion, and even surpassing or matching translation-based baselines across most languages.

Our model effectively leverages the capabilities of existing MLLMs, demonstrating strong generalization in multilingual image generation. In the future, we plan to further integrate the native multilingual, contextual, and reasoning abilities of MLLMs into the image generation process.

References

[a1] https://huggingface.co/laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k

[a2] Evaluating Text-to-Visual Generation with Image-to-Text Generation. https://arxiv.org/pdf/2404.01291 ECCV 2024

Dear reviewer RQVZ,

Thanks very much for your valuable comments. We hope our responses can address your concerns and clarify our contribution.

Q1. Comparison with translation baseline.

A1: In fact, previous multilingual text-to-image generation works (e.g., AltDiffusion) typically compare against open-source translation models such as NLLB [a1]. As shown in the following table, we also compared the results obtained using NLLB as a translation tool, and in 11 non-English languages of XM12, our performance exceeded the baseline of NLLB as a translation tool. We can also observe that using translation tools to handle non-English input is highly dependent on the performance of the translation tool, being very sensitive to its performance.

Google Translate is a strong commercial system with high development and data costs, and has undergone long-term refinement to reach its current performance, it remains a closed-source tool, making it unsuitable for offline or privacy-sensitive scenarios.

In contrast, MuLan is a low-cost baseline that achieves multilingual image generation by training on English image-text pairs and is entirely based on open-source suites. In terms of the FID metric, our model generally achieves better results. On CLIPScore, our results are already comparable to those obtained using Google Translate, and we show clear advantages in less common languages. These results highlight the potential of achieving multilingual image generation through training, without relying on external translation systems. We will include these points in the main paper in future revisions.

Q2. Adapt to the new SoTA model.

A2: One of our main contributions is the low adaptation cost (e.g., MuLan-SD15 training only requires 96 GPU hours). As demonstrated by the visual results in Figure 1 and Figure 4, the Adapter trained for Mulan-SD15 can be effectively applied to other fine-tuned variants of SD15 (e.g., Dreamshaper), and also works well with plugins such as LoRA and ControlNet. The same holds for models like SD21, SDXL, and PixArt-$\alpha$. This indicates that our method exhibits strong generalizability across this family of models, with minimal additional cost. For newly emerging SoTA models, our method can still be employed to support them and their related plugins and variants with similarly low adaptation cost.

Q3. More analysis of performance differences in table 5

A3: Based on the experimental results in Table 5, we can draw two main conclusions.

1. Only the aligned text encoder can support multilingual image generation through MuLan.

By comparing the models in Rows 1–3 and Rows 4–8 of Table 5, it is evident that without Language-Centered Alignment (LC) or Image-Centered Alignment (IC), the models are unable to support multilingual image generation through MuLan. We provide visualizations of the feature distributions for some of these models in Figure 5 of the appendix. As shown, models without alignment—such as LLaMA2-7B (Figure 5(d)) and XLM-R Large (Figure 5(a))—produce scattered features when processing inputs with the same meaning but in different languages. In contrast, aligned models—such as XLM-R Large* (Figure 5(b)), Mul-OpenCLIP (Figure 5(c)), and InternVL-LLaMA (Figure 5(e))—can cluster features of semantically equivalent inputs across different languages. This is a key reason why our method, trained solely on English image-text data, is still able to support multilingual image generation.

2. Image-Centered Alignment outperforms Language-Centered Alignment.

By comparing the models in Rows 4–6 and Rows 7–8, we observe that LC (Language-Centered Alignment) yields inferior results compared to IC (Image-Centered Alignment). This is primarily because LC relies on translated data, which may introduce noise due to inaccuracies in translation. Moreover, IC aligns multilingual semantic features through the shared image feature space, effectively providing MuLan with a prior that facilitates further refinement of the alignment. In contrast, LC relies entirely on MuLan to establish the connection between language features and image features from scratch. These factors collectively contribute to the inferior performance of LC compared to IC.

References

[a1] https://huggingface.co/facebook/nllb-200-3.3B

Dear reviewer BvZv,

Thanks a lot for your insightful reviews and support for our work! We hope our responses can address your questions.

Q1. Lack of a cross lingual consistency evaluation

A1: We appreciate the reviewer’s suggestion to evaluate cross-lingual conceptual consistency. Our work focuses on enabling multilingual text-to-image generation by leveraging a pretrained multilingual text encoder and English image-text pairs. While our current method demonstrates some degree of cross-lingual consistency, it may struggle with culture-specific corner cases, which fall outside the scope of this work and would require human curation or high-quality data to resolve. We will acknowledge this limitation in the main paper and plan to address it in future work. As a potential direction, we may explore collecting data with explicit culture-specific concepts and developing a dedicated benchmark for evaluating cross-lingual consistency in multilingual T2I generation.

Q2. Clarification on hyperparameter settings

A2: The training hyperparameters for MuLan-SD15 and MuLan-PixArt used in Table 3 are already provided in Section 4.1. Here we provide additional details for completeness: we use the AdamW optimizer with $\beta = (0.9, 0.999)$ and a weight decay of 0.01. For SD 1.5, SD 2.1, and PixArt, we follow the original models’ resolutions (512x512 or 768x768). For SDXL, we adopt a two-stage training strategy with different resolutions to ensure stability. We adopt classifier-free guidance by randomly dropping text conditions, following [a1].

In Table 5, we only replace the language model; all other training settings are identical to those of MuLan-SD15.

We will include these details regarding the selection of experimental hyperparameters in the updated manuscript.

Q3. Ablation study on the model size

A3:

For the size of the adapter

In the main paper, our proposed adapter uses a lightweight one-layer Transformer encoder-decoder structure (see Section 3.2). To further investigate the relationship between model size and performance, we have conducted additional ablation experiments on Stable Diffusion 1.5, testing adapters with 2, 4, and 6 Transformer layers. The experimental setup follows Appendix A.1. We calculated the average CLIPScore across seven languages from the XM12 dataset.

We observe that increasing the number of Transformer layers in the adapter does not lead to improved performance; in fact, it slightly degrades performance. This result aligns with previous findings in works such as LLaMA-Adapter [a2] and MiniGPT-4 [a3], which demonstrate that adapter-based tuning for LLMs typically does not require large parameter counts or massive training data. Our lightweight adapter design is consistent with these community practices, suggesting that a compact architecture is sufficient and may even be more stable and efficient for adaptation.

For the size of the language model

Our approach relies on a series of pretrained multilingual language models, making it difficult to perform strictly controlled ablation studies on the size of the language model. These models differ not only in parameter count but also in architecture, tokenizer, training corpora, which complicate direct comparisons.

Although a strictly controlled study is not feasible, a coarse-grained observation from Table 5 reveals a trend suggesting that the size of the language model may impact the performance upper bound. Specifically, the first four models—MultiLang-CLIP (33.2), AltClip-m18 (33.3), XLM-R Large* (34.7), and Mul-OpenCLIP (36.1)—all use XLM-R Large(335M) as the language model. In contrast, InternVL-LLaMA(7B), which achieves the highest score of 37.8.

Since our approach builds upon existing pretrained multilingual models, conducting strictly controlled comparisons on model size is non-trivial. As part of future work, we plan to explore how our adapter performs when integrated with increasingly stronger language models, which could further reveal its potential in real-world multilingual generation.

References

[a1] Ho, Jonathan. “Classifier-Free Diffusion Guidance.” https://arxiv.org/abs/2207.12598

[a2] Zhang, Renrui et al. “LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention.” https://arxiv.org/abs/2303.16199.

[a3] Zhu, Deyao et al. “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models.” https://arxiv.org/abs/2304.10592