Multilingual Diversity Improves Vision-Language Representations

We thank the reviewer for the feedback.

It is very important that we clarify a misconception: our work deals with vision-language models and vision benchmarks. We do not train any LLMs or make claims about the state of multilingual NLP research.

Framing should be more sensitive to multilingual efforts in NLP/ This work uses data to improve English-only models while we already lack people working on multilingual models. To clarify,

(i) We support multilingual and multicultural efforts. While benchmarking and improving performance on multilingual tasks is not the focus of our work, as one step in this direction, we enable English-language models to gain exposure to more samples of diverse origins and consequently, do better on geographically diverse benchmarks (e.g. GeoDE and DollarStreet). We discuss how to adapt models from our paper to further enhance their multilingual capabilities under the “Future work” section. Comparing the multilingual performance of these post-training adaptation methods to directly training on non-English captions merits a separate investigation.

(ii) Instead, the goal of our work is to challenge the status quo: many ML practitioners seek to improve state-of-the-art performance on popular benchmarks, which happen to be English vision tasks like Imagenet. In the process of overfitting to these benchmarks, previous work discards a lot (if not all) of non-English samples due to the belief that non-English data does not benefit English evaluations. We question this practice. But in order to incentivize change, we need to show that it is possible to improve the English-centric “state-of-the-art” by using more data of non-English origins. We hope that our positive findings could help multilingual and multicultural efforts become the default design choice for data curation, instead of existing as a second priority or as a separate (societal) consideration.

The improvement from using translated data diminished as the training converged, which is not surprising. We want to clarify that the results we reported show the opposite. From Table 1 of the main paper, our best baseline, “Filtered raw captions & Filtered translated captions”, outperform “Filtered raw captions” by 2.0% on ImageNet and 1.1 percentage points on average across 38 tasks. When training for 10x more steps, the performance gaps increase to 4.2% and 2.1 percentage points respectively.

Using more training data to improve the evaluation results on downstream tasks is also expected. For our baselines, we fix the training budget and the amount of raw data available (i.e. initial pool). With each caption distribution, we tune the filtering threshold, and consequently the size of the resulting filtered subset used for training (this was reported in Table 1 of the main paper). We note that with this setup, increasing filtered data quantity is not guaranteed to lead to better performance on downstream tasks, as larger subset (with less strict filtering) means fewer passes through each datapoint. In Table 1 we only report the best performance obtainable after filtering each data distribution. Table 6 (Appendix) contains the full results, accounting for the number of samples used. While “Top 20% raw captions & Top 20% translated captions” indeed uses more training data than “Top 20% raw captions” (51.2M versus 25.6M), when controlling for data quantity taken from the latter distribution, “Top 20% raw captions & Top 20% translated captions” still significantly outperforms “Top 40% raw captions” (both having 51.2M samples).

Add more multicultural evaluation datasets. Given the reviewer’s concern, we have added a new evaluation on Google Landmarks dataset (Weyand et al., 2020). When training for 10x longer, using filtered raw captions yields 16.9% classification accuracy, while our best baseline (using a mix of filtered raw captions and translated captions) gets 18.5% accuracy.

Show the benefits of using diverse training data could benefit multiple models sizes and types, and other languages. Due to compute constraints, we did not experiment with more model types and translating to other languages, but we will add these points to our Discussion section. For context, obtaining the baseline numbers for our paper (see Appendix Tables 6 & 7) took about 850 GPU hours with 8 A40 GPUs. Besides, we find that extending our approach to other languages (e.g. translating all captions to Chinese and measuring performance on Chinese tasks) is sufficient scope for a separate paper.

We thank the reviewer for all the valuable suggestions!

Showcase the diversity for certain concepts. Per your suggestion, we ran additional analysis to compare the diversity of images for the same concept. We sampled 1K images from each data distribution for which the corresponding (translated) captions mention “stove”. With DINOv2, we obtained embeddings of these images and used them to cluster each data distribution into 20, 50 and 100 clusters. Figure 2 in our global response shows the average inter-cluster distance of “stove” images in English data versus non-English data. Overall the latter group of images yields higher inter-cluster distance, suggesting that the clusters are more well-separated and the non-English “stove” images are more diverse. We will add this experiment to the paper.

Benchmarking on culturally diverse data is still limited. Our work does not explicitly seek to improve performance on culturally diverse tasks. We explore whether increasing the diversity of the data origins, and consequently allowing the training set to be dominated by non-English samples, can improve performance on vision tasks in general, especially on those that dictate the field’s state-of-the-art like ImageNet. Improvement on geographically diverse benchmarks is an added benefit in this process. However, given the reviewer’s feedback, we have added a new evaluation on Google Landmarks dataset (Weyand et al., 2020). When training for 10x longer, using filtered raw captions yields 16.9% classification accuracy, while our best baseline (mixing filtered raw captions and translated captions) gets 18.5% accuracy.

By filtering on top of the translated image-text pairs, potentially train-test similarity also increases. Changes in train-test similarity may happen as a result of using translated captions, but that should not be a confounder in the paper takeaways because (i) the proposed translation method does not directly optimize for train-test similarity, (ii) we apply the same filtering process (i.e. based on DFN score) to both raw and translated captions, (iii) we measure performance on a large number of vision tasks (38).

Checking how the distribution of CLIPscore look before and after the translation. We have added this analysis in our global response (see Figure 1 of the attached PDF). We sampled 10K images with English captions and 10K with non-English captions from the initial pool, and compared how the DFN scores change with translation. Unsurprisingly, DFN scores for non-English samples generally increase after the captions are translated into English. This in turn leads to a right shift in the distribution of image-text similarity scores when looking at the score histograms before and after translation. As image-text alignment (measured by DFN score) tends to help with empirical performance, this shift suggests that translation increases the availability of good training data.

Estimate that English takes up ⅓ of any web crawl. This estimate is computed from using the NLLB model to detect languages in the DataComp dataset. Since DataComp performed minimal preprocessing (i.e. NSFW removal and test set deduplication), we hypothesize that the initial DataComp pool is representative of the natural distribution of raw data on the web that is appropriate for training.

Improvement on ImageNet contradicts the results of the No-Filter paper. We note some differences between our experiment setup and that of concurrent work:

(1) We compare training on filtered translated captions (which is dominated by non-English data) to training on filtered raw captions (which is dominated by English data). In contrast, the No-Filter paper compares multilingual data (globe) to its strict subset containing only English data (en).

(2) We perform filtering and retain only a small fraction of the original pool (20-30%), whereas the No-Filter paper trains on web data with “minimal filtering”. We posit that filtering is important to obtain competitive performance with translated captions, especially given the noise introduced by the translation process.

We believe our setup better mimics how pre-training data curation is commonly done in practice (e.g. LAION, which was also heavily filtered and contains a mix of English and non-English data).

Why filtered translated captions might help improve English-centric performance. While the fraction of images of English origins is lower when we use filtered translated captions instead of filtered raw captions (60% → 40%, Figure 2 of our paper), the number of such images in the final training set is still significant. As a result, the model can still learn English-centric concepts to a certain extent. We hypothesize that the performance on English vision tasks is further reinforced by training on high-quality, diverse multilingual data, which helps induce better visual features and robustness in general.

Average performance improvement in Figure 3. The analysis in Figure 3 of our paper is performed on models trained with the top 30% of the initial pool (see description in Lines 251-254), whereas the first section of Table 1 involves training on the top 20% of the pool. We pick the 30% threshold for further analyses because (i) it is the baseline that yields the best performance when training until convergence, (ii) the average performance gap between filtered translated captions and filtered raw captions is larger at this threshold, allowing us to observe the differences on individual tasks better.

Why does performance on Food101 go down? We note that (1) the accuracy change for the Food101 task between using filtered translated caption and using filtered raw captions is relatively small (-0.2%), and (2) the Food101 class names are still dominated by English-centric concepts. Besides, the noise in the translation process may affect whether the class mentions in the original languages are still preserved after translation.

Thank you for the review and for recognizing the strengths of our work!

How the proposed automated translation pipeline ties in to the original motivation (culturally specific items not being captured in English data). We discuss the presence of culturally salient concepts in the Introduction mainly to motivate how multilingual data is inherently enriching and complements English data. Improving the availability of culturally specific items in the final training set is not what our method specifically optimizes for, but is likely to happen with the inclusion of substantially more samples of non-English origins after filtering (see Figure 2 of main paper).

Deeper description of what the GeoDE task is for unfamiliar readers. Thank you for raising this point. GeoDE is a geographically diverse dataset containing 61K images spanning 40 common object categories, collected from 6 different world regions via crowd-sourcing. We will add more details of this task to the paper.

Where are the examples in Figure 1 from? Are they actually in the datacomp dataset? Yes, these examples are taken from our training data, i.e. DataComp.

Doesn't translating datacomp captions into other languages keep the learned representations in the English concept space? We interpret this question as whether translation would reduce the richness of the concept space (especially when it comes to translating culturally salient concepts). We empirically observe that after translating multilingual captions to English, some culturally salient concepts are still preserved - refer to Figure 1 (left) of the main paper for examples. Our work has also acknowledged this potential limitation, that “translation can sometimes be too literal, subject to losing the intent and richness of the original phrasing”. Nevertheless, we hope that our findings inspire future work to (i) be more inclusive of data of non-English origins during training, (ii) investigate other ways to effectively leverage the diversity of multilingual data.

We thank the reviewer for their time and feedback!

No quantitative assessment for the quality of translation. Given your concern, we have performed additional analysis of the translation quality. We sampled 100K multilingual captions from our raw data pool and backtranslated the English-translated caption into the original language (e.g. Chinese text -> English translation -> Chinese translation of the English-translated text). Then we computed the cosine similarity between the initial web caption and the backtranslated caption using embeddings from the multilingual Sentence-BERT model (Reimers et al., 2019), to assess how much semantic meaning is preserved after translation. We find that on average the cosine similarity (thus, translation quality) remains relatively high (0.63). Below we report the top 5 and bottom 5 languages that observe the highest and lowest translation quality as captured by our metric (each has at least 30 samples). We will add the full analysis in the next version of the paper.

Only evaluation on representation task. Does such data strategy also work for the training of multimodal LLM? We note that our evaluations already contain 38 datasets involving recognition and classification of a wide range of domains (e.g. texture/ traffic sign/ scene/ metastatic tissue/ geolocation/ animal/ etc.), in addition to image-text retrieval and commonsense association.

Extending the findings from this work study to multimodal LLM training would be an interesting direction for future work, but is orthogonal to our contribution; we will add this to the Discussion section. Our results demonstrate that there is so much diversity and visual information to be leveraged from multilingual data, to the extent that training on predominantly non-English samples can improve CLIP’s performance on standard English benchmarks. We hypothesize that leveraging this diversity will also be beneficial for training multimodal LLMs, especially since many of which still rely on English-dominated image-text datasets (e.g. LAION) and still use CLIP as the image encoder (e.g. LLaVA-1.5).