No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models

We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate their positive assessment of the importance and presentation of our work. We provide our answers to the reviewer’s raised questions and concerns below, are looking forward to their response, and hope for a favorable reassessment of our scores.

Q1: Shouldn’t it be expected that the globe models outperform the en model on benchmarks measuring global understanding?

While we agree that this might seem intuitive at first sight, it is not necessarily true. There are English websites published in most countries and pictures of landmarks are frequently captured by tourists, so it is possible that the English subset of the web is sufficient to capture cultural diversity. Our contribution is to provide a comprehensive evaluation of cultural diversity in SOTA contrastive VLMs, and we hope that these results suffice to clear any lingering doubts about the necessity of training on global data when building foundation models. Please refer to our global response for more detailed information.

Q2: Are the findings related to prior work on multilingual understanding?

We thank the reviewer for the additional pointers to prior work on multilingual understanding and will make sure to include these references in the revised version of our paper. In the context of this work, however, we purposely disentangled multilinguality from cultural diversity because we believe these measure different aspects. As noted in Section 5, we do, however, see the investigation of their intersection as an intriguing subject for future research and are looking forward to new insights in that area!

Q3: Do the findings hold for other contrastive VLMs beyond SigLIP (e.g., CLIP)?

SigLIP and CLIP operate on the same principle of aligning representations/embeddings for texts and images and the difference is only in the choice of the loss function. We expect the same results to hold for CLIP as well. The reason we use SigLIP is because of its superior performance and widespread use (>1M downloads/month in Huggingface).

Q4: How does additional, quality-based data filtering affect the findings presented in this work?

Thank you for raising this question. We have conducted new experiments to verify this. Please refer to the global response for a summary of our findings. Generally, we observe that the same conclusions hold even after applying quality filters based on image-text similarity. We will add more details about this in the revised version of the paper.

Q5: Are the findings limited to geographic diversity?

While geographic and socioeconomic diversity are inherently linked due to different income levels and economic circumstances in countries and regions around the world, Dollar Street does contain data from different income groups for every region. Kindly refer to Figure 3 (left), in which we specifically address these differences.

Q6: Does the proposed removal of the English-only data filter increase computational cost?

We believe our findings are orthogonal to having additional, quality-based data filters that can be applied in practice to limit computational cost incurred during training. In particular, while we do not challenge the use of quality filters, our work warns against using those filters that are based on English-only datasets or favor English-only captions. Please refer to the global response for a more detailed response.

We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate their positive assessment of the originality, quality, significance, and clarity of our work. We hope that our rebuttal below addresses all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.

Q1: How does additional, quality-based data filtering affect the findings presented in this work?

Thank you for raising this question. We have conducted new experiments to verify this. Please refer to the global response for a summary of our findings. Generally, we observe that the same conclusions hold even after applying quality filters based on image-text similarity. We will add more details about this in the revised version of the paper.

Q2: Does the proposed removal of the English-only data filter increase computational cost?

We believe our findings are orthogonal to having additional, quality-based data filters that can be applied in practice to limit computational cost incurred during training. In particular, while we do not challenge the use of quality filters, our work warns against using filters that are based on English-only datasets or favor English-only captions. Please refer to the global response for a more detailed response.

Q3: Size of WebLI

As is detailed in Section 2 of our paper, all models are trained on 10B image–text pairs (roughly 610k training steps). Hence, all models are compared on a compute-matched setup: en models are trained for ~3 epochs, while globe models are trained for a single epoch. Please refer to the global response for more details.

Q4: What evidence supports the claim that English-only data filtering exacerbates existing disparities?

We show examples of this in Figure 1.c and Figure 3 (left). For instance, accuracy on Dollar Street for low-income groups ($0-200) drops from 35% using globe-tl to less than 29.9% when trained on English-only data. Similarly, accuracy on examples from the African continent drops from 38.4% using globe-tl to less than 35.8% using English-only data. Because of this, pretraining on English-only exacerbates existing disparities because there is already a large gap in performance between low-income and high-income regions (as shown in the figures) and training on English-only data increases those gaps.

We also thank the reviewer for the comment regarding the commas and will make sure to carefully check this for future revisions of our paper.

We would like to thank the reviewer for the careful and insightful feedback. We provide our answers to the reviewer’s raised questions and concerns below, are looking forward to their response, and hope for a favorable reassessment of our scores.

Q1: How is the paper structured?

We are sorry to hear that the reviewer found the structure of our paper to be unclear. We motivate and present a brief overview of our work in Section 1, before moving on to discussing the chosen model architecture as well as training and evaluation datasets in Section 2. We also briefly summarize our key findings at the end of Section 2 and explain them in more detail in Section 3. We conclude with a discussion of related work, limitations and future work. If the reviewer has any specific suggestions for improving clarity, we would be happy to hear them and incorporate them in the revised version of the paper.

Q2: How does the proposed geo-localization task compare to alternative metrics of cultural diversity in contrastive VLMs?

As can be seen in Figure 5 (right), the proposed geo-localization task is well correlated with other metrics of cultural diversity in contrastive VLMs (such as zero-shot classification on Dollar Street, GLDv2, GeoDE and MaRVL). One important distinction, however, is that the few-shot geo-localization task only evaluates the learned image embeddings while discarding the text tower, which can provide insightful additional information. In addition, it provides a very strong signal as shown in Table 2, where differences in accuracy can reach up to 10%.

Q3: How is socioeconomic diversity defined and measured in the context of this work?

As noted in Section 5, we do not claim to offer a precise definition of cultural or socioeconomic diversity in the context of VLMs. Instead, we use model performance across different countries, regions, and income groups as a useful proxy for these concepts. We acknowledge in Section 5 (limitations) that this may not cover all aspects of cultural and socioeconomic differences.

We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate their positive assessment of the thoroughness, importance and practicality of our work. We hope that our rebuttal below addresses all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.

Q1: How much does training on en instead of globe increase the performance gap between groups of high and low socioeconomic status?

We show examples of this in Figure 1.c and Figure 3 (left). For instance, zero-shot classification accuracy on Dollar Street for low-income groups ($0-200) drops from 35% with globe-tl to less than 29.9% with en. Similarly, accuracy on examples from the African continent drops from 38.4% using globe-tl to less than 35.8% using en. Because of this, pretraining on English-only data exacerbates existing disparities because there is already a large gap in performance between low-income and high-income groups (as shown in the figures) and training on English-only data increases these performance gaps.

Q2: How does additional, quality-based data filtering affect the findings presented in this work?

Thank you for raising this question. We have conducted new experiments to verify this. Please refer to the global response for a summary of our findings. Generally, we observe that the same conclusions hold even after applying quality filters based on image-text similarity. We will add more details about this in the revised version of the paper.

Q3: Shouldn’t it be expected that the globe models outperform the en model on benchmarks measuring global understanding?

Not necessarily. Please refer to the global response for our answer to this question.

Q4: How and for how long are the models trained?

As is detailed in Section 2 of our paper, all models are trained on 10B image-text pairs (roughly 610k training steps). Hence, all models are compared on a compute-matched setup: en models are trained for ~3 epochs, while globe models are trained for a single epoch. Please refer to the global response for more details.

Q5: Can the trade-off between good performance on standard benchmarks and maintaining cultural diversity be avoided?

While pre-training on global data and fine-tuning on English data allows to balance performance on standard and culturally diverse evaluation metrics, neither fine-tuning nor data mixing allow for complete avoidance of the tradeoff between these metrics (as can be seen in Figure 5 (left)). It is however possible that other approaches (such as different training paradigms or model weight merging) might lead to improved tradeoffs and maybe eventually even a complete avoidance thereof. We are looking forward to future work in this area!