See It from My Perspective: How Language Affects Cultural Bias in Image Understanding

We’d like to thank you for your thoughtful comments. We especially appreciate that you found our work to be comprehensive in its analysis and valuable in its insights.

We’ve included responses to your questions below along with the line numbers of relevant areas of the paper where answers can be found. Unless otherwise specified, all line numbers refer to the updated version of our paper (where newly added content appears in blue).

1. Other languages and cultures

   We agree that we would love to see our work extended to more languages and cultures. While we were able to draw evaluation examples from both the narrow set of English/Chinese speaking countries and the broader set of their geographical Western/East Asian neighbors to establish Western/East Asian bias (see Table 1, lines 254 - 264 of both our original and revised submissions), we are limited in our ability to explore the effects of other languages because of the lack of comparable LLMs trained on equivalent amounts of English/Chinese/other language text. We emphasize here that comparability is key to our analysis as it allows us to make more confident claims about the effects of different language modeling choices on downstream bias in image understanding. With comparable LLMs, we have the computational resources to do multimodal post-training but unfortunately (as an academic group) we do not have the resources to train comparable LLMs in other languages.

   Still, we believe that our experiments leave us with a good sense of what we can expect if our work were extended to other languages and cultures. As Chinese is the most resourced non-Western language online, we suspect the bias (and effects on bias) we measure in our paper should be a floor for bias in other cultures – we should expect bias in those settings to be worse. However, we argue that we needn’t accept this outcome. Multilingual VLMs should be built on LLMs that are trained on a large amount of naturally occurring multilingual text – in particular, model developers should leverage as much of the unimodal text as is available online in languages that are spoken by members of their target culture.

2. Bias sourcing

   As described above, the challenge in isolating the effects of language modeling choices on bias in image understanding is in ensuring that the base LLMs we work with are comparable except in the language mix used in their construction. In selecting Llama2 and Baichuan2, we strove to do just that. We’ve added a section to the Appendix of our revised submission (section A.3.1, lines 902 - 929) that compares the architectures and construction of these two models beyond parameter and token counts. You’ll see that the models are very close with the exception of the vocabulary size of their tokenizers (which were learned to accommodate each model’s specific pre-training corpus, English vs English & Chinese). Given the similarity of these two models (in architecture and pre-training), we believe our work effectively measures the effects of language on culturally informed visual tasks.

We’d like to thank you for your thoughtful comments. We especially appreciate that you found our work to be systematic in both our experimental results and our mechanistic study.

We’ve included responses to your questions below along with the line numbers of relevant areas of the paper where answers can be found. Unless otherwise specified, all line numbers refer to the updated version of our paper (where newly added content appears in blue).

1. Effect of prompting in Chinese in OTS models

   The caption in Figure 4 (lines 378 - 389 of both our original and revised submissions) says “While Western bias reductions are seen across all tasks when prompting in Chinese, they are not consistently seen in every model.” As you say, in some cases (15/30), prompting in Chinese increases Western bias (the purple x appears to the right of the gold circle). However, in the remaining cases (15/30), prompting in Chinese reduces Western bias (the purple x appears to the left of the gold circle). As such, though bias reductions are seen in a few models for every task (3 for DollarStreet, 3 for VQA and 2 for ArtELingo), they are not seen consistently (lines 368 - 371 in our original paper). We’ve updated the caption for Figure 4 (lines 393 in our revised submission) to emphasize this.

2. Prior work on multilingual VLMs

   Thanks for linking these two papers on improving multilingualism in VLMs. Rather than multilingualism, the focus of our work is multiculturalism – when a VLM generates Chinese, is the Chinese it’s generating culturally aligned? For example, when identifying the emotion evoked by a work of art, beyond generating fluent Chinese, does the VLM select the emotion chosen by a majority of Chinese speaking annotators? We visualize this in Figure 1 (lines 54 - 67 in both our original and revised submissions).

   In the first paper, Qiu et al. find that machine translation is an effective way to improve the multilingual performance of VLMs. In contrast, our work demonstrates there are limits to the gains one can expect from machine translation – it does not bridge the cultural gap as effectively as using a base LLM pre-trained on large quantities of naturally occurring text in a culturally relevant language.

   In the second paper, Ramos et al. find that parameter efficient training coupled with retrieval augmentation in a target language can allow a VLM (built with a multilingual base LLM) to transfer more effectively to a target language. In contrast, we focus on the choice of base LLM and its effect on the cultural alignment of multilingual generations.

   We’ve updated our Discussion to include both of these works (lines 534 - 537 of our revised submission).

3. Overall Performance

   We include the overall performance of our mLLaVA variants in Table 4 of the Appendix of the original paper (lines 925 - 971) , Table 5 of the updated version (lines 1087 - 1133). In nearly all of our evaluation settings, performance is above chance (25% for VQA, 12.5% for ArtELingo). The one exception is for the subset of ArtELingo where some models (especially the Llama based variants) perform below chance on the Chinese split of the task (but in these cases, the models are favoring the label the English split, as can be seen in the above chance performance when judged against English labels).

   We’ve updated the Appendix to include the overall performance numbers from our OTS variants (Table 4, lines 1036 - 1072 in the revised version). All models perform above chance with the exception of Blip2-Flan-T5-XXL when prompted in Chinese as the model cannot process or generate fluent Chinese. We’ve also updated the paper to reference this table (lines 365 - 366 in the revised version).

We’d like to thank you for your thoughtful comments. We especially appreciate that you found our work to be novel, well designed/executed and of interest to the broader community.

We’ve included responses to your questions below along with the line numbers of relevant areas of the paper where answers can be found. Unless otherwise specified, all line numbers refer to the updated version of our paper (where newly added content appears in blue).

1. Western bias

   We completely agree that in exploring Western bias and East Asian bias, it is important to evaluate models on examples drawn from countries beyond those that are majority English or Chinese speaking. As such, on both object identification and VQA, we evaluate our off-the-shelf VLMs and our mLLaVA variants on examples from the narrow set of majority English/Chinese speaking countries and a larger set of their geographical neighbors (which we refer to as “broader region”). We feel that this extended subset expands our analysis from English/Chinese bias (where models are biased on examples from English/Chinese speaking countries) to Western/East Asian bias (where models are biased on examples from the West / East Asia). You can find the list of countries that comprise the “broader region” in Figure 3 (original & updated, lines 162 - 181) and Table 1 (original & updated, lines 254 - 264) and you can see the results for the “broader region” subset in Figures 4 (original & updated, lines 378 - 389) and 5 (original & updated, lines 432 - 469).

   Our analysis is focused on understanding the effects of using a culturally closer language (in this case, English and Chinese) during pre-training, multimodal fusion and inference on this Western/East Asian bias. While we could not expand this set of languages to a broader set because of the unavailability of comparable LLMs pre-trained on other languages, our object of study is still Western/East Asian bias as our evaluation examples are drawn from a narrow and broad set of countries from those regions (please see our Ethics Statement, lines 546-553 in both our original submission and our revised submission, where we address this).

2. Objective Tasks and DollarStreet

   We measure bias reductions on objective tasks using two benchmarks, DollarStreet for object identification and A-OKVQA/CVQA for scene understanding. On A-OKVQA/CVQA, prompting in Chinese yields significant reductions in bias (from 1.45x to 1.17x) in our Baichuan2-based VLM which saw large quantities of Chinese during pre-training (Figure 5b, lines 444 - 452 in both our original and revised submissions). Thus, on both objective and subjective tasks, prompting in a culturally closer language does in fact reduce Western bias, especially if it was common during LLM pre-training.

   On DollarStreet, we see (and show, Figure 5b, lines 444 - 452 in both our original and revised submissions) an increase in Western bias from 1.07x to 1.19x. While we include this result in our paper for the sake of completeness, we do not emphasize it in our conclusions as the effect size is much smaller than the reductions in bias we see on the objective VQA and the subjective ArtELingo (a bias increase of 0.11x compared to bias reductions of 0.285x and 1.487x respectively).

3. Factors in VLM Performance

   Thanks for pointing us to this interesting work. We agree that including this result can only strengthen our conclusions – namely that different VLMs have different task-specific strengths that are a function of the genres of data they see during pre-training. Within a given task, bias manifests as disparity on the Western / East Asian splits. Thus, rather than drawing conclusions on Western bias by comparing absolute scores or absolute score differences, we measure relative bias and relative bias differences. We’ve expanded our discussion in lines 239-244 in our revised submission to include this reference.

We’d like to thank you for your thoughtful comments. We especially appreciate that you found the work interesting / relevant and that you think our controlled investigation of the effects of language modeling choices fills a consequential gap in the literature.

We’ve included responses to your questions below along with the line numbers of relevant areas of the paper where answers can be found. Unless otherwise specified, all line numbers refer to the updated version of our paper (where newly added content appears in blue).

1. “Bias Characterization” and “Bias Sourcing”

We’ve expanded the description of the methods in lines 154 - 158 to explicitly introduce Bias Characterization and Bias Sourcing. Additionally, in subsequent subheadings (3.1, 3.2, 4.2, 4.3, 5.2) where we use Bias Characterization and Bias Sourcing, we have additionally specified “Step 1” and “Step 2” respectively to more clearly link the two terms to our Approach figure (Figure 2, lines 108 - 119).

2. Classification vs Open-Generation Evaluation

We evaluate against three classification tasks (object identification, multiple choice VQA and emotion identification) that we believe are reasonable proxies for VLM behavior in open-ended generation settings for several reasons:

• There is broad alignment around using such classification tasks for benchmarking model performance in the vision-language community. For example, in the classification-focused MMLU benchmark (Hendrycks et al., 2021), the authors say “Some researchers have suggested that the future of NLP evaluation should focus on Natural Language Generation (NLG) (Zellers et al., 2020), an idea that reaches back to the Turing Test (Turing, 1950). However, NLG is notoriously difficult to evaluate and lacks a standard metric (Sai et al., 2020). Consequently, we instead create a simple-to-evaluate test that measures classification accuracy on multiple choice questions.”
• Moreover, we believe our chosen tasks test the cultural alignment of vision-language abilities (object identification, scene understanding through VQA, affective analysis through emotion detection) that are core to generating culturally aligned long-form text. In “What Are We Measuring When We Evaluate Large Vision-Language Models?” (Meng Huat Tiong et al., 2024), the authors show that fine-tuning on VQA produces the best transfer to other target vision-language tasks including open-ended generation. This suggests that our evaluation (which includes VQA) is a reasonable proxy for the cultural alignment of general image-conditioned language generation.
• Finally, we evaluate our VLMs on object identification and emotion classification in an open-ended generation setting, using an LLM-as-a-Judge to ensure correctness for the former and using an LLM to extract the nearest emotion for the latter. As such, we believe our results can be extended to other open-ended generation tasks.

While prior work has tackled multilingual captioning benchmarks like XM3600 (which we discuss in lines 156-157 of both our original and revised submissions), cultural differences in image descriptions tend to be subtle (different language speakers emphasize different image elements) and are difficult to differentiate via existing automatic captioning metrics, complicating evaluation. This further motivated our decision to use easy-to-evaluate classification-style benchmarks.