Cultural and Linguistic Diversity Improves Visual Representations

4. “Why are the final conclusions reliable when you have used a machine translation tool to convert the non-English text into English text?”

We run a user study and show that native speakers find that the content of languages is preserved across translation (see Appendix A.2). We ensure that objects, their attributes, and relations between them are preserved. Consider the example of a bird in the background. A caption which mentions a bird will result in a translation which has a bird, and a caption which does not will result in a translation which does not have a bird. Therefore, we believe the signal we are measuring (the presence of objects, attributes, relations) is not likely to be sensitive to artifacts introduced during translation.

5. “What are the actual benefits of the Google Vertex API?”

Google Vertex API represents one popular commercial solution to multilingual image captioning. By "outputs reflect established standards and have gone through quality checks", we simply mean that Google has done extensive testing to ensure the model’s behavior [13] and that it is a worthy benchmark to analyze.

While we were running the experiments, we struggled to find comparable, high-quality, and open-source multilingual image captioning models. Since then, we have gained access to GPT-4V and added several examples in the appendix – see Table 19 in Appendix D.9.

6. “How much money was spent on API calls?”

For the Vertex API, we made [3,600 images × 7 languages × 3 captions each = 75,600] calls and spent [75600 × 0.0015 = 113.4] for CrossModal images; and made [2,000 images × 7 languages × 3 captions each = 42,000] calls and [42,000 × 0.0015 = 63.] for Visual Genome images. Translation costs with GPT-4 total approximately $597 to translate over 260k captions from non-English languages to English. We will also release all of our datasets upon acceptance for reproducibility.

We are grateful to the reviewer for taking the time to provide valuable feedback on our work. We address below the questions raised in the review:

1. “What efforts did you make to conduct your analysis in the original languages?”

The goal of our paper is to evaluate differences in caption content. We use a range of metrics to measure qualities of content. Some of these are implemented only in English. For instance, the LIWC metrics [1] are hand-crafted by linguists and therefore difficult to reproduce in other languages. In these cases, it is not possible to use non-English data. Other metrics have multilingual implementations, but they are not language agnostic. For instance, even advanced multilingual embeddings have been shown to encode language-specific information, especially typology [2, 3, 4, 5]. Implementations of linguistic parsers across languages vary widely in implementation and performance because their design depends on features of that language [6, 7]. Neural-based approaches for parsing multiple languages with one model usually rely on translation anyway [8]. In sum, we found analysis in original languages to be technically infeasible and introducing too many confounders, compared to translating to English.

2. “What are the main types of differences driving the differences in different non-English translated text?”

We agree that the examples in Table 16 on ground truth coverage in Visual Genome are not so strong. We have since updated that figure with several more representative examples. We do suggest looking at Figure 2, which gives some examples of English and non-English captions and demonstrates that captions in different languages are mentioning different objects.

It was hard for us to empirically validate clear, human-interpretable patterns between languages, although we do have many hypotheses we are testing for future work. For instance, some languages seem to mention objects in the background with greater frequency or mention certain types of attributes more often, which appears to align with existing work in cross-cultural psychology. See additional examples we have added in Tables 16, 19, 20 for more samples.

The main contribution of our work is to show substantive differences between the content of captions in different languages across many metrics, and we hope future work can investigate explanations.

3. “Why did you use the GPT-4 API instead of a dedicate machine translation model, such as M2M-100 or NLLB?”

M2M-100 and NLLB are both good models, but we found that we can specify to GPT-4 that we want our translations to preserve all factual details and to be exact. Machine translation models can optimize over many metrics, including factual correctness, but also including style, tone, coherence, and fluency, which can be in conflict with factual correctness [9]. For our purposes, factual correctness takes ultimate precedence over other factors.

Although GPT-4 was released recently and we could not find published evaluations of GPT-4 on translation directly, several works have demonstrated GPT-4’s competence and even superior performance in some cases on multilingual tasks [10, 11, 12].

In response to each of the weaknesses raised:

1. Conducting analysis only in English.

We believe that conducting our analysis in English is actually a strength. Suppose running analysis entirely in captions’ original languages were possible (which, as discussed in response to Question 1, it is not). Even then, a strong objection to any results we might produce is that language acts as a confounder. For instance, in our fine-tuning experiments, it might be that observed differences are simply due to differences in pretraining material, data availability, tokenization procedures, etc.

Given that we have measured that relevant content (objects, attributes, relations) are preserved over translation (response to Question 4), conducting the analysis in English allows us to make a strong claim about content differences in captions from different languages.

2. Weak examples of differences between English and non-English.

In addition to Table 2, we have added more examples of English and non-English captions in Table 20 (Appendix D.10). Additionally, as mentioned before, we have added GPT-4 examples in Table 19, Appendix D.9.

3. Not discussing previous work in multilingual multimodal modeling.

We appreciate your recommended literature on multilingual multimodal research. We have added these as references in the paper. However, we believe that the suggested work is complementary yet ultimately different from our contributions.

The suggested previous work in multilingual vision covered by [14] has focused on expanding vision capabilities from monolingual (English) contexts outwards to other languages (“getting models in different languages to see the same things”). We investigate how vision models built from different languages might behave differently (“observing how models in different languages see different things”).

To this end, the suggested VaTeX [15] and WIT [16] datasets are helpful, and be good bases for future work. The suggested Liu & Bugliarello et al. paper [17] focus more on differences in knowledge of cultural objects and practices, e.g., dresses or foods particular to some culture. While this is parallel to our work in that it also studies differences in model behavior across human groups, we focus on investigating differences in perception of generally universal objects, e.g. birds, trees, and cars.

If interested, we further discuss our unique contributions in Appendix A.1 (Discussion).

We hope this helps clarify! Please let us know if there are any other comments or questions which may arise. If we addressed your concerns, we would appreciate an increase in your score.

– the authors

We are grateful to the reviewer for taking the time to provide valuable feedback on our work. We address below the questions raised in the review:

1. “Regarding the diversity of the scene graph, could issues in normalization give the impression that multilingual descriptions contain more concepts?”

This is certainly a possibility, and we appreciate the reviewer for bringing it up. Concept alignment will always be a messy process. As such, we designed our canonicalization process to err on the side of grouping two concepts together as opposed to apart. In our canonicalization process, we first evaluate WordNet similarity (the commonly used procedure). If it fails, we evaluate similarity of embeddings. If even this fails, we look towards other associated information (e.g. attributes and relations from objects) to make a match if possible, with even lower thresholds for similarity. Therefore, we believe that our results stand despite, rather than in part because of, irreducible error in concept alignment.

2. “Could you address the possibility that the observed differences in your study are simply artifacts of the translation process?”

It is true that translation introduces some minimal amount of error or diversity into text. However, we are principally concerned with the content (objects, attributes, relations) of captions, which we confirmed in our human studies are preserved across translation. Consider the example of a bird in the background. A caption which mentions a bird will result in a translation which has a bird, and a caption which does not will result in a translation which does not have a bird. Therefore, we believe the signal we are measuring (the presence of objects, attributes, relations) is not likely to be sensitive to artifacts introduced during translation.

It is possible that translation artifacts may have a larger effect on the representational and linguistic diversity measures, which evaluate over raw text rather than scene graphs, which are parsed for content. However, observe from Tables 4 and 5 that the diversity metrics across all 7 languages increases substantially over the diversity metrics from groups of 3 languages. If the text was more or less the same and translation was introducing most of the diversity, we would not expect as much of an increase. Likewise, observe that the diversity metrics for groups of 3 non-English (i.e. all translated) languages are generally similar to the diversity metrics for groups of 1 English and 2 non-English languages. All of this is to suggest that although there may be an irreducible translation error, we have good reason to believe it does not significantly affect our diversity metrics.

In response to each of the weaknesses/disappointments raised:

1. Not enough focus on cross-cultural differences

We agree with the reviewer. Work in psychology demonstrating cross-cultural differences in perception (e.g., [1]) were the principal motivation for our work. Many of these studies consider language as a dimension across which to study cross-cultural differences. Ultimately, we found that it is much more difficult to track “culture” in an ML context than in a closed, experimental psychology context. One reason why is because we often don’t have information about the cultural background of someone who wrote an image caption. Therefore, the focus of our paper is to demonstrate content differences across languages in general across different metrics. Our hope is that this will motivate future work which explicitly investigates cross-cultural differences in vision-language models.

2. "Linguistic diversity" doesn't seem to have all that much to do with linguistic information: it's more about psychological attributes.

This may have been a communication issue on our part. The differences which we observe arise both from culture and language. Culture affects our experiences and the importance-judgements we make in perceiving the world. Language may also affect our perception in certain ways (see [5] and discussion in Appendix A.1, “Linguistic Structure and Information”), but more substantively, language is a structure which incentivizes the expression of some information and disincentivizes others. For instance, German’s complex morphosyntactic system provides events with nuanced understanding of spatial relationships [6].

As documented by anthropologists and linguists [7, 8], it is difficult to disentangle the effects of culture and language. As such, we use language as our independent variable but study both culture and language effects indiscriminately (hence “cultural and linguistic diversity”). This approach of studying culture and language jointly has a precedent in various social sciences fields, e.g.. in [9].

Note: The LIWC metrics we used are technically psychometric indicators, but importantly for us they provide linguistic information about how sentences are being structured, what information is being included or excluded, etc. See [10, 11] which use LIWC to investigate language formation styles.

We hope this helps clarify! Please let us know if there are any other comments or questions which may arise.

– the authors

References

[1] Masuda, T., & Nisbett, R. E. (2001). Attending holistically versus analytically: comparing the context sensitivity of Japanese and Americans. Journal of personality and social psychology, 81(5), 922–934. https://doi.org/10.1037//0022-3514.81.5.922

[2] Merrielle Spain and Pietro Perona. Some objects are more equal than others: Measuring and predicting importance. In European Conference on Computer Vision, 2008.

[3] Alexander C. Berg, Tamara L. Berg, Hal Daume, Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa C. Mensch, Margaret Mitchell, Aneesh Sood, Karl Stratos, and Kota Yamaguchi. Understanding and predicting importance in images. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3562–3569, 2012.

[4] Chua, H. F., Boland, J. E., & Nisbett, R. E. (2005). Cultural variation in eye movements during scene perception. Proceedings of the National Academy of Sciences of the United States of America, 102(35), 12629–12633. https://doi.org/10.1073/pnas.0506162102

[5] Lera Boroditsky. Linguistic relativity. Encyclopedia of cognitive science, 2006.

[6] Jakob Prange and Nathan Schneider. Draw mir a sheep: A supersense-based analysis of german case and adposition semantics. KI-Kunstliche Intelligenz , 35(3-4):291–306, 2021.

[7] Goldstein, L. J. (1957). On Defining Culture. American Anthropologist, 59(6), 1075–1081. http://www.jstor.org/stable/666466

[8] Lazear, E. P. (1999). Culture and Language. Journal of Political Economy, 107(S6), S95–S126. https://doi.org/10.1086/250105

[9] Laesser, Christian & Beritelli, Pietro & Heer, Samuel. (2014). Different native languages as proxy for cultural differences in travel behaviour: Insights from multilingual Switzerland. International Journal of Culture. 8. 10.1108/IJCTHR-02-2014-0010.

[10] Qiu, L., Lin, H., Ramsay, J.E., & Yang, F. (2012). You are what you tweet: Personality expression and perception on Twitter. Journal of Research in Personality, 46, 710-718.

[11] Ireland, M., & Pennebaker, J.W. (2010). Language style matching in writing: synchrony in essays, correspondence, and poetry. Journal of personality and social psychology, 99 3, 549-71.

We thank the reviewer for taking time to provide valuable feedback on our work and are pleased that they found our work interesting.

The reviewer makes the insightful observation that perception might be constant while what humans choose to describe might vary, and raises the question of how we use the term “visual perception”. We follow from previous work in psychology and computer vision on object salience and attention in defining what is perceived as what the subject later describes. For instance, the important work by Matusda and Nisbett (2001) [1] notably compares American and Japanese subjects’ perception by asking them to describe what they saw after shown the scene for several images. Other examples in which human documentation is taken to indicate perception and salience include [2, 3].

It is possible to measure “low-level” perception or attention with eye/gaze-tracking [4], but perception also involves importance/salience-judgements which are more important to study for the purpose of computer vision. If someone’s eyes passed over some object but they didn’t register it enough to mention it when asked to describe what was in the image, did they really perceive it? For the purposes of our paper, we assume the answer is no.

To clarify, we are not making claims about what people see or their gaze patterns. Instead, we are suggesting that the objects, attributes, and relations people perceive (in the sense of registering and finding salient to mention) differ across the language they speak. Indeed, cultural and linguistic structures/norms may play a role in affecting this process, which we discuss in “Linguistic Structure and Information” in Appendix A.1.