First of all, we want to thank you for taking the time to review our paper and share your opinion about it.

Weaknesses

Analyses on multi-modal layers – More comprehensive investigation into intermediate layers is indeed an interesting avenue to explore. This would especially be interesting for dual-stream models that learn additional multi-modal layers between the text and vision modality (e.g. BLIP2) or single-stream models like VILT. However, the results of Verhoef et al. 2024 indicate that these models do not display behaviour consistent with a bouba-kiki effect. We, therefore, do not explore these models in this work and focus on CLIP.

Since CLIP has been trained to map textual and visual inputs into a single joint multi-modal embedding, it does not have intermediate multi-modal layers that must explicitly align text and vision. In an attempt to better understand why we observe misalignment between human and CLIP’s behaviour, we explored whether the individual modalities are in principle separable in Appendix A & B and found that this seems to be the case. This indicates that CLIP’s method to align texts and images (cosine similarity) seems to be the culprit (please also see our response to your first question below). Nevertheless, your suggestion of investigating intermediate layers in dual-stream models is an exciting idea for future work.

Limited significance – We argue that the significance is not limited. First of all, our Grad-CAM method is novel (which you identified as a strength of this paper) and goes beyond confirming inconsistencies by showing that CLIP does not attend to human-like image features (Figure 1 and Figure 10). Secondly, our work nuances the findings by Alper and Averbuch-Elor (2023), who make claims that require more substantiated argumentation. Our work offers contrary findings and accompanying argumentation that we see as a significant message for the community. Finally, we do address the surprising finding that Japanese VLMs do not exhibit expected bouba-kiki effects in lines 140-145. We draw on the findings by Iida and Funakura (2024) and conclude that investigating sound-symbolism in VLMs requires methodologies aligned with human experiments.

Clarification line 333 – Thank you for this comment. Our phrasing should have been more concise here. The ViT variant indeed performs better on English adjectives, but crucially does not do so for the targets of interest: the pseudowords. The English adjectives merely serve as a baseline (line 187) to inform us whether the models can, in principle, recognise the relevant visual features and map them to meaningful words (which they can). The claim in line 333 corresponds to the performance on the pseudowords; we will make this clear in the revised version of this paper.

Questions

Reflection on misalignment – This is a very interesting suggestion. We think that there are two reasons for the observed discrepancy. First, CLIP’s training data indeed does not explicitly reflect symbolic or phonetic associations. Nevertheless, human language and its orthography are not arbitrary either (see also our response to reviewer xGW7). Characters that represent rounded sounds (like rounded vowels or sonorant consonants) tend to have more curves (Koriat, 1977;Turoman & Styles, 2017). Since these sounds are related to shape features, CLIP’s training data could (but it doesn’t) still provide enough information to learn human-like cross-modal associations. Secondly, CLIP combines textual and visual information through cosine similarity and is trained with a contrastive loss at the sentence level. Neither seems to specifically promote learning relations between visual features and phonetic elements in language. Interestingly, however, as mentioned at line 247, the individual modalities seem to be separable in CLIP’s embedding space. We therefore expect that finetuning on (parts of) targeted datasets could reveal human-like associations. However, the bouba-kiki effect is merely one of many cross-modal associations present in human cognition. Instead of finetuning for each specific human-like bias, research towards a more fundamental approach that combines multi-modal information more naturally seems more desirable.

Tokenisation – We want to thank you for your question and your argument that tokenisation oftentimes keeps syllabic structures intact. This increases its relevance and need for clarification.

Interestingly, analyses reveal that humans even display a bouba-kiki effect at the level of single syllables (Nielsen and Rendall, 2013), suggesting that the associations do not rely on complete words. Since, as you mention, tokenisation often preserves syllabic structures, our analysis is not so different from investigating the bouba-kiki associations in humans.

To add strength to this argument, we tokenised the most common labels mentioned in our answer to your first question. Some examples:

Bouba → “bou” and “ba”

Kiki → “kiki”

Takete → “take”, “te”

Maluma → “mal”, “uma”

Xehaxe → “xe”, “ha”, “xe”

Lohlah → “loh”, “lah”

Sepise → “sep”, “ise”

Kaykuh → “kay”, “ku”, “h”

Loomoh → “loom”, “oh”

Mohmah → “moh”, “mah”

These examples, which we will add to an updated version of our paper, show that most of the target words are tokenised in a way that preserves at least some phonological structures that could be matched to shape features.

Nonetheless, the tokenisation process in language models may split syllables or pseudowords into tokens that would not necessarily evoke the expected cross-modal associations in humans either (e.g., a separate evaluation of ‘H’ in OH may invite a jagged association instead of a curved one). If it is indeed the case that the tokenisation process breaks a pseudoword into syllables that would also show no bouba-kiki effects in humans, we would argue that the set of possible other pseudowords is large enough to contain other pseudowords that could invoke the cross-modal associations. If there would truly be a model preference, another pseudoword (which is tokenised into complete syllables) would then be preferred over the pseudoword that is broken up into non-syllables. Yet, our results show that this is not the case.

To answer your question: While tokenisation may have an impact on the results, we think that the extent to which this influences them is somewhat limited. We did, therefore, not discuss this in detail in our paper, but we will do so as a response to your rightful question.

References

Koriat, A. (1977). The symbolic implications of vowels and of their orthographic representations in two natural languages. Journal of Psycholinguistic Research, 6(2), 93–104.

Nora Turoman and Suzy J Styles. 2017. Glyph guessing for ‘oo’and ‘ee’: Spatial frequency information in sound symbolic matching for ancient and unfamiliar scripts. Royal Society open science, 4(9):170882.

First of all, we want to thank you for your feedback and are delighted to see that you are positive about our paper. Please find our justifications for your questions below.

Weaknesses

Grad-CAM for ViT – It is indeed correct that Grad-CAM was originally introduced for CNNs. However, recent works (Chefer et al. 2021, Mamooler, 2021) have shown that it can be adapted to transformer-based models by treating attention maps as similar to convolutional feature maps. In our paper, following the approach in these previous works, we applied Grad-CAM to the last attention layer of the ViT variant of CLIP, which contains spatial information through its attention heads. We specifically focused on the attention from the [CLS] token to the image patches (like in Caron et al. (2021)). To identify which image regions contributed most to the decision, we computed the gradients of the similarity score between the image and text embeddings (which we used as the target for classification) with respect to the attention weights. Then, we averaged the attention maps across heads, extracted the attention weights from the [CLS] token to the image patches by removing the [CLS] column, and reshaped the resulting vector into a 2D spatial grid. We hope that this justification is satisfactory.

Audio-Vision methodology – This is a very legitimate question. We agree with you that sound-vision entanglement is underexplored in our work and that it is a very interesting future direction. There are, however, some points that should be noted. While it is widely assumed that the bouba-kiki effect in humans is largely driven by sound-shape correspondences, many prior experiments with humans actually provide both an acoustic and written form for tested pseudowords (which was also the case in the study by Nielsen and Rendall (2013) which formed the basis for one of our word sets), so many prior studies cannot clearly disentangle the roles of orthography versus sound in the bouba-kiki effect. One study by Cuskley et al. (2017) explicitly explored the role of orthography and showed that auditory presentation cannot eliminate potential effects of orthography in literate human participants, since characters are typically strongly associated with a particular speech sound. Moreover, these associations are often not arbitrary but also shaped by human word-shape biases, resulting in systems where characters that represent rounded sounds (like rounded vowels or sonorant consonants) tend to have more curves (Koriat, 1977;Turoman & Styles, 2017). Correspondences between vision and spoken words are therefore expected to be highly correlated with correspondences between vision and written form. This makes it unlikely that a model trained on spoken language would be more likely to exhibit the bouba-kiki effect than a model trained on text would.
Given the strong relationship between sound and wordforms and the fact that these words cannot be reliably associated with images, it would be surprising if an audio-visual model would pick this up. Despite our scepticism, we definitely think that this is worth exploring further with models that differ in how they combine multiple modalities.

All in all, we believe that we can make a fair comparison. We will elaborate further on the dichotomy between human sound-shape associations and the current study, focusing on text-shape associations, in the limitations section of our updated paper.

Questions

We believe that your questions are addressed in the response above, which addresses the weaknesses you identified. We are happy to answer any additional questions that may arise.

References

Chefer, H., Gur, S., & Wolf, L. (2021). Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 397-406).

Mamooler, S. (2021). CLIP explainability. Github repository

Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9650-9660).

Koriat, A. (1977). The symbolic implications of vowels and of their orthographic representations in two natural languages. Journal of Psycholinguistic Research, 6(2), 93–104.

Nora Turoman and Suzy J Styles. 2017. Glyph guessing for ‘oo’and ‘ee’: Spatial frequency information in sound symbolic matching for ancient and unfamiliar scripts. Royal Society open science, 4(9):170882.

First of all, we wish to thank you for the positive feedback. It gives us joy to read that you found our work interesting.

Weaknesses

Sound-vision entanglement – We agree with you that sound-vision entanglement is underexplored in our work and that it is a very interesting future direction. There are, however, some points that should be noted. While it is widely assumed that the bouba-kiki effect in humans is largely driven by sound-shape correspondences, many prior experiments with humans actually provide both an acoustic and written form for tested pseudowords (which was also the case in the study by Nielsen and Rendall (2013) which formed the basis for one of our word sets), so many prior studies cannot clearly disentangle the roles of orthography versus sound in the bouba-kiki effect. One study by Cuskley et al. (2017) explicitly explored the role of orthography and showed that auditory presentation cannot eliminate potential effects of orthography in literate human participants, since characters are typically strongly associated with a particular speech sound. Moreover, these associations are often not arbitrary but also shaped by human word-shape biases, resulting in systems where characters that represent rounded sounds (like rounded vowels or sonorant consonants) tend to have more curves (Koriat, 1977;Turoman & Styles, 2017). Correspondences between vision and spoken words are therefore expected to be highly correlated with correspondences between vision and written form. This makes it unlikely that a model trained on spoken language would be more likely to exhibit the bouba-kiki effect than a model trained on text would.
Given the strong relationship between sound and wordforms and the fact that these words cannot be reliably associated with images, it would be surprising if an audio-visual model would pick this up. Despite our scepticism, we definitely think that this is worth exploring further with models that differ in how they combine multiple modalities.

Limited model scope – We indeed only test two CLIP variants for two reasons. First, CLIP is a foundational model that is often used in more contemporary models (lines 176-178). If this model does not show human-like associations, it is difficult to imagine that further-tuned models do show human-like preferences. Second, Verhoef et al. (2024) tested CLIP, VILT, BLIP-2, and GPT-4o in a similar vein to the first experiment in our paper. Out of these models, only CLIP and GPT-4o showed limited evidence of a bouba-kiki effect. Given the fact that we extend the existing approaches with an interpretability-based methodology to unravel the attention preferences of VLMs, we focus only on CLIP here because GPT-4o does not provide the required openness for this method. It goes without saying that newer methods are released as we speak (e.g., the pointing behaviour of AI2’s Molmo—which also uses CLIP as a visual backbone), which require further investigation.

Incremental contribution – Since the topic of our paper is relatively new in this community (introduced by Alper and Averbuch-Elor at NeurIPS 2023), we would argue that it is still maturing, and we believe our contributions are more than incremental. This is also corroborated by reviewer 74nD, who states that our work differentiates itself from prior work. Given the strengths you identified (points 2 & 3), it seems to us that you agree with this. Finally, our methodology is strongly aligned with humans and can inspire others to align their methods with human experiments as well.

Questions

Label collapse – This is super interesting, and we did indeed explore this in Appendix C. Table 2 shows that both CLIP versions tend to pick some words more often than others. However, out of 34 trials (images), the average ratio of ±33% across ten prompts indicates that around 10 different pseudowords are picked across all the images. As such, it seems like there is a preference for some pseudowords but no such thing as label collapse.

As a response to your question, we also looked into the most frequently chosen pseudowords and provide some examples. For ViT and Alper et al’s pseudowords, some of the labels that appear most are “manuma”, “kiteki”, “gologo”, “xehaxe”, “lunulu”, “mogumo”. For ResNet these were: “xekaxe”, “sepise”, “xesixe”, “xehexe”, “gologo”. In the case of ResNet and Nielsen et al. pseudowords: “lahloh”, “mohmah”, “kaykuh”, “mohmoh”, “puhkuh”. And for ViT: “teepee”, “lohlah”, loomoh”, “lohlah”, “looloh”, “loomah”. The updated version of our paper will include this extra information.

Spatial distribution of Grad-CAM – Thank you for the suggestion. This is indeed something we explored but did not mention in the paper since we argue that the sum of attention aligns with the holistic view humans have (lines 272-274). Nonetheless, we also used different spatial distributions, such as the centroid of attention and the entropy method you mentioned.

Comparing the predictions chosen by the centroid of attention with the sum of attention, we find that 86.8% of the predictions are the same. Given this large overlap, the results likely would not change if we used this alternative method.

In the case of entropy as a source of prediction (where the image half with the lower entropy acts as the model’s preference), we find that only 50.6% overlaps with the predictions based on the sum of attention. Yet, this method results in a lower number of correctly predicted images, with only 50.7% correctly identified shapes as opposed to 55.4% for the method which uses the sum of attention. So if we were to use entropy-based predictions, the performance would be even worse than predictions based on the sum of attention.

Tokenisation – This is an interesting point that requires further clarification since it is not immediately clear whether tokenisation hurts or benefits bouba-kiki-like associations (see also the question of reviewer 74nD).

Interestingly, analyses reveal that humans even display a bouba-kiki effect at the level of single syllables (Nielsen and Rendall, 2013), suggesting that the associations do not rely on complete words. Since tokenisation often preserves syllabic structures, our analysis is not so different from investigating the bouba-kiki associations in humans.

To add strength to this argument, we tokenised the most common labels mentioned in our answer to your first question. Some examples:

Bouba → “bou” and “ba”

Kiki → “kiki”

Takete → “take”, “te”

Maluma → “mal”, “uma”

Xehaxe → “xe”, “ha”, “xe”

Lohlah → “loh”, “lah”

Sepise → “sep”, “ise”

Kaykuh → “kay”, “ku”, “h”

Loomoh → “loom”, “oh”

Mohmah → “moh”, “mah”

These examples, which we will add to an updated version of our paper, show that most of the target words are tokenised in a way that preserves at least some phonological structures that could be matched to shape features.

Nonetheless, the tokenisation process in language models may split syllables or pseudowords into tokens that would not necessarily evoke the expected cross-modal associations in humans either (e.g., a separate evaluation of ‘H’ in OH may invite a jagged association instead of a curved one). If it is indeed the case that the tokenisation process breaks a pseudoword into syllables that would also show no bouba-kiki effects in humans, we would argue that the set of possible other pseudowords is large enough to contain other pseudowords that could invoke the cross-modal associations. If there would truly be a model preference, another pseudoword (which is tokenised into complete syllables) would then be preferred over the pseudoword that is broken up into non-syllables. Yet, our results show that this is not the case.

References

Koriat, A. (1977). The symbolic implications of vowels and of their orthographic representations in two natural languages. Journal of Psycholinguistic Research, 6(2), 93–104.

Nora Turoman and Suzy J Styles. 2017. Glyph guessing for ‘oo’and ‘ee’: Spatial frequency information in sound symbolic matching for ancient and unfamiliar scripts. Royal Society open science, 4(9):170882.

First of all, we would like to thank you for the feedback. We are happy that you had only a few weaknesses/questions, that the paper was easy to follow, and that you enjoyed the overview of the literature. Please find our response to the weaknesses and questions below.

Weaknesses

Contribution to machine learning models – We acknowledge that our work is primarily in the domain of vision and natural language. Nevertheless, there are several reasons why we think it is as interesting and relevant for the NeurIPS community as for venues like EMNLP or ACL. Our work fits squarely within several themes listed in the CfP, such as ‘deep learning’, ‘neuroscience and cognitive science’. Moreover, the NeurIPS community oftentimes advances deep learning techniques (generative AI, foundation models, LLMs, etc.). Our work reveals an important shortcoming in current approaches, which will hinder the continued development of models that aim to align human and computer visio-linguistic, and more generally multimodal, processing. This is arguably equally important as introducing new methods, as evidenced by the fact that Alper and Averbuch-Elor (2023)—whose conclusions appeared to have been drawn somewhat prematurely, given our new findings—also published their findings (spotlight) at NeurIPS. Finally, as also pointed out as a strength by reviewer xGW7, our work evaluates the same problem via different methods to provide a more comprehensive and nuanced picture. Such practices are, in our opinion, of value to the NeurIPS community as we believe that our findings not only could inspire others to develop models that take cross-modal associations into account, but also introduce several ways to test for such associations in a human-aligned manner.

CLIP as tested model – There seems to be a misunderstanding. You are completely correct that alignment in CLIP is calculated at the sentence level. As spelled out in lines 202-204 and shown in table 1 (appendix A), we indeed embed the pseudowords into sentences (10 versions) such that this alignment score can be calculated at the sentence level. Importantly, for each image, we only differ the pseudoword in question, so variation in probability must be a result of the pseudoword. We hope that this alleviates your concerns about the use of CLIP.

Questions:

Contribution and value of this work – Please see our answer as a response to weakness 1.

Practical impact – The importance of shared perception/experiences/assumptions is often overlooked in the machine learning community (see also reviewer ZTfi). The bouba-kiki effect, or cross-modal associations in general, is one example for which we would like there to be alignment between humans and models since this results in a more intuitive understanding of each other’s prior expectations and assumptions. A lack thereof may hinder natural interactions between humans and machines (Verhoef et al. 2024), especially when considering the dynamic and adaptive nature of human language. Training models in a way that allows them to develop human-like prior expectations or instilling models with them may help alleviate this and could even improve learning efficiency (Lake et al., 2017). The practical impact of our work will be incorporated in the updated version of our paper.

References

Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ. Building machines that learn and think like people. Behavioral and Brain Sciences. 2017;40:e253. doi:10.1017/S0140525X16001837