IMP: Benchmarking Image Polysemy in Vision-Language Models

We thank the reviewer $\textcolor{purple}{R5}$ for insightful comments and valuable suggestions on the subjectiveness of IMP.

Task difficulty and false negatives

We thank the reviewer $\textcolor{purple}{R5}$ for starting the discussion on the difficulty of the task and false negatives as this goes directly to the core of the issue we are studying in this paper. Indeed, considering image polysemy amplifies the difficulties concerning evaluation, yet we argue that when dealing with real-world examples these difficulties are unavoidable.

Increasingly datasets are collected by webscraping image-text pairs, and inherently these will contain pairs with varying levels of 'subjectivity' - some image-text pairs may exist solely because a single person decided to pair them. As such, even what is considered a 'true positive' is in the eye of the beholder (as exemplified by the reviewer's comments on the pairings in Fig 3 & 4). However, as humans we can deal with such ambiguity - or perhaps better, with such polysemy - yet, vision-language models trained with a contrastive loss presumably cannot.

In our paper we test to what extent VLM trained with contrastive loss cannot deal with the challenge of polysemy, by proposing a dataset that makes explicit the presence of polysemy. Since IMP-5k is human annotated we have reliable ground truth for the true positives, making it possible to get better insight into the polysemy challenge. However, as pointed out by the reviewer we do not have annotations for true negatives - and whilst the procedure specified by the reviewer is viable we must also recognise that this would be a very costly exercise.

We strongly believe that even with the existing setup, IMP dataset provides a useful testbed for getting a better understanding of image polysemy in VLM. While this understanding may at this early stage of research into image polysemy in VLM be somewhat coarse, we do believe that any model that explicitly handles image polysemy would perform significantly better on IMP.

the captions are abstract enough that many of the might match plausibly match to other images

During annotation we aimed to excluded captions which are very non-specific.

For example: “I took this photo with my tablet.” -- Adds no contextual value to the image and can basically used as caption for any image in this dataset.

The annotators are also instructed to avoid “over-thinking”. Annotators will first see the caption and then the image. The judgement is based on their first and second impression (directly finding the link or after a few seconds of thinking) of whether the caption and image can be a good pair. We find that this instruction helps reduce far-fetched pairings, but leaves the possibility for creativity and highly polsemic pairings.

We thank reviewer $\textcolor{blue}{R4}$ for valuable comments on dataset collection process and the core challenge in this paper. We respond to the reviewer's concerns and questions in detail below.

MPL2D and Maximize Diversity among Captions

We thank reviewer $\textcolor{blue}{R4}$ for critical question on MPL2D and diversity maximization using sentenceBERT. We first apologize for the confusion we have made in the paper, and any potential misunderstanding of reviewer's question.

If the used embeddings correspond to CLIP's normalized inner product space

We indeed used the normalized version for computing further statistics. We appreciate the concern when applying euclidean analysis with normalized inner product space. The aim of this metric is to measure how well each caption is paired with its image. We directly use the image enbedding as the pseudo center of its caption embeddings. This is because we argue that an image can have (potentially) infinitely many captions, which the collected captions are just a small subset of them.

whatever euclidean distance can be associated with in sentenceBERT's embedding representation

To clarify the clustering process during caption selection we add further details about what is described in appendix A. We first extract the text embedding of all captions (including noisy captions) using sentenceBERT. We then proceed all the text embeddings to UMAP to learn a low-dimentional projection of the text embedding while maximally maintaining the structure of those embeddings in the original high-dimentional space. Noisy captions are also included to have more samples for better UMAP learning.

After we obtain the UMAP projection, for each caption we select those captions which passed human-evaluation, and performing clustering in the low-dimentional 2D space. This is the commonly used visualization pipeline being used with different encoders and dimentionality reduction techniques. The results are also checked by human-annotators with randomly selected images. For image with more than 100 captions, we can see that each cluster represent different meanings. While for those with fewer captions (still larger than 10), the captions are still selected with maximal potential diversity.

Contributions

We thank reviewer $\textcolor{blue}{R4}$ for comments on our contributions.

I don't think the contributions of this work sufficiently shed light on any new aspects of the problem that the community was already aware of.

To the best of out knowledge there are no works that address image polysemy in-depth. Our aim is to not only make the community aware of the issue, but to also make it possible to be able to quantify to what extent it is a problem. Our results show that polysemy has the potential to heavily degrade retrieval performance, which affects VLM applied to real-world data at large.

Data Leakage

We thank reviewer $\textcolor{blue}{R4}$ for valuable concern on the data leakage problem. We would like to clarify first that we did not include captions from MSCOCO or Flickr30k in IMP. We can only verify that image-caption pair in IMP does not exist in CC3M and CC12M. We observe that there exists handful cases of very similar captions (but not identical) existing in IMP and MSCOCO, but the image is not identical. We are not able to test data leakage of whether pre-trained models have been exposed to MSCOCO or Flickr30k, either due to the private dataset, or the massive public dataset which might need a lot of time to do the leakage test.

We thank the reviewer $\textcolor{green}{R3}$ for insightful comments and recognition of IMP as a useful resource for evaluating VLM.

Modification of the figure 1 and 3

We thank the reviewer $\textcolor{green}{R3}$ for suggetion on better organizing the main text. We follow the reviewer's suggestion, now Figure 1 has been moved to appendix so there is no duplicated information.

Choice of Unsplash Images

We thank the reviewer $\textcolor{green}{R3}$ for insightful question on database we select. We would like to give two main reasons we chose Unsplash:

1. These images from Unsplash, due to their nature (high resolution, less noise, aesthetic value), are more likely to be used as wallpapers or figures in blogs, forums, news, etc. This allows us to search for the use of this exact image over the internet and see how real humans use the image. On the other hand, images from other datasets are mostly likely to only appear once on the internet.
2. We would like to make our dataset public, and Unsplash can be openly shared. So researchers can easily use the dataset for experiments or extend the dataset. For each image we have at least 10 perfect-matched image found by google-vision web entity search.

Pre-training data, which contains Unsplash images

We thank the reviewer $\textcolor{green}{R3}$ for valuable comment on potential data leakage. We can confirm that in CC3M and CC12M there is no image-text pair exists from IMP-5k. Since many models either pre-trained on private dataset or massive public dataset, we cannot test the data leakage for all these models. However, since our pipeline can get new image and captions from the web, the leakage problem should be at most a handful number.

Human-authored captions

We thank the reviewer $\textcolor{green}{R3}$ for critical comment on details of our caption gather process. While CLIP similarity is not sufficient to find semantically identical images fully, it did enable us to select candidate captions which we could then use within the further pipeline, including human annotation. Due to our pipeline we do finally end up with high quality captions. Our aim with this procedure was to increase the number of conceptual captions in our dataset, i.e., if a caption can be transferred from one image to a similar image then it is presumably less descriptive.

Evaluation Metric

We thank the reviewer $\textcolor{green}{R3}$ for valuable comment on evaluation metric. We are interested in learning how well the model can retrieve all its corresponding captions, which means the ability to deal with image polysemy. We largely agree with the reviewer that RSUM can not fully reflect this aspect. We have added medr and meanr for more analysis and more insights into the experiments, which is included in the shared response and in Appendix C.

Qualitative analysis

We thank the reviewer $\textcolor{green}{R3}$ for insightful question on qualitative analysis. In the qualitative analysis section, we have reported several “hard captions” and the false-negative rate from the image-text matching task. These hard captions passed the human evaluation, and the image-text pairs make sense to most annotators, but they failed to be matched by the model either zero-shot or after fine-tuning. Assuming we have a candidate image, there can be captions from other images which can be paired with the candidate image. However, since the candidate image and its captions has passed the human check, we would like to evaluate this by the false-negative rate, which should highlight the core difficulties for when models learn from polysemic images.

We thank the reviewer $\textcolor{orange}{R2}$ for insightful comments and recognition of the benefit of this dataset to the community. In addition to what listed in the common response, we would like to response:

Characterisation of polysemy

We purposefully do not try to strictly define polysemy as any such definition would quickly exclude valid examples. In essence, image polysemy emerges when considering images in real-world scenarios: each person who observes an image may have a different interpretation, the existence of these different interpretations (or meanings) is what defines image polysemy. Polysemy does not necessarily have to be non-descriptive - however, we found that existing captions for images are mainly descriptive and by adding non-descriptive captions we were able to increase the diversity of our dataset. We do not that descriptive versus non-descriptive is not a binary, but rather a spectrum, and we did not find that any natural ontology emerges from the captions.

Distinguishing between noise from diversity is indeed an open challenge, and MPL2D cannot do this in a straightforward manner indeed. However, both MSCOCO and IMP/5K consist of human annotated or human verified captions, which significantly lowers the noise present in these datasets. For a comparison between MSCOCO and IMP/5K we can thus conclude that differences in MPL2D are due to differences in diversity.

For a large-scale webscraped dataset like Conceptual Captions the MPL2D score is most likely strongly influenced by noise - yet it gives us an understanding of the range of MPL2D and what a possible upper-bound may be.

Experiments and future work

We thank the reviewer $\textcolor{orange}{R2}$ for critical comments on the cross-modal retrieval task, evaluation, and suggestions.

The intuition of having these experiments is given in the shared reply. We would like to answer further regarding:

Further, an adaptation of models that address image polysemy does not lead to improved performance (see, e.g., SE models in Table 4).

The SetEmbedding (SE) model uses slot attention to have different slots focusing on different regions of the given input. According to the SetEmbedding paper, it can be treated more as a multi-view approach and can achieve SOTA performance on MSCOCO and Flickr30k. Due to the nature of object-centric learning, slot-attention modules would enable the model to link different combinations of objects in the image to different captions. However, we have performed small-scale experiments of how the SE model learns to utilise the slots. For example, when given four descriptive captions and one conceptual caption, it would learn to link the conceptual caption to the background noise (one slot always appears to be dedicated to 'noise'). Increasing the number of slots can somehow mitigate the problem, but we argue that an image can potentially have infinitely many different meanings. So this would need further modification on the SE model to deal with image polysemy, as well as moving beyond the multi-view focus of current SE models, which is out of the scope of this paper.

These experiments are a good start but the paper would be stronger with additional tasks that focus on measuring model’s capability in addressing polysemy such as image captioning generation given a “caption sense”.

We appreciate the reviewers recognition of the benefits our current experiments. The central question in our work is to show that current contrastive learning-based models fail to deal with the polysemy challenge. Whilst testing other types of models may give further insight into the polysemy challenge, it also adds complexity due to the difference in how these models are trained. Retrieval directly maps onto contrastive-learning and the alignment between images and text representations - and we believe that if the model cannot overcome this challenge in image-to-text retrieval then image captioning would suffer a similar fate. Additional "sense" prompts may be used during generation, however, as we cannot determine the set of possible prompts apriori this would leave the polysemy challenge unsolved still.

Expand the discussion to additional datasets and Data Collection Details

We thank the reviewer $\textcolor{orange}{R2}$ for valuable suggestions on including more datasets for comparison and more details of diversity and quality control. We have included SBUcaptions in the table, the analysis is based on the entire SBUcaptions (849k). And due to the nature of RedCaps, doing analysis on only one topic will be biased. Instead we select all topics of year 2020, and perform the analysis. We still need some time to get statistics and analysis for RedCaps, once we finished we will add them to the main text.

We thank reviewer $\textcolor{red}{R1}$ for highlighting the importance of conceptual captions and their distinction from descriptive captions in our dataset. The reviewer's points raise crucial aspects of our work, which we aim to clarify further in addition to the shared response:

Definition of "Image Polysemy":

We thank reviewer $\textcolor{red}{R1}$ for valuable comments on "Image Polysemy".

It seems that the “image polysemy” considered in this paper mainly means: the model should have the ability to match images to both “descriptive” and “conceptual” captions

It’s correct that in our dataset, we can say captions are "descriptive" or "conceptual". A more structured way to classify captions is possible, and we would like to leave this to be discovered by models instead of human design.

The concept of image polysemy in our work revolves around the inherent capacity of an image to convey multiple, diverse meanings. This multifaceted nature of images presents a significant challenge for current vision-language models, particularly those based on contrastive learning.

As noted in our shared reply, the central aim of this paper is not to develop or expect models to match both descriptive and conceptual captions accurately. Instead, we focus on demonstrating that contrastive learning-based models struggle with the presence of multiple meanings expressed across captions.

However, I am not fully convinced that the collected datasets have more “conceptual” captions than the web image-text data such as CC3M.

The MPL2D also does not indicate that the collected dataset is more “conceptual”.

We would like to clarify that we are not expecting more “conceptual” captions than CC3M. We would expect a mixture of captions with different conceptual levels. The use of MPL2D is to illustrate the level of diversity per dataset instead of more “conceptual”. Moreover, we emphasise that our 5K test set has gone through human annotation, whereas datasets like CC3M have not. Any diversity in our IMP/5K thus comes from polysemy, whereas in CC3M it may still be noise.

Importance of multiple captions per image:

We thank reviewer $\textcolor{red}{R1}$ for insightful questions on the structure of the dataset.

b) the importance of having multiple captions per image

We believe that multiple captions differs from having a larger dataset with single captions per image. We show in the paper that when a model is exposed to multiple interpretations of the exact same image, it fails to recognise the multifaceted nature of visual content. The model could learn to match only one caption to the image if we use a single caption dataset like CC3M. To get a deep understanding of the challenge of image polysemy we believe it is necessary to explicitly model it in the dataset. We see this is a crucial step in advancing vision-language models beyond their current limitations, and beyond the typical paradigm of associating a single image with a single meaning.

Training models on datasets with single captions per image, even if they include both descriptive and conceptual captions, does not provide the same learning opportunity. It does not explicitly teach the model that multiple equally valid captions can describe a single image, an essential aspect of understanding image polysemy.

Moreover, creating a dataset with multiple captions per image allows us to control the experimental variables, thereby bridging datasets like CC3M and MSCOCO. By choosing a multiple-caption style, we facilitate a more balanced and reasonable comparison of how well models can handle diverse challenges not adequately addressed by existing datasets.

In conclusion, the multiple captions per image in our dataset are not merely about increasing the quantity of data but are a strategic decision to advance the study of image polysemy. This approach provides a unique opportunity to understand how vision-language models handle real-world challenges such as image polysemy.

We sincerely hope that these changes better reflect the intentions and implications of our work. We hope this paper can make the community aware of the importance of image polysemy and its impact on VLMs, as well as the limitations of the assumption of semantic equivalance between webscraped image-text pairs. We look forward to the opportunity to engage with the community in a broader dialogue.

For brevity, we refer to reviewer $\textcolor{red}{9GYX}$ as $\textcolor{red}{R1}$, $\textcolor{orange}{6sQn}$ as $\textcolor{orange}{R2}$, $\textcolor{green}{FPXN}$ as $\textcolor{green}{R3}$, $\textcolor{blue}{KVAJ}$ as $\textcolor{blue}{R4}$, and $\textcolor{purple}{VqT9}$ as $\textcolor{purple}{R5}$.