Retrieval-Enhanced Contrastive Vision-Text Models

1. It is not clear why it happens w.r.t "it is hard to align the image and text modalities."

We explain why we think it is hard to align image and text modalities in the following sentences in the text:

While every image is metaphorically valued at a thousand words, it is often paired with a short, sometimes noisy, text that neither exclusively nor comprehensively describes it. For example, current vision-language models are good at associating images of cars with generic concepts such as “car”, “mechanics” or “road trip”, because these are common words paired with car images, but less at finegrained, instance-level, associations such as the specific brand, series or year of that car...

2. Abstract mentions that "fine-grained entities which are rare". Having concepts rare seems like a different reason from "being hard to align image and text modalities". Which reason is more reasonable?

Both are valid reasons. It is hard to align image and text, and it is especially harder for model to align concepts that are rare.

3. "One caveat that we identify in this approach is that initial captions are augmented within their modality only, hence limiting the potential added-value brought by the retrieved items." Can authors clarify?

Thanks for the comment. We will make sure to make this point clearer. In our paper, we show that it is more beneficial to combine text and image modalities. In K-Lite, the captions are only augmented with additional text and they are not combined with image modality to create cross-modal embeddings.

4. The sentence is unclear -- "However, when crossing modalities, these representations are less successful in identifying suitable matches, such as finding the text with the closest representation to a query image representation."

Thanks for the comment. We will make sure to make this point clearer. Crossing modalities means computing the similarity between images and text, i.e. two different modalities. We show that off-the-shelf CLIP models struggle with this. But it can do a much better job when retrieval is performed within modality (images retrieved given query image, or text retrieved given query text). See cross-modal search in Figure 4 for some examples. In this work, we address this issue by augmenting the input with retrieved images.

5. The sentence is unclear -- "Through this process, we successfully transform the image and text representations into multi-modal versions, which significantly simplifies their alignment"

When the query is an image, we augment its embedding with text embeddings corresponding to the similar images. On the other hand, for the text corresponding to the class names, we augment them with the image embeddings corresponding to the similar captions. We perform a similar process when the query is text also. Hence, we end up with multi-modal embeddings. The resulting multi-modal embeddings are easier to align.

6. The paper does not discuss computation cost, complexity, etc.

We do discuss the complexity and computation cost in Section 4.3:

Another limitation is that the performance gains of RECO come at the cost of increased inference time. In practice, we use a highly-optimized approximate k-NN algorithm (Guo et al., 2020). It takes about 14ms to query Webli (956M examples) with a single 512-d ViT-B/32 CLIP embedding. Using retrieval at inference time incurs an overhead of 25% compared to not using any retrieval at inference time (e.g. baseline CLIP model), but improves the accuracy by up to 10.9.

7. The paper uses a dataset called WebLI and explains that it is a private dataset. However, how to access the private dataset? Does it mean that authors own the private dataset (indicating a leakage of author identities)? How to fairly compare methods if authors use a private dataset?

WebLI is used as the dataset in a series of previous works, including PALI (Chen et al., 2023), PALM-E, CLIPPO etc. Moreover, we show in Table 6 of the Appendix that our method is not specific to WebLI, and we still have large gains when using publicly available LAION dataset.

8. When discussing "Is the performance boost merely due to additional training?", the paper uses ResNet backbone and learns a transformer. Given that transformer might be better than convnets, it is questionable to claim using a transformer is a novel technique. Can authors discuss results if using a transformer backbone in CLIP along with transformer head for fine-grained recognition?

The reviewer mentions the section "Is the performance boost merely due to additional training?". We do not use ResNet for results in that section. Instead, we use the ViT-B/32 transformer backbone of CLIP. This is clearly stated in the beginning of Section 4.3: “We use ViT-CLIP-B/32 throughout this section.”

We always make sure that our model uses the same backbone as the baselines we compare against. We will make sure that this becomes more clear in the paper.

1. The best results are obtained using a non-public dataset (WebLI), for which the authors do not provide any instructions on how to reproduce it.

The details about how the WebLI dataset was collected is described in (Chen et al., 2023) (See Section 3.2, and Appendix B and G of Chen et al., 2023). We further investigate the differences between WebLI and LAION below, in response to other questions from the reviewer.

2. Some tables do not report results on the Dogs dataset, it would be better to add them since this dataset is used in the main results Table.

We add the results for the Dogs dataset from Table 3 below, and will update the paper accordingly:

3. Why do the results for Text-to-Image retrieval improve more than Image-to-Text?

We observe that the baseline CLIP performance is higher for Image-to-Text than Text-to-Image. Therefore, it is easier for our retrieval-augmented approach to provide gains for the text-to-image.

4. Why is WebLI a better memory bank than LAION? Is the number of images, better alignment between images-text, better captions...?

We thank the reviewer for their insightful question. To further examine whether the difference is due to the number of images, or WebLI being higher quality, we randomly subsample 40% of WebLI (to make it the same size as LAION), and re-train our model with this version. We report the results below:

As shown in the table above, our WebLI-400M results are more comparable with LAION-400M results. The gap in accuracy reduces, but probably WebLI has slightly less noise than LAION.

5. Would a model trained with WebLI perform well using LAION during inference?

We report the results below. We do not observe any meaningful gain in performance compared to a LAION trained model. The trained models learn to fuse the retrieved items in a similar way. Then the quality of the retrieved items at inference play an important factor for the overall performance.

1. If the image-text pairs are noisy, the retrieved cross-modal information may be inaccurate, potentially undermining the final performance.

The reviewer is correct in saying that the noisy image-text pairs may undermine the final performance if they are not processed correctly. Our method learns a light-weight transformer, which learns to pool a set of embeddings from the nearest neighbors. The transformer intrinsically learns to filter out irrelevant examples from the kNN list, while keeping the relevant ones. Therefore, we don’t make the assumption that every retrieved item must be relevant.

To study the impact of the noise in our data, and whether the transformer is needed, we design a “Mean fusion” baseline in the paper, which tests whether all the nearest neighbors are already correct, and if we can simply take the average of their embeddings. Table 3 in the paper shows that this is not the case. Mean fusion significantly degrades the performance of the baseline model, which indicates that the nearest neighbors contain more irrelevant examples than the relevant ones.

Finally, we visualize some examples with their retrieved kNNs in the revised PDF (Figure 9 - Appendix). It is shown that RECO predicts the correct class, even if the number of correct images do not outnumber the number of incorrect images in the kNN list.

We thank the reviewer for their suggestion, and will include this discussion in more detail in the future revision.

2. The uni-modal search process seems to have a significant overhead (in terms of computation and IO access) during inference, since it has to perform retrieval from a large number of image-text pairs.

We use the approximate k-NN algorithm SCANN (Guo et al., 2020) for retrieval. It takes about 14ms to query Webli (956M examples) with a 512-d embedding to perform retrieval using the SCANN index. Using retrieval at inference time incurs an overhead of 25% compared to not using any retrieval at inference time (note that retrieval runs on CPU and doesn’t use GPU), but improves the accuracy by up to 10.9 points.

3. While this method enhances performance on fine-grained tasks, how does it affect the accuracy of recognizing common generic concepts? Can it be applied to common visual recognition tasks?

We show the results for ImageNet1k and Places365 as common visual recognition tasks on Table 1. ImageNet1k contains about 60 bird species and about 120 dog breeds, but also “common generic concepts ”, e.g. bookcase, boathouse etc. Places365 contains generic scene classes, e.g. basement, park, tower etc. In these benchmarks, the performance is still improved, but by more moderate margins (i.e. +1.1 on ImageNet1k and +1.6 on Places365). We hypothesize that this is because the pre-trained CLIP models are already good at recognizing common, generic classes, but don’t perform as well on fine-grained classes (Cars, CUB, Flowers etc).