Close the Gap: Lightweight Image Captioning via Retrieval Augmentation

We address the mentioned weaknesses as follows:

We explicitly define lightweight in the second paragraph of the introduction as the number of trainable parameters. The overall method is NOT slower than other vanilla VL approaches. We mention in Section 3.2 that inference is on-par with SmallCap (the most lightweight approach thus far). Furthermore, LLaVa consists of a pre-training and a fine-tuning stage, where the entire LM is trained during the latter. Even when considering only the pre-training stage of LLaVa, which only trains the linear mapping, our method is more lightweight, since the **embedding space of CLIP is 4 times smaller than the embedding space of LLama, which results in 4 times fewer trainable parameters. **

We admit that we did not reach SOTA on any benchmark. However, our method is comparable with other lightweight captioning approaches, while being more efficient. Further, we could slightly improve performance by using a different prompting strategy, as highlighted in our revised version.

Generally, the retrieval set (denoted datastore in the paper) contains the captions from the training set. This is the same during training and evaluation. Only in case when the self-improvement loop is applied, we add synthetic (i.e., self-generated) captions for the training images. We do not use additional data sources. To evaluate out-of-distribution performance we will add results on NoCaps in the revised version.

Thank you for making us aware of additional relevant literature, we will add a discussion on these works in our revised manuscript.

We agree with the reviewer that our method is limited by the contents of the reference set (denoted datastore in the paper), that is the training set. However, it is a general property of machine learning algorithms that their generalization abilities are restricted to the training distribution. We do not claim that our method generalizes beyond the training distribution.

We will add results on NoCaps for a linear mapping trained on the MS-COCO dataset to evaluate the out-of-distribution performance of our method.

We agree that adding low-quality synthetic captions can amplify errors. This is the reason why we threshold synthetic captions according to a certain metric, which can be any of CIDEr-D, SPICE, B@4, ROUGE-L, etc. After additional experiments we found that thresholding according to the CIDEr-D metric works the best and usually leads to improvements across metrics.

We believe that there is a misunderstanding regarding the self-improvement loop. The self-improvement loop is only part of the training algorithm and does not run during evaluation/inference. Further, our reference set only contains texts which are pre-computed once. During self-improvement, synthetic captions are embedded once and added to the reference set. We will make this more clear in the paper.

There must be a misunderstanding, since we did not propose a new metric.

A new image does not need to be exactly covered by captions contained in the reference set. Several captions can contain objects contained in a new image and the FLAN-T5 can interpolate between them and find a suitable caption for a new image. We will add some qualitative examples on that in our revised version.

Questions:

• If a concept of a new image does not appear in the training captions then it will also not appear in the generated caption. However, this problem is not specific to our method, but applies to any other image captioning pipeline.
• We believe there is a misunderstanding. Our method does not rely on any additional data source. It only retrieves from the training set, whereas other image captioning pipelines are trained end-to-end on the training data. As any other method we rely on the assumption that the test distribution follows the training distribution.
• We will add results on NoCaps for a linear mapping trained on the MS-COCO dataset to evaluate the out-of-distribution performance of our method. Since our method uses the training captions as reference set, our method would not produce appropriate captions for test images that substantially differ from training images.

We address the mentioned weaknesses as follows:

We believe there is a misunderstanding. After the retrieval of related captions from the training set, we use a LM (FLAN-T5) to generate a new caption for the input image. As long as the retrieved captions cover different aspects of the input image, the LM can combine them to a new caption that fits the image. We make this more clear in the revised version.

Again, we believe that there is a misunderstanding. Our method does not rely on any additional data source. It only retrieves from the training set, whereas other image captioning pipelines are trained end-to-end on the training data. Thus, training captions are generally available, which is what we retrieve from during testing. Therefore, we only require an input image and no additional data. We try to clarify this in the revised version.

In principle, we agree that there is no proof that a linear mapping is sufficient in all situations. However, there are prior works that optimize a single linear layer on large-scale datasets [1,2]. This indicates that linear mappings are practically sufficient in similar settings. Moreover, these methods require huge efforts to re-train a system of [1,2] on new data, while for our method this is feasible within minutes. We agree that the datastore requires additional memory. However, FAISS allows for compressing the datastore to reduce the memory footprint in case of large datasets. We adapted the paragraph on the datastore in Section 5 accordingly.

We believe the comparison in Table 1 is adequate since it compares our method to others that aim at parameter efficiency, i.e. training as few parameters as possible. All compared methods (including ours) only use the training set for training and evaluate on the test set. Further, one of our baselines (SmallCap) uses retrieval and stores the training data in its datastore (same as ReCap).

Caption quality compared to large-scale models: We believe this is an unfair comparison, since all of the mentioned methods train much more parameters on datasets that are orders of magnitude larger than the ones we use. Further, they usually consist of two-stage training and use language models of much larger size.

[1] Mini-GPT4: Enhancing vision-language understanding with advanced large language models, Zhu et al., arXiv:1909.11059, 2023

[2] Visual Instruction Tuning, Liu et al., arXiv:2304.08485, 2023

We adress the mentioned weaknesses as follows:

We extend the related work section to highlight the differences to existing approaches to mitigate or avoid the modality gap.

To the best of our knowledge we are the first to efficiently bridge the modality gap for image captioning which, as opposed to existing approaches, alleviates end-to-end training of the captioning pipeline. We elaborate the difference to other vision-language approaches in the related work section. Further, our self-improvement loop is novel and provides a way to leverage synthetic captions for retrieval augmentation. Finally, our analysis on metrics provides a new insight that CLIP-score is not reliable for evaluation of image captioning.

We added asymptotic complexity of the retrieval (O(n)) and the computation of the least squares solution via pseudoinverse (O(d³)), where d denotes the dimensionality of the CLIP space and n the number of embeddings stored in the datastore. We want to highlight that neither of these components are a bottleneck of our method. The embeddings for the datastore are pre-computed once before training. As mentioned in Section 3.2, inference time is approximately equal to the one of SmallCap, which, to the best of our knowledge, is the most lightweight approach that currently exists.

We threshold synthetic captions in order to achieve a certain trade-off between quality and diversity. We ran additional experiments and found that thresholding according to the CIDEr-D metric works the best. Since CIDEr-D is based on n-grams, a high threshold would enforce captions as close as possible to reference captions, hence no additional information. Lowering the threshold results in more diverse synthetic captions that can be combinations of reference captions for different images.

W is a linear mapping from CLIP image space to the CLIP language space. The hyperparameter k is the number of retrieved captions per image. We show in the appendix (Table 9 and Table 10) that it slightly affects the downstream captioning performance. The hyperparameter l is only used during the self-improvement cycle, to generate several diverse captions. In our revised version we avoid the need for this hyperparameter by using nucleus sampling.

We agree that reproducibility is very important, which is why we added our code in the supplementary material. Further, to enhance understanding of our method, we added algorithms for ReCap as well as the self-improvement loop.

We address the mentioned weaknesses as follows:

• We agree that retrieval augmentation has been done in prior works. However, these works require end-to-end training which we alleviate due to the linear mapping to bridge the modality gap. Further, our self-improvement loop is a novel approach to augment the retrieval datastore with synthetic captions.

• We will add experiments on additional datasets, i.e. VizWiz and MSR-VTT in the future version.

• If a concept of an image does not appear in the training captions then it will also not appear in the generated caption. However, this problem is not specific to our method, but applies to any other image captioning pipeline. The purpose of our linear mapping is to ground images to training captions. In Table 4, it can be seen that without this grounding (ReCap⁻_Captions) there is a substantial drop in performance.

• We believe that our analysis on CLIP-score does add value. The main reason we considered it is that we searched for an appropriate metric to threshold our synthetic captions in the self-improvement loop. We observed that when optimizing for CLIP-score, our method started to hallucinate which resulted in a decrease in all metrics except for CLIP-score itself. We make this more clear in our revised version.