Revisiting the Role of Language Priors in Vision-Language Models

After reading through the comments, we believe Reviewer 85bY’s concerns may stem from some misunderstandings about our work. We first clarify:

Question 2: If VLMs are deployed as zero-shot models, why do we care about the gap between test and training (p_te vs p_tr) since the model does not use p_tr?

As we focus on zero-shot applications of generative VLMs, $P_{train}$ in our paper refers to VLMs’ pretraining datasets, such as LAION for BLIPv1 and BLIPv2. In fact, most of the recent popular benchmarks such as ARO and SugarCrepe are designed to evaluate VLMs in a zero-shot fashion, and thus do not even provide a training split.

Question 1.1: I suggest the authors perform the experiments using a more recent VLM like (LLaVA or BLIPv2, also BLIP is not SoTA) which both can be directly used to compute the language marginals.

The language marginal ($P_{train}(t)$) refers to VLMs’ pretraining captions. Therefore, without sampling any images, one simply cannot approximate this marginal from the language decoder of LLaVA or BLIPv2, whose language models are pretrained on text corpora different from VLMs’ pretraining captions.

Also, our paper already includes SOTA captioning-model BLIPv2 results (both stage-1-Q-Former and stage-2-FlanT5-XL). For instance, in Table 9, we show that our $\alpha$-debiasing solution consistently improves all BLIP model variants even with a fixed $\alpha$=1.

(Lack of novelty) Because this generative score has been applied before, what is particularly novel about our work (e.g., compared to paper [1])?

We agree that the generative score has been applied in prior art such as [1]. In fact, our goal is not to introduce a new method. Instead, we aim to revisit the language priors and biases in generative VLMs (as noted by Reviewer ANt2). This issue has been neglected in recent popular image-text retrieval benchmarks until now.

We also carefully review the paper [1] and note two crucial differences:

• While [1] focuses on training this score from scratch, we focus more on its zero-shot application, especially on how to improve its performance under a substantial distribution (marginal) shift between VLMs’ pretraining data and downstream tasks.
• [1] only applies this generative score to text-to-image retrieval tasks (as we also show in Table 3-b). However, it chooses not to report image-to-text retrieval performance, although it uses the same COCO and Flickr30K benchmarks. We think this omission is likely due to the issue of language biases, which our paper thoroughly addresses. Concretely, we also provide algorithms for debiasing the generative score from [1], which dramatically improves performance on challenging benchmarks (such as COCO and Flickr30K).

(Soundness) Is our calibration approach motivated by pointwise mutual information (PMI)? How does the connection between our approach and PMI help the reader?

To clarify, our approach (dividing by the marginal $P_{train}(t)^\alpha$) is motivated by our analysis in Section 3 (not by PMI). Here is a concrete example (cf. Figure 1) for ease of understanding: in some recent compositionality benchmarks such as SugarCrepe, a negative caption such as “people are cooking in a kitchen” is more common in the VLM’s trainset (thus a higher $P_{train}(t)$) than the positive caption “people are posing in a kitchen”. As such, dividing by $P_{train}(t)$ forces the model to select the caption that matches better to the image (rather than the caption with a larger marginal prior).

Our Appendix A shows that our approach $\frac{P(t|i)}{P(t)^\alpha}$ happens to be the same as smoothed estimates of pointwise mutual information (PMI$^k$), which also reduces the effect of marginal priors from training data. We include this connection because some readers may find it interesting.

(Minor: Efficiency) VisualGPTScore is slower than CLIPScore especially when the size of the retrieval index grows. [1,2] show some methods for reducing this cost. Can the authors comment more on this in the manuscript?

While efficiency is not the focus of our work, we are happy to cite [1,2] that suggest re-ranking or distillation to reduce the inference cost. This could be a promising future direction.

Question 1: The proposed trick to estimate the marginal over text p(t) is not sound. Why should averaging Gaussian noise fed as input to the VLM work more efficiently than averaging over the distribution of natural images? Is there any theoretical guarantee that this is the correct thing to do? Especially given the fact that Gaussian noise has never been used during training of the VLM and is therefore out of distribution for the model.

Our empirical trick is motivated by [3] that approximates P(answer) using a null prompt “N/A”. Yet, the language models used in [3] also are never trained on P(answer | “N/A”). Studying its theoretical guarantee could be an interesting future work.

We appreciate Reviewer AoET's feedback. We begin by clearing up some potential misunderstandings:

Question 3: In the case of "blind models", how is the measurement of $P(t^i_{positive})$ with respect to $P(t^i_{negative})$ conducted for each instance? Elaboration on the methodology for evaluating individual test instances in "blind models" is needed.

$P_{train}(text)$ is measured for each individual text independently using Monte Carlo estimation (Equation 9): $P_{train}(text) \approx \frac{1}{n} \sum_{k=1}^n P_{train}(text | image_k)$, where $k$ represents the number of random Gaussian noise images sampled.

Weaknesses 1: Measuring $\alpha^{*}$ demands test-time privileged information, which raises concerns about the approach's validity.

Our method does not demand test-time privileged information. In practice, one only needs a small validation set from the target task to search for optimal alpha, or can simply make practical assumptions without tuning alpha, e.g., choosing $\alpha=1$. We show in Table 2-a that both consistently improve performance without using test-time privileged information.

Weaknesses 2 + Question 1: Addressing language bias is a crucial aspect, yet the method employed for debiasing, measured by a, appears to operate at the dataset level ($P_{train}$ vs. $P_{test}$) rather than the instance level. Evaluating the total effect, as P(t|i) - P(t|i =$\phi$), would provide a more meaningful approach to remove the "language bias". Shouldn't the value of $a^{*}$ vary based on the specific target (t)?

Our $\alpha-$debiasing method ($\frac{P_{train}(text | image)}{P_{train}(text)^\alpha}$) is motivated by the widely existing train-test shift from VLM’s training set (web corpora) to testing benchmarks (such as SugarCrepe), as thoroughly discussed in Section 3. Without additional training, it is unclear how to measure an $\alpha$ that varies based on specific target texts. As our paper focuses on training-free debiasing, we leave learning instance-specific $\alpha$ to future work.

Weaknesses 3 + Question 2: While the analysis is extensive in the context of I-to-T retrieval tasks, it falls short in terms of assessing a broader range of downstream tasks. Incorporating analyses of zero-shot classification, VQA2.0, and GQA tasks would offer a more comprehensive perspective.

Our score and debiasing method are designed to calculate the similarity score between an image and a caption. Thus, it does not naturally support VQA tasks.

While BLIP was neither designed nor benchmarked on zero-shot classification, we follow your suggestion to evaluate our methods on ImageNet1K and show that our $\alpha$-debiasing solution (using a fixed $\alpha=1$) can nearly double the performance from 18.6% to 36.2%. We also try to search for alpha by sampling an additional one-shot validation set (and repeat using 3 random seeds). This extremely low-shot validation set can lead us to a near-optimal alpha, achieving 40.0% without retraining or finetuning the model:

In this experiment, by sampling only a single Gaussian noise image for the calculation of $P_{train}(t)$, we incur a negligible inference cost, equivalent to the testing of one additional image.

Question 4: Is it plausible that negative captions rarely occur in web corpora? This factor might be affecting the performance of "blind models" and deserves further investigation.

You are right that negative captions in benchmarks such as ARO rarely occur in web corpora. This explains why even “blind models” are able to outperform SOTA methods on many recent popular benchmarks. As said by Reviewer ANt2, our experiments suggest that “existing benchmarks are in some sense, still not “hard enough” and contain correlations that can be exploited or can be solved by using language priors”. We hope our experiments are useful for everyone designing new benchmarks for vision-language tasks.

We sincerely appreciate Reviewer ANt2's positive and insightful review. Your understanding and articulation of our work's core contributions are highly encouraging. We agree that exploring the extent to which benchmarks can be solved by exploiting language priors is a critical question, and we are glad that our work contributes meaningfully to this discussion. We are also grateful for your recognition of the simplicity and generality of our debiasing approach.

We address your (minor) concern below:

A minor weakness of the paper is that they do not compare with the recent crop of truly "large" generative vision-language models like BLIP-2, LLAVA, etc. However, this is a minor weakness and I do not think it needs to be really addressed in this work, since these models are still new enough that training them is extra engineering work. Also, if anything, the language prior should be worse in LLM-based VLMs.

We have included the results of the state-of-the-art captioning model BLIP-2 (both stage-1-QFormer and stage-2-FlanT5) in Appendix D. For example, Table 9 shows that our debiasing solution, even with a fixed alpha = 1, consistently improves performance on balanced benchmarks across all model variants. For your convenience, we attach the image-to-text retrieval results (text score) for Winoground and EqBen below:

We appreciate Reviewer eKtF’s feedback. We will address your concerns below:

My primary concern is the application's real-world viability. Image-text retrieval, often utilized in search engines, demands high time efficiency. With this method, for every new image, the VLM must process all texts, resulting in substantial computational costs. In contrast, traditional methods like CLIP pre-compute text embeddings and only require a dot product with each image embedding. Hence, while the experimental results are impressive, I question this method's practical value.

While our proposed score is slow for large-scale retrieval, it can still be practical for other applications like evaluating text-to-image generation, e.g., measuring the similarity score between a text prompt and a generated image. As an illustration, our method can complement the widely used CLIPScore [1,2], which struggles with understanding complicated texts that involve compositions of objects, attributes, and their relations. In fact, recent image-text retrieval benchmarks, such as ARO and SugarCrepe, are designed specifically to evaluate VLMs’ compositional reasoning capabilities.

Since our primary focus is on studying the issues of linguistic priors in both generative VLMs and vision-language benchmarks, we leave it to future work to adapt our method for more efficient large-scale retrieval tasks. For instance, one could use established re-ranking technique or distill the VisualGPTScore into CLIPScore to improve inference speed [3].

Following the first weakness, could you provide a time-efficiency assessment comparing various methods?

We provide a time-efficiency comparison between ITCScore (CLIPScore), ITMScore, and VisualGPTScore, using the same BLIP model for large-scale inference (1000 images x 5000 texts):

As demonstrated in Table 7 in the appendix, as the dataset size increases, the OTS scores progressively deteriorate, and the gap with ITM widens even when using the optimal alpha. How can this be explained? Might this indicate an inherent limitation of the method?

Good question! We include Table 7 (retrieval performance on training set) to motivate a scientific discussion on the inherent linguistic priors of VisualGPTScore, which make it a biased estimator of $P_{train}(image | text)$. For a thorough discussion, please refer to Appendix C, titled “Is VisualGPTScore a biased estimator?”. Intuitively, such estimators can suffer from well-known “long-tailed” issues that arise when learning from imbalanced datasets (i.e., in our case, certain sentences are more common than others). We hope that our initial discussion on this issue can serve as a useful reference point for future work.

The paper mentions, "To apply this to our generative VLMs, we choose to sample 'null' inputs as Gaussian noise images." Why are Gaussian noise images suitable as "null" inputs?

Our empirical findings echoes [4], which uses a language model to approximate P(text) = P(text | “N/A”). Their null text prompt “N/A” is also not seen during training, but can empirically help calibrate language models. Also, we try both gaussian noise images and training set images, and find that they show similar performance (Table 4).

References:

[1] Ruiz et al. “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation”.

[2] Brooks et al. “InstructPix2Pix: Learning to Follow Image Editing Instructions”.

[3] Antonie Miech et al. “Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers”.

[4] Zhao et al. “Calibrate Before Use: Improving Few-Shot Performance of Language Models”