Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models

We thank you for a detailed review of our paper and helpful suggestions as well as highlighting our “solid” improvements and supporting analysis. We address your comments below.

On complexity of our pipeline

• Applicability of CoT: While our system contains multiple components, we demonstrate through our ablations that (as you point out) each component is necessary. We experimented with CoT approaches, finding that LVLMs generally did not benefit. This is in line with prior work highlighting the limited applicability of in-context learning (Li et al. 2023, Section 5 (1st paragraph) and, Huang et al. 2023, Table 13 shows little to no improvement from CoT across multiple tasks). In our setting, in-context learning is a required ability for CoT, since we need to give the model examples of the types of CoTs we would like it to generate (as is common practice, e.g. Lu et al. 2023, Fig. 5). In our case, the addition of multiple images into the context led the model to answer questions about the in-context images, rather than the test image.
• Comparison of token usage of RepARe with CoT: Disregarding poor performance of CoT (as discussed in Sec 2 paragraph 2 on LVLMs of our original submission), we have additionally collected estimates for the number of tokens shown to models with RePARe and with CoT, using a reasonable number of demonstrations (8). Here, RepARe requires between 430-550 LVLM tokens per task instance on an average, while CoT requires ~460 LVLM tokens on average. Given that some of the modules in RepARe can be parallelized, we contend that RepARe is comparable to CoT with in-context learning in terms of efficiency (measured in LVLM tokens) while having the added benefit of strong performance. We note that the computational cost of the off-the-shelf tools we use is negligible compared to the LVLM (which would be shared by CoT approaches). We also add this clarification to Sec 5’s limitations paragraph.

Addressing writing clarity We have updated Sec 3 to improve clarity, focusing on the areas you have mentioned. First, we have split the stages up differently (see new Fig. 2). For 3.1, we have clarified what the output from the stage is. We have reordered “salient question entities” and “information from rationales” to be grouped together to improve clarity, and have added details in the last paragraph under “Stage I(a)”. We have also added clarification to Stage I(b) and Section 3.2 to indicate that what we are scoring in 3.2 is in fact question-answer pairs, not just questions or answers. Additionally, we provide an exhaustive list of prompts used in this work and describe their usage within each module in detail in updated A.3 for further clarification. We hope that this addresses any lack of clarity in the prior version of the section.

Thank you for highlighting our “solid” performance improvements, “extensive experiments” as well your insightful questions and comments. We include our detailed responses below.

On underlying assumptions. Thank you for raising this issue. To clarify, our assumptions/hypotheses are regarding the strengths/weaknesses of models, rather than about the alignment to the pre-training data. Our core assumptions are that

1. LVLMs generally have a stronger LLM component than visual encoder (asymmetric strength)
2. LVLMs are better at tasks like captioning than QA (asymmetric abilities)
3. Finally, we assume that we can use answer confidence to rank answers (and thus, implicitly rank the rewritten questions that generated them).

We see in our results that these assumptions are borne out (updated Table 5, where we see a large boost to the LLM’s performance from RepARe and from using generated captions, and Table 2, where we ablate the scoring method). We have updated our introduction (Sec 1, page 3) as well as Sec 5 to more clearly state this. Additionally, for the third assumption we experiment with alternate ways of computing answer confidence and find that it does not significantly impact downstream task performance of RepARe (Table 8 updated).

Q1: On numeric improvement. We qualitatively examined the best candidate questions for numeric examples. We found that correctly-answered questions often include additional localization. For example, rather than answering “how many teddy bears are in the photo?” the model would answer “how many teddy bears are on the table?”. We hypothesize that this kind of additional localization helps the LVLM focus on the correct image regions, simplifying the overall task.

Q2: Clarification of terms. We apologize for the confusion. By gold answers, we are referring to the ground-truth (gold) answers contained in the datasets. The oracle setting (in the original Sec 3.2, now highlighted) refers to using the ground-truth answers to select the question candidate, i.e. choosing the question that yields the correct answer. However, such information is not available at test time. Similarly, the paraphrase oracle (in original Section 4.2, now highlighted) uses the ground-truth answers to select a candidate from a pool of paraphrases of the original question. This setting is designed to set an upper-bound on improvements from rephrasing/paraphrasing by directly using the ground-truth answer.

Q3 & Q4: Revisions to Sec 4.4. We understand the source of your confusion in interpreting Table 5. In the original table, we had endeavored to show that RepARe’s improvements are largely, but not exclusively, coming from the LLM. We have since overhauled the table and the bulk of Sec 4.4 to convey this message more clearly, highlighting the following key points in our updated paper:

• RepARe improves LLM-only performance showing that the modified questions leverage the asymmetric QA strength of the LLM.
• Despite the improvement in LLM-only performance, there remains a substantial gap in performance with and without the image (~25% points). Therefore, the modified question is complementary to the image and does not make answering the question trivial without the image.
• Following your suggestion, we include a baseline in which the caption and the original question (excluding the image) are given to the LLM and find that while this setting outperforms the question-only baselines, it still trails behind the image + original question and image + RepARe settings. This shows that RepARe incorporates visual information that is complementary to the question and the caption (added as row 5 of updated Table 5).

We hope that our changes have satisfactorily addressed your concerns. Please feel free to reach out to us with any additional questions and writing suggestions.

Thank you for your detailed feedback and helpful comments. We appreciate that you find our idea of focusing on the text modality to improve VQA performance interesting. In addition to the general response, we have addressed your comments below.

Improving evaluation suite. Your comments about expanding the evaluation framework by including more datasets and models are well-taken. Following your suggestion, we have evaluated the VizWiz dataset in Table 1 (updated) and also included performance of LLaVA-1.5 on all 3 datasets in Table 1 (updated) using the Vicuna-7B as the underlying LM. Note that much like TextQA, a considerable portion of VizWiz questions also require the model to utilize the text in the images. Also taking your feedback into account, we have added BLIP2-XXL to our evaluation.

Takeaways: Results in Table 1 (updated) indicate that our method RepARe proves to be effective on VizWiz improving accuracy by up to 7.94% (absolute points) for MiniGPT-4 models and yields 7.61% absolute improvement in accuracy on the A-OKVQA dataset with the LLaVA-1.5 model (with additional tuning experiments running). These improvements show that RepAre is robust to poor image quality, as VizWiz images are often blurred, under/over-exposed, or rotated (Gurari et al. 2018) since they were collected by visually-impaired people on mobile devices. Lastly, we show that RepARe is also effective at improving performance of BLIP-2 XXL in Table 1 by 3.84%, 5.47%, and 3.46% on VQAv2, A-OKVQA (direct), and VizWiz datasets respectively. In addition to Table 1 (updated), we discuss these promising results in the revised Sec 4.1 (highlighted).

Addressing “minor” comments. We understand the confusion caused by citing LENS in the introduction, however our objective was to reiterate LENS’s motivation about additional pre-training costs of Q-former-like models despite frozen image encoders and language models. We have now removed this citation from the sentence in question to avoid confusion.

We hope that these additional experiments address your concerns. Please let us know if you have any follow up questions.