Long-Form Answers to Visual Questions from Blind and Low Vision People

Dear Reviewer sGq5,

Thank you for your thoughtful review!

Dataset contains answers from GPT-4V and Gemini
See rebuttal to Reviewer LFVf with the same title.

Functional roles of LFVQA
Prior research utilized functional roles in downstream NLP tasks including summarization [1], information retrieval [2], and generation of sentences of a specific functional role (e.g., explanation) [2]. To our knowledge, we are the first to design and annotate functional roles of answers to visual questions. We believe that understanding the functional roles of LFVQA can help us understand the behavior of a VQA model, compare its answer to that of other models or experts, and further utilize them for core answer extraction and fine-grained evaluation of long-form answers. We will clarify these motivations in Sections 1 and 3.3.

Human evaluation
We conducted human evaluation to complement the limitations of the automatic evaluation. 6 automated metrics used to evaluate LFVQA have limitations (e.g., BERTscore can yield higher scores for semantically related but factually incorrect answers.) Human evaluation can bypass these deficiencies, provide flexibility (recognizing multiple valid answers), and assess diverse aspects including clarity or relevance. As human preferences vary, we conducted significance testing to ensure the trends we claim are reliable (Figure 2).

We will also add full instructions provided to evaluators including evaluation scenario and definitions of the 4 evaluation criteria to A.1.2.

Connection to abstention experiment
Our abstention experiment (paragraph 6) is motivated by the findings from the human evaluation (paragraph 5) which revealed that BLV users are more likely to be misled by hallucinations from VQA models. To better connect them, we updated the section orders and Introduction to describe automatic evaluation, human evaluation, and abstention experiments in order. We hope this better explains how human evaluation complements the automatic evaluation, and how the findings from the human evaluation motivate our abstention experiment.

Others
We revised the sentence: “Vision language models can now generate long-form answers to questions about images, which we call long-form visual question answers (LFVQA).” We will further clarify the difference between BLV and the traditional VQA dataset in the paper.

[1] “A Discourse-Aware …” Cohan et al., NAACL 2018.
[2] “Modeling Exemplification …” Wang et al., NAACL 2022.

Dear Reviewer PViS,

We appreciate your thoughtful review!

Presentation and clarity
We will highlight the important numbers (e.g., bold highest values) in Table 4-5 to help readers focus on the key results. We will also swap the age and gender columns in Table 16.

Usefulness of LFVQA for BLV people
We will clarify that our work on long-form answers is motivated by several reasons: (1) prior research in QA has shown that people have diverse preferences in length of answers [1], (2) many human studies with BLV people reveal that they want detailed descriptions [2, 3, 4], (3) some questions in the VizWiz dataset even reveal this desire explicitly (“Yes this is a Google Street View image, I need a detail, a detail of the description, as much as possible.”), and (4) BLV people are already active consumers of long-form VQA services through applications like Be My AI. Given the request for comparison, we additionally conducted a survey to understand how BLV people evaluate human short-form answers (from VizWiz) vs. human long-form answers (from VizWiz-LF) for overall preference and fine-grained criteria (relevance, helpfulness, plausibility, and clarity). BLV people preferred long-form answers to short-form answers (long: avg. $0.75$ vs short: avg. $0.25$, range: $0$-$1$, stdev. $0.43$) and this difference was significant (p-value: $3.66 \times 10^-8$). We will add a table of this survey results in the Appendix.

Abstention experiment section feels detached
Our paper’s abstention experiment is motivated by the findings from the human evaluation study (Section 4) that BLV users are more likely to be misled by false information hallucinated by VQA models, and that unanswerable questions are prevalent in VQA datasets for BLV users. To better connect Section 6 to other parts of the paper, we will revise the paper to change the section orders: 4. Automatic Evaluation, 5. Human Evaluation, and 6. Experiments on VQA Model Abstention. We hope this better explains how we conducted human evaluation studies to complement the automatic reference-based evaluation, and how the findings from the human evaluation studies motivate our experiments on VQA model abstention.

[1] “Decontextualization…” Choi et al., TACL 2021
[2] “Understanding Blind …” MacLeod et al., CHI 2017
[3] “Toward Scalable …” Salisbury et al., HCOMP 2017
[4] “Twitter A11y …” Gleason et al., CHI 2020

Dear Reviewer 1BBk,

Thank you for your helpful review!

Dataset contains answers from GPT-4V and Gemini
See rebuttal to Reviewer LFVf with the same title.

Dataset collects only one expert answer per question
We prioritized our annotation budget on collecting one expert long-form answer per visual question to create a larger dataset (600 questions) over collecting multiple long-form answers for a smaller dataset. This choice aligns with numerous other popular dataset publications [2, 3] that only collect one annotation per example because they rely on expert annotators. In our case, people with experience describing images for BLV people were challenging to recruit and expensive to hire (see A.1.2). Many dataset papers that collect redundant annotations do so because they rely on less expensive, less-trained, anonymous crowd-workers, or contain noisy data crawled from online resources [4]. Still, we agree that collecting more expert annotations could be valuable. Similar to prior work [1], future work can augment our VizWiz-LF dataset with more ground truth references and perform human correlation studies. We will mention this in Section 2.

Dataset size
While our dataset has a smaller number of questions due to cost and API rate limits, the dataset still represents a variety of question types as we intentionally evenly sampled questions for each of four question types (Identification, Description, Reading, Others). We collected multi-sentence answers for each of the questions from multiple sources and provided a thorough analysis of our dataset with detailed qualitative analysis and human studies.

Open-source models not evaluated on long-form answers generation
We additionally conducted automatic evaluation of six open-source models using multiple reference-based metrics (ROUGE, METEOR, BERTScore, LAVE) and will add these results to A.3.2. We chose to perform human evaluation only on human generated long-form answers and two models GPT-4V and Gemini (demonstrated as state-of-the-art in image understanding tasks [5, 6]), due to the cost of human evaluation and the difficulty of recruiting BLV participants.

[1] “Investigating Evaluation …” Gupta et al., SIGDIAL 2019
[2] “Vizwiz-priv …” Gurari et al., CVPR 2019
[3] “Scene Parsing …” Zhou et al., CVPR 2017
[4] “ELI5 …” Fan et al., ACL 2019
[5] “MMMU: A Massive…” Yue et al., CVPR 2024
[6] “Gemini: A Family of Highly Capable Multimodal Models, Team et al.

Dear Reviewer LFVf,

Thank you for your thoughtful review! We are encouraged that the review recognized the importance of our problem and our comprehensive evaluations and experiments.

Dataset contains answers from GPT-4V and Gemini
We included model generated answers as well as expert answers in the dataset so that it is easy to compare the structure (functional roles) and quality (human ratings) of the long-form answers from multiple sources. We agree that GPT-4V and Gemini can introduce noise/inaccuracies. Thus, we collected the model answers to conduct an in-depth analysis in this new domain of long-form visual question answering rather than using the model answers as ground truth to evaluate other models. We will clarify the goal of our dataset in the introduction and conclusion. While we already discuss the limitations of potential hallucinations in the dataset in our ethics statement, we will also mention these limitations when we introduce the dataset in Section 3.

Expert describers’ information
We recruited expert describers with experiences in describing images for BLV people using Upwork (e.g., professional audio description writers, full-time visual interpreters). We compensated describers with their hourly rate (28 USD, STDEV = 8), and describers answered 30 questions each, spending an average of 3.9 minutes (STDEV = 1.2 minutes) per visual question. We included compensation and task time in Appendix A.1.2, and we will add a reference to this A.1.2 when we introduce the description task in Section 3.2.