Voila-A: Aligning Vision-Language Models with User's Gaze Attention

We sincerely appreciate your critique of our work and will take your comments into consideration for future revisions. We would like to clarify a few points that may have been misunderstood, hoping this will enable you to re-evaluate our work more accurately.

W1 & Q1, Q2 on Motivation We envision that gaze-facilitated visual question answering may take place on future wearable head devices, such as VR/AR and smart glasses. During such an interaction process, users may ask contextual questions that have proven to be ambiguous by many human-computer interaction (HCI) researches, such as using pronouns to refer to surrounding objects. Therefore, using "it" instead of the object name, which is the indirect question in our case, is very likely to happen in real-life scenarios. Furthermore, with advanced devices like the Apple Vision Pro utilizing gaze as a crucial interface for spatial computing, the integration of gaze tracking into Vision-Language Models (VLMs) presents an important yet unexplored challenge.

Another important use case of incorporating gaze is visual question answering for visually impaired people. The VizWiz-VQA-Grounding, which we also evaluated VOILA on in Appendix I, presents questions collected with blind people and the majority of these questions have at least one pronoun in them.

W2 & Q3 on the process of gaze collection The main experiment in this paper is the gaze-incorporated VLM, which refers to Voila. To train Voila, one challenge lies in the training data, where it is hard to collect a large-scale gaze VQA dataset. Alternatively, we used a mouse trace dataset called Localize Narrative as a substitution and constructed visual questions using GPT-4, thus we derived a trace-augmented VQA dataset, which we refer to as VOILA-COCO in this paper. The above is the main workflow of our project. However, we need to prove that mouse trace can be an approximation for gaze trace, even though this approximation has been used for many HCI research already. This is the reason why BubbleView is used in Section 2: we want to illustrate one basic belief of our work before we dive into it. To illustrate this, the VOILA-GAZE dataset we collected already contains gaze trace data because it is collected by gaze-sensing smart glasses, then we use BubbleView to derive mouse trace annotation on the same dataset. By comparing the two trace modalities, we provide strong evidence for using VOILA-COCO as the training dataset for a gaze-incorporated VLM.

W4 & Q4, Q5 on Evaluation We choose Otter and kosmos-2 for several reasons:

1. these two models each represents a typical VLM architecture. Current dominat VLM architectures are either cross attention based where visual feature is injected to each layer of language backbone, or token based where visual tokens were treated similarly to language tokens in an autoregressive manner. For the baselines we've chosen, Otter and Kosmos-2 are both fully open-sourced and received hundreds of citations, representing each type of VLM architecture mentioned above.
2. Compared to other Open-flamingo based VLMs, Otter is tuned on a Multi-modal in-context instruction tuning dataset (MIMIC-IT), which we believe that building upon Otter should be the best choice for a user-centered QA manner like Voila, because Voila's use case also involved in-context querying and need to follow user's instruction.
3. Compared to other VLMs that have similar architecture to Llava, Kosmos-2 has grounding ability by inputing bounding-boxes, which is the most common strategy of incorporation location information currently, thus forms a comparable baseline against our location injection strategy.

There are 2 major concerns about using human preference:

1. For the majority of questions in VOILA-COCO, the correctness of an answer can be judged with a reference answer and ground truth description given, while human preference is necessary for tasks that can not be judged objectively such as writing or image generation.
2. The goal of this paper is to establish a groundwork for gaze-facilitated VLMs, therefore reproducibility is very important. This yields the demand for an autonomous evaluation pipeline instead of human preference, which results may vary depending on multiple factors, such as region/religion/education/gender/etc. However, as a user intention incorporated VLM, reflecting user preference during the evaluation process is indeed a matter. We believe the GPT-4 ranking metric can reflect user preference since it is trained with tons of user preference data through RLHF, further, we believe that the user preference data used in GPT-4 should generally be more unbiased. This has also been demonstrated in many other works such as GPTScore[1], GPTRank[2].

[1] Fu, Jinlan, See-Kiong Ng, Zhengbao Jiang and Pengfei Liu. “GPTScore: Evaluate as You Desire.” ArXiv abs/2302.04166 (2023): n. pag. [2] Liu, Yixin, Alexander R. Fabbri, Pengfei Liu, Dragomir R. Radev and Arman Cohan. “On Learning to Summarize with Large Language Models as References.” ArXiv abs/2305.14239 (2023): n. pag.

I thank the authors for providing the detailed clarification and appreciate the effort.

Regarding motivations: the current statement doesn't mention anything related to "interpretability and effectiveness" which was originally mentioned in L8, instead it focuses on using gaze in applied settings to refer to objects. What are the relevant HCI works that show languages are ambiguous? Gaze is also ambiguous as people naturally move their eyes and identifying the right fiction that is intent-relevant is also not trivial.

I might miss something here, how is gaze going to help visually impaired people?

Could you please comment on this? Thanks!

We sincerely thank you for your support of our work and for your attentive reading and thoughtful suggestions.

Baseline models

Note that except for Otter, we also have Kosmos-2 as baseline, as shown in Figure 5 and 6. For reference [43], this is a paper published in 2020 and in their experiment description, they stated that all the models they used were published before 2019, which VLMs haven't emerged back then, therefore comparing our Voila to their models may be unfair and liekly to have obvious outcome. We choose Otter and kosmos-2 for several reasons: (1) these two models each represents a typical VLM architecture. Current dominat VLM architectures are either cross attention based where visual feature is injected to each layer of language backbone, or token based where visual tokens were treated similarly to language tokens in an autoregressive manner. For the baselines we've chosen, Otter and Kosmos-2 are both fully open-sourced and received hundreds of citations, representing each type of VLM architecture mentioned above. (2) Compared to other Open-flamingo based VLMs, Otter is tuned on a Multi-modal in-context instruction tuning dataset (MIMIC-IT), which we believe that building upon Otter should be the best choice for a user-centered QA manner like Voila, because Voila's use case also involved in-context querying and need to follow user's instruction. (3) Compared to other VLMs that have similar architecture to Llava, Kosmos-2 has grounding ability by inputing bounding-boxes, which is the most common strategy of incorporation location information currently, thus forms a comparable baseline against our location injection strategy.

Presentation Issue

1. Hyperparameter settings can be find in Appendix F.
2. We will proof read again and fix format issues.

Dear reviewr 3dvo, we sincerely thank you for your support of our work and appreciate your thoughtful suggestions that have helped us improve.

W1 Regarding the Perceiver's choices for Key (K), Query (Q), and Value (V), we can delve into a clearer explanation here. The primary function of this component is to condense visual information into compact latent representations, which can subsequently be utilized for cross-attention with language models. In this setup, the Q are derived from the latent representations, while the K and V originate from the visual inputs. However, calculating K and V from the latents as well allows for an efficient self-attention mechanism within the latents themselves. In section 4.3.2 and Figure 16, we compare more design choices beyond Flamingo structure.

W2 Thank you for your kind suggestion. Our primary goal is to establish a reproducible automated benchmark, and as such, most of our evaluations are conducted without the inclusion of supplementary user surveys. To guarantee that the question-answer pairs are amenable to automatic evaluation, each reference answer has undergone meticulous review by our dedicated annotator and has been subject to a rigorous double-checking process during the selection of the final test cases. We acknowledge the potential benefits of a user study on the voila-gaze results and will take into consideration the incorporation of such a study to further validate the dependability of our automated metrics.

Q1 Note that Section 2's motivation is to demonstrate the similarity between gaze trace and mouse trace so that training with VOILA-COCO can make sense. Therefore, we collected some mouse trace data on the VOILA-GAZE dataset to compare with its gaze trace. For this comparison, the gaze trace contains 1k sample of gaze points.

Q2 We observe that the value of 'g' is larger in the median layers, whereas it is relatively smaller in both the later and the earlier layers. We initialize 'g' to zero to guarantee that the output matches that of the original model at the start of training.

Q3 RefCOCO's referring expression comprehension task requires models to predict a bounding box with a given referring phrase, however, our VOILA model does not possess the ability to generate bounding boxes as it is not trained on such data. Instead, we were able to test VOILA's overall referring expression comprehension ability using VizWiz-VQA-Grounding dataset (as shown in Appendix I). VizWiz dataset is specially collected for answering visual question for blind people and the majority of VizWiz-VQA-Grounding dataset contains pronoun usage in the questions. Therefore, to accurately answer VizWiz's question, the model should understand what the pronoun refers to in the image.

We sincerely thank you for your support of our work, for your meticulous review, and for your thoughtful suggestions that have helped us improve.

W1

• Both Kosmos-2 and Otter are finetuned on the same training set of VOILA-COCO.
• During the data generation process, we generate one direct question and one indirect question for each query content. During the training process, we randomly select one question from each question pair, i.e., the direct questions and indirect questions are expected to be balanced during the training stage. This training question composition remains unchanged for all tables and figures in the Experiment session. During the validation process, by default, we use only indirect questions (Figures 5 and 6, Tables 3 and 4). The only exception is Table 2, which shows the ablation results on query types, row 1/3/5 use indirect questions and row 2/4/6 use direct questions as validation set.
• During evaluation, kosmos-2 takes bounding boxes as input, following the same format as in their paper and code repo. The way we preprocess bounding boxes for kosmos-2 is consistent with VOILA's variant in Table 3 (row 3 and 4). Specifically, we calculate the minimum bounding box that includes the gaze trace of each question. For Otter evaluation, we do not directly inject gaze traces or bounding box data because Otter hasn't seen such kind of data during training. As shown in Figure 16 you've mentioned, the top-right and bottom-right setting maintains the same model architecture as Otter. These settings add special trainable tokens for the gaze trace/fixation bounding box, which can be considered as a location information-injected version of Otter. As shown in Table 3, our VOILA also surpasses these Otter variants.
  In Figures 5 and 6, although we compared VOILA to no location information-injected version of Otter, it is because the main goal of our paper is to (1) examine the necessity of additional signals for generating appropriate responses in everyday scenarios, and (2) evaluate whether gaze signals offer a superior representation of user intent compared to other modalities such as bounding boxes, instead of establishing a traditional benchmark for competitive model architectures based on specific metrics.

W2 The GPT-4 ranking has 3 outcomes, that is, win, tie, or lose. As shown in Figure 5 "Voila vs Kosmos-2" Grounding column, approximately 40% winning rate already surpass its competitor. Therefore, in Table2, comparing row 1/3/5, VOILA surpasses Otter in indirect questions, and comparing row 2/4/6, VOILA surpasses Otter in direct questions. Also, comparing row 1/2, 3/4, 5/6, both model variants and prompt strategy performs better on direct questions compared to indirect ones. We will update tie/loss rate into the table in later revision.

W3 As you kindly point out, instead of chasing technical novelty, our paper aims to implement modifications and prove them to be necessary and beneficial, establishing the groundwork for gaze-incorporated VLMs. In developing VOILA, a critical consideration is the adherence to the established architecture of current VLMs, making it easy to follow and generally applicable. It is essential to avoid introducing a significant number of new parameters or making extensive structural modifications. This constraint is due to the limitations of the current gaze dataset, which does not support large-scale pretraining. Additionally, we must be vigilant in preventing catastrophic forgetting during the fine-tuning process.

Q1 Thanks for pointing out and we will update our statement accordingly: Our study on Otter demonstrate that current VLMs fails on various daily usecases because of disalignment with users' intention. Also, recent HCI research like GazeGPT[1] incorporates only one gaze point as location information to GPT-4v in the form of bounding boxes, illustrating the improved alignment and user preference after the incorporation of gaze. [1] GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear Q2 Will update high resolution illustrations.

Q3 For better numerical stability, we choose addition instead of multiplication.

Q4 This architecture, drawing inspiration from Perceiver and Flamingo, seamlessly integrates self-attention within latent features and cross-attention between latent and visual features in a unified step.

Q5 The vector $g$ is a trainable multi-channel variable with dimensions corresponding to (Number of Heads, Dimensionality).

Q6 That means word embedding and final projection after the last layer of language model.

Q7,8,9,10 Sorry for the mismatch and we will revise it. Here is the fact: In the first stage colored in pink, we train the MLPs and gate $g$ of gaze inputs for 1 epoch, then tune together with Perceiver resampler module for 1 epoch. In the second stage colored in orange, we finetune MLPs, $g$, Perceiver, Cross attention, word embeddings together for 1 epoch. In total, that's three epoch as stated in L737.

Q11 To maintain the model's versatility in handling scenarios without trace input, the inclusion of a special token like 'fixation' is crucial for ensuring the model's robustness across diverse situations.

Thank you for taking the time to write a rebuttal. I will be taking it under advisement.

That being said, could you elaborate more on W2? I'm still not sure I'm interpreting the results properly.

My understanding from your response is that Table 2 is missing the tie and loss rates, which means the win rate alone is not enough to actually determine which method is winning. For example, comparing row 3 to row 1, VOILA is achieving a win rate of 41% compared to Otter-base on coreference queries, if the tie rate is 19% or higher, this would mean that VOILA is better, otherwise, Otter-base would be better. Is my reasoning correct? If the answer is yes, then how are we supposed to interpret those results based on the win rate alone? The scenario you describe in Figure 5 where Voila has around 40% win rate against Kosmos and is winning overall may not be happening in Table 2 if the tie rate is smaller.

Thank you.