Voila-A: Aligning Vision-Language Models with User's Gaze Attention

Thank you for your insightful feedback on our work. Your input is invaluable in helping us identify improvements and refine our research. We're committed to addressing any shortcomings and appreciate your constructive suggestions.

Presentation and Technical Details:

We recognize the need to enhance our presentation and include crucial information in the main paper by providing detailed explanations where needed. To address your concerns, we have made significant improvements to our manuscript, including extensive reorganization, clarification, and rewriting as we replied in General Response.

Baseline Selection and Technical Novelty

We would like to begin by clarifying the rationale behind our choice of Kosmos-2 and Otter as the two baselines for comparison. Our method aims to demonstrate improvements in two key areas: state-of-the-art multimodal conversation capabilities and grounding ability with additional modality input.

First, Otter is based on the Flamingo architecture, a well-recognized milestone in vision-language research. Otter has been fine-tuned on egocentric data (e.g., Ego4D) and instructional data (e.g., COCO), making it more suitable for the scenarios we are targeting compared to other similar VLMs. Second, Kosmos-2, at the time of submission, is a widely known and comprehensively evaluated model with superior grounding capabilities. Since our model has not been pre-trained on massive grounding data like Kosmos-2, comparing our grounding capability to Kosmos-2 presents a significant challenge, requiring us to explore efficient ways to guide VLMs in learning grounding.

Regarding the reliability of our chosen baselines, both Kosmos-2 and Otter are cited extensively (72 and 110 citations, respectively), open-sourced, and actively maintained. We believe that the research community widely recognizes and reviews these models.

In terms of fairness in comparison to the baselines, for grounding, we contend that our method performs on par with, if not surpassing, Kosmos-2, which also has an additional modality indicating user focus and pre-trained on a large grounding dataset. For Otter, our method not only demonstrates significant grounding improvements but also exhibits better helpfulness. The quality of our coreference question responses is comparable to the direct question responses of Otter's base model, which we believe is a fair comparison.

As for technical novelty and contribution, we argue that our model design is far from trivial. Numerous approaches exist for incorporating gaze data, and our final choice was made after conducting a series of experiments detailed in the ablation study. To further elaborate on these aspects, we have included a detailed illustration of different gaze integration methods in Figure 13. Additionally, our work extends beyond model design, as we strive to establish a method for a new problem. The data annotation pipeline and gaze benchmark construction have not been explored by other works, making it difficult to claim that our work lacks novelty.

Interpretability

We mentioned interpretability twice in our conclusion, as we believe integrating more human intention signals into the model can enhance mutual understanding between the model and the user. This means the model can better interpret the user's intent based on the additional modality, and the user can understand the model's specific response according to the pre-input gaze grounding signal. However, we acknowledge that using the term "interpretability" might lead to confusion with the traditional interpretation concept in machine learning. Therefore, we have decided to temporarily remove this term and reserve it for further evaluation and demonstration. We genuinely appreciate your valuable feedback on this matter.

We would like to express our gratitude to Reviewer 428q for your astute and concise critique of our work. While we appreciate your valuable feedback, we believe that there may be some misunderstandings about certain aspects of our study. To address your concerns, we have provided detailed explanations below and welcome the opportunity for further discussion:

Baseline Methods:

We are grateful for your suggestion to discuss saliency models in our paper. In the revised version, we have included a dedicated discussion of saliency models in Section G.4 and clarified their relationship with our proposed method. However, we would like to emphasize that saliency models, which predict human gaze fixation from images, cannot be directly compared to or used in our approach as baselines. In our task, the ground truth gaze, image, and user query are used as inputs to generate a response, which is a different problem setting.

The most relevant work in this context involves using gaze as input for generating captions, such as the work by Sugano and Bulling [1]. However, these methods predate the era of large-scale language models and are designed for single-purpose tasks like captioning. As they have not been pretrained on massive datasets, comparing their performance to our modern approach would be unfair, as their performance is undoubtedly lower than ours.

Instead, we have conducted an ablation study to investigate different aspects of the model design space, such as input types, gaze integration methods in VLMs, tuning techniques, and data variations. We believe that these analyses provide a more meaningful baseline for comparison and offer valuable insights into the impact of various design choices on performance. And we have update a clear illustration of those baselines in Figure 13, it may be helpful for you to check.

[1] Yusuke Sugano and Andreas Bulling. Seeing with humans: Gaze-assisted neural image captioning. ArXiv, abs/1608.05203, 2016.

Writing and Presentation:

In response to all reviewers' suggestions, we have extensively reorganized, clarified, and rewritten the manuscript, as well as expanded the material in the revised version as we replied in general response. We hope that these changes address your concerns effectively. We remain open to further suggestions and would appreciate any additional feedback on areas that may require improvement or clarification.

Technical Contribution

We respectfully disagree with your assessment of our work's technical valuation and contribution, and we would like to provide the following reasons for your reconsideration:

1. Motivation: As discussed in the revised version, although there is existing research on gaze-related user interfaces and gaze behavior modeling such as saliency models, there has been no investigation into integrating gaze data into modern models and applications. Our work pioneers this area by providing a scalable solution to address this challenge, paving the way for further advancements with high potential.

2. Effort and Innovation: We are the first to define a new gaze-enhanced assistance task that is crucial for the upcoming era of spatial computing. We have invested considerable effort in building a reliable automated data annotation pipeline and providing guidance on potential issues that may arise during this process. Additionally, we have developed a comprehensive solution for scalable training and evaluation of models, established new benchmarks using real-life user data, and offered various insights into model design and training through extensive ablation studies. The work presented is far from trivial and should be recognized as a significant contribution.

3. Contributions to the Research Community: We plan to release both datasets, the automated data annotation pipeline, pretrained model weights, and training code. Given the challenges in obtaining gaze collection equipment, invoking massive APIs for automated annotation, and fine-tuning and evaluating comprehensive ablations on models with billions of parameters, we believe the resources we are providing will be valuable and directly enhance future research in this area.

In light of the points mentioned above, we kindly request that Reviewer 482q reconsider the rating for this work. We also welcome any further questions or suggestions you may have.

We are sincerely grateful for your positive assessment of our work and appreciate the constructive and valuable suggestions you've provided to help us improve it further.

Automatic data annotation process in Section 4.1

We acknowledge that our original presentation of this process was not sufficiently clear, making it challenging for readers to follow. In the revised version, and in accordance with your recommendations, we have provided detailed formulations and explanations of the annotation process alongside Figure 3. We have also included final annotated data examples in Figure 11 to enhance comprehension. Furthermore, to facilitate better viewing, we have replaced the original PNG figure with a PDF format, allowing readers to zoom in as needed.

Voila Perceiver and Resampler Explanation:

In the revised version, we have provided a clearer architectural overview to better illustrate how the Voila Perceiver and Resampler components are integrated within the VLM framework. Additionally, we have marked the different fine-tuning stages on the structure to clarify which components are tuned in each phase. We acknowledge that the original terms gaze linear and gaze attention were not clear, and we have revised the text to specify that they refer to the components tuned during the first stage, without introducing any new formulations. Furthermore, we have rewritten the introductory part of the method section to provide more context on our base structure and input, ensuring a more comprehensive understanding for our readers.

Interpretation of Figures 4 and 5:

In the revised manuscript, we have provided additional insights on interpreting the results, along with explanations for the 'Overall score,' 'Helpfulness,' and 'Grounding score,' which can be found in Figure 12 in the appendix. The primary distinction between these scores is as follows:

1. For Helpfulness, the ground truth answer is provided as a reference, allowing GPT-4 to determine which model's output effectively addresses the user's question.
2. For Grounding, the original captions from COCO and Localized Narratives are given, enabling GPT-4 to decide which one aligns better with the actual content of the image.
3. The Overall score involves both ground truth captions and answers as input, allowing GPT-4 to evaluate which of the two models' answers is superior in general.

Conclusion section:

In the revised conclusion section, we have incorporated a discussion on potential applications, future directions, and limitations of our method. Specifically, our approach can be employed in HMD AR/VR devices to serve as an egoview copilot for everyone, including visually impaired individuals who may not have clear vision but can still use their gaze to convey their intent. Our method offers greater applicability than similar mobile apps that require users to raise their phones to capture a scene for assistance.

The limitations and future directions include:

1. Improving inference efficiency to enable instantaneous responses.
2. Incorporating voice modalities to facilitate seamless interaction.
3. Supporting higher resolutions to accommodate scenarios requiring OCR and screen or UI understanding.

Additionally, we have addressed the hallucination problem and discrepancies with GPT-4V in the appendix. By discussing these aspects, we aim to provide a comprehensive understanding of our method's potential and areas for future improvement.

Other Issues

1. We have merged Section 5.1.2 with Section 4.1 to streamline the presentation of the related content and improve the overall flow of the manuscript.

2. Regarding the phrase "superior ability to handle multi-turn real conversations," we have updated the statement to clarify its meaning and avoid any potential misunderstandings. The updated statement emphasizes that when appending more in-context conversations, our method's performance continues to improve compared to single-turn baselines. This change ensures that readers have a more accurate understanding of the method's capabilities.

We hope that our responses have adequately addressed your questions and concerns. If so, we would greatly appreciate it if you could consider increasing your score to further support our work. Thank you!

We appreciate your recognition of our work's ideas and experiments. We would like to address your concerns as follows:

VOILA-COCO dataset release We thank the reviewers' interest in the VOILA-COCO and VOILA-GAZE datasets. In response to their inquiries, we would like to confirm our intention to release both two datasets upon the publication of our work. Our aim is to foster further research and collaboration within the community by providing these valuable resources. In addition to the datasets, we will also make our annotation pipeline, training code, and model weights publicly available, ensuring that our work is transparent and easily reproducible. To address this concern explicitly, we have included a reproducibility statement in the revised manuscript. We hope that these efforts will contribute positively to the field and facilitate the development of new ideas and methods.

Ethical Considerations

Our experiments have been reviewed and approved by the Institutional Review Board (IRB) at our institute. All participants have provided informed consent, agreeing that their data will be used as part of a publicly available dataset. To ensure privacy, all personal information has been properly anonymized. Additionally, we will conduct a thorough review of any potential privacy issues before releasing the dataset to the public. We are commit to upholding ethical standards in research and protecting the privacy of our study participants.

Existing works on mouse trace approximation

We sincerely appreciate your suggestion to consider the relevant background literature on mouse trace approximation. It is encouraging to discover that, despite differing motivations and experimental settings, these works provide robust support for some of our hypotheses. In response to your feedback, we have conducted a comprehensive survey and included an overarching discussion in Section 2, as well as detailed analyses from various perspectives in Appendices G.3 and G.4.

In our revised manuscript, we have incorporated references to prominent works such as SALICON and BubbleView, and provided a comparative analysis to elucidate the connections and distinctions between our approach and these existing methods. We believe this addition strengthens our paper and offers valuable context for readers. Thank you once again for your valuable input.

Improved Organization and Presentation We recognize that the paper's organization and presentation could benefit from enhancements. In the revised version, we have introduced a comprehensive Figure 4 to elucidate the utilization of gaze data in vision-language models. Furthermore, we have restructured the sections pertaining to data collection and result analysis to improve readability and facilitate a better understanding for our readers.

We hope that our responses effectively address your questions and concerns. If so, we kindly request that you consider raising your score to further support our work. Thank you!