UniAR: A Unified model for predicting human Attention and Responses on visual content

Thank you for the comments and we address each point below.

---

Metrics in Table 3 and Table 6

A: We include the important metrics and baseline methods in Table 3 so that it is more readable with larger font size. We have included reference to Table 6 in Table 3 caption for completeness. Some metrics are not chosen in Table 3, because the baseline papers did not report them, and hence caused empty columns in Table 6.

---

Different comparison methods across different testing datasets

A: For fair comparison, we refer to each baseline’s performance in their original papers and follow the same evaluation protocols / metrics, which can vary on different benchmarks.

---

Why test Deep Gaze II / IIE only on CAT2000 and not on Salicon and OSIE?

A: Deep Gaze II / IIE papers reported results on CAT2000 but not on Salicon validation set / OSIE. We did not re-implement any baseline due to the large number of datasets / tasks we needed to compare.

---

Why not include SalGAN on the Salicon dataset in Table 3 or Table 6?

A: There are two versions of Salicon data, Salicon 2017 (http://salicon.net/challenge-2017) and 2015 http://salicon.net/challenge-2015). All results in our table are on the newer Salicon 2017 dataset, in line with the baselines in Table 3, while SalGAN results were obtained on the Salicon 2015 dataset. The 2015 version differs in fixation ground truth, affecting metrics like NSS scores (higher for Salicon 2015 than 2017), so the results on Salicon 2015 are not comparable to Salicon 2017.

---

Why no SequenceScore for COCO-Search18?

A: The computation of the SequenceScore requires a standardized superpixel segmentation, which is not available after we communicated with the authors. To avoid inaccurate / unfair comparison, we did not include this metric on this dataset.

---

FFM [79] results on COCO-Search18 different from the ones in Table 4?

A: We refer to the results of FFM in Gazeformer [56] paper, which appeared in Table 6 in Appendix. There were some issues regarding the evaluation results on COCO-Search18 due to the misalignment of the ground-truth semantic segmentation map. The authors have updated their results in Appendix, which are the correct version that we cited in Table 4.

---

Why not compare scanpath prediction performance of ALOHA and baselines on COCO-FreeView?

A: We have the scanpath results on COCO-FreeView, but as a relatively new dataset, we did not find a good baseline for scanpath prediction on this dataset, so we did not include this dataset in our table. One relevant work is [1]. However, it didn’t report scanpath prediction performance on COCO-FreeView.

[1] Characterizing target-absent human attention. (CVPR'22)

---

Additional experiments on MIT300 and MIT1003

A: We have a model that did not include MIT1003 as training and directly tested on MIT300, and the results are suboptimal. However, since many methods on MIT300 benchmark dashboard are trained on MIT1003, so for a fair comparison, we will train a model on MIT1003 and report the results on MIT300.

---

The experiments do not clearly demonstrate the actual benefits of unifying the attention, rating, and scanpath tasks

A: The immediate benefit of an unified model is that only one model is needed instead of separate models for each task, leading to easier model serving with better performance. Furthermore, there are strong connections between attention and subjective experiences like aesthetic preference, subjective interpretation, emotional response, or ease of information finding (e.g., often focused attention means easy, while scattered attention means hard). To this end, we collect and will release a new dataset which contains both scores (easiness scores for question answering tasks) and gaze heatmaps (while performing question answering tasks) on digital images (designs, UIs, etc) , and show that the score and entropy of gaze heatmaps are correlated, to better motivate the unified model. Please see more details in Figure 3 and 4 in the rebuttal PDF.

---

How are the three predictors chosen for each task?

A: During training, we specify the task in the text prompt and compute gradients only using the loss of that task (e.g., scoring task if “OUTPUT: score” is in the prompt). During inference, the task info (i.e., “OUTPUT: score”) in the prompt will inform the model which task to perform and we retrieve results from the corresponding predictor.

---

Differences between the salience map and the importance map?

A: The two tasks use the same heatmap L2 loss and only differ in data. Note that the two tasks will have different text prompts, namely “OUTPUT: saliency heatmap” for saliency, and “OUTPUT: importance heatmap” for importance.

---

Invalid output format for scanpath prediction?

A: We first split the output sequence by the separator (“and” in our case). Then each component should contain two coordinates of x and y. If a <x, y> coordinate is invalid (e.g., only x but no y value), we skip that coordinate. All predicted scanpaths are valid after the models were finetuned.

---

Why text generation as a scanpath predictor?

A: Text decoder can generate arbitrary length of tokens (up to max length) and is suitable for scanpath which also has variable length. As the vision-language model is pretrained on coordinate prediction task the decoder learned the notion of coordinate token and performed well for predicting scanpath, which mainly consists of coordinates. As a unified model, we hope the model can generalize to different kinds of sequences, which can be predicted by the text decoder.

---

How is the model fine-tuned on various datasets, which parts are freezed?

A: All model weights are finetuned and no parts are frozen. We randomly sampled batches from all the datasets during training (more details in Section 4.2).

Thank you for the comments and we address each point below.

---

Model's limitations in real-world situations, such as how its performance is in dynamic environments or its adaptability to changes in user behavior over time.

A: Current model does not consider dynamic environments or changes in user behavior over time, which is indeed a limitation. We will add this limitation to the discussion section. Please also see our response to the question of “Stay up to date” from reviewer FgZM.

---

ALOHA has 848 million parameters. Training such a large model requires significant computational resources, which may not be accessible to most researchers or practitioners.

A: We acknowledge the resource requirement might be difficult for researchers without access to enough GPUs / TPUs. There are techniques to reduce model training cost, like LoRA, or Parameter-Efficient Fine-tuning (https://huggingface.co/blog/peft), which could be employed with much less resources. However, using such techniques would be beyond the scope of this work. We will include this info in the paper and develop models with lower resource requirements in our future work.

---

How can it be used to optimize elements, such as user interfaces, graphic designs, and content creation?

A: ALOHA can help optimize content in several ways. For example, the predicted saliency heatmap can be used to remove the distracting areas in visual content, similar to “Deep Saliency Prior for Reducing Visual Distraction”: https://arxiv.org/abs/2109.01980. If the content is generated by a generative model, ALOHA’s predicted score can be used as a reward score to improve the generative model, with learning from human feedback/preference method, such as DPOK (https://arxiv.org/pdf/2305.16381) and DRaFT (https://arxiv.org/abs/2309.17400).

---

Are there any potential biases in the current model's predictions?

A: Please see the global rebuttal.

---

Format issues

A: We will make the edits according to the feedback.

We appreciate you bringing the ethical concerns to our attention. We recognize that such concerns are common with machine learning models, particularly those involving user preference and behavior modeling. We are committed to addressing these issues to the best of our ability by expanding the ethics and limitations section.

---

Uniform vs diverse human preferences

We acknowledge that models carry the risk of converging towards a more uniform notion of preference, a concern shared by all machine learning models. To promote visual diversity in our model, we propose,

1. Hybrid Approach: Initially, the model should be used in a hybrid manner, providing insights without replacing human decisions in web optimization, and
2. Personalized Models: Develop personalized models based on our initial unified model. This approach will help generate more diverse predictions based on user attributes.

---

Demographics of annotators

Please see our response in the global comment.

---

Aligning with human preference

While our model should be approximately aligned with human preferences, we recognize that using it as a reward model may lead to reward hacking. We plan to incorporate techniques [1, 2] for mitigating this when we use it as a reward mode.

[1] Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation
[2] ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization

---

Properly using ALOHA model

Like many technologies, human gaze prediction models could potentially be misused. Without regulations, such as unrestricted access to users without credentials and irresponsible data collection, the risk can be increased. To ensure the proper usage of our model, we propose,

1. Placing restrictions on model access to prevent unauthorized applications or users;
2. Adhering to strict ethical guidelines on data collection, and actively monitoring the collection process at scale; and
3. Being transparent about our model's capabilities and limitations. We believe that keeping humans in the loop and only using the model prediction as a reference but not replacing humans is important.

Internally, we strive to follow the ethical guidelines of our institution, ensuring that the model usage remains controlled and responsible. This has also caused us to be cautious about making our model publicly available.

---

Blind and low-vision users

Our model can benefit blind and low-vision users for screen users by highlighting the most important areas of a webpage via heatmap predictions and making them more accessible via screen readers. We recognize that the current training data may have limitations in representing the full spectrum of user experiences. We plan to enhance the inclusiveness by,

1. Multi-Modal Preference Modeling: We're developing our model to incorporate not just visual cues, but also how users interact with content through screen readers, voice commands, and other assistive technologies;
2. Collaboration with Accessibility Experts: We plan to collaborate with accessibility experts and organizations representing blind and low-vision users for future iterations of our work.

---

Stay up to date

Staying up to date is essential for models, and can be achieved by fine-tuning with more recent data. Continual learning techniques also support this idea [1]. However, updating the training data is out of the scope of our paper, as we want to focus on developing the first unified model on the diverse user modeling tasks.

[1] A comprehensive survey of continual learning: theory, method and application

---

Error Analysis

We analyzed 50 error examples in the Koniq-10K dev set and found some interesting patterns, see examples in Figure 1 in our rebuttal PDF. The first example shows the most common error category when our model predicts a higher score for colorful but blurry images. The second example shows another common model error where a low-light image gets a higher prediction score. For the last black-and-white example, on the contrary, our model predicts a lower score than the groundtruth. These errors demonstrate that human preferences for image aesthetics can depend on diverse factors including clear focus, correct lighting, and artistic style.

Similarly, for heatmap prediction, we analyzed the 30 examples with the lowest NSS scores on the Salicon validation set. The example in Figure 2 in the rebuttal PDF demonstrates the most common error category where the groundtruth contains a more scattered gaze heatmap. In such cases, our model prediction might not focus on the right objects.

We will conduct more error analysis on other model predictions and include this info in the paper.

---

New content to include

Happy to incorporate your recommendations and include a discussion on the following action items:

1. The diversity / demographics of annotators
2. Model failure cases and discussions
3. A model card

---

Model’s name

We are happy to choose a new name from alternative names for our model, for example,

1. GLAM: From Glances to Likes: A Unified Model for Understanding Human Visual Behavior
2. TOTORO: from attention TO likes TO human RespOnses – a unified model of human visual behavior
3. HABITAT: modeling Human Attention and Behavioral InTeractions Across diverse visual content using a multimodal Transformer

Thank you for the comments and we address each point below.

---

The writing can be improved in several places

A: We will edit the paper according to the suggestions.

---

How hyperparameters are tuned

A: We did a training-validation split on our larger datasets and then tuned the hyperparameters (e.g., learning rate, batch size, dropout rate, loss weights of the 3 heads) on the validation set through grid search. The model was then trained with the full training data with the chosen hyperparameters.

---

What happens with images larger than 512x512?

A: We resize the images to have a max height or width of 512 and center-pad the smaller dimension to 512.

---

Code will not be made public, which is a major concern in the current reproducibility crisis in ML

A: Given the sensitivity of this topic, and our concern that the code may be misused if made public (see comments from Reviewer FgZM and our rebuttal on "Properly using ALOHA model"), our institution has a strict policy on opening source code in this area. But to help advance the research, we will provide enough details (including all the info in the rebuttal), all necessary communication and support for o researchers in this area to reproduce our work, who we believe will make use of this technique appropriately.