Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning

W: "The reliance on eye-gaze data from...where the collection of eye-gaze data is not practical."

We thank the reviewer for this great comment. In fact, when designing experiments for zero-shot classification and zero-shot image-text retrieval, EGMA took this issue into account. Experimental validation has shown that after multimodal training on datasets with eye-gaze data, the model performs well on other datasets without eye-gaze data (Tab. 1&2&3 in main paper). Furthermore, with advancements in eye-gaze data collection systems [1-4], acquiring multimodal data with eye-gaze has become increasingly feasible. Compared to having radiologists spend additional time and effort annotating refined labels, eye-tracking allows for data collection during their routine diagnostic work, minimizing disruption and reducing annotation burdens. Additionally, some studies have demonstrated [3,5,6] that eye-gaze data can achieve performance comparable to refined annotations. Therefore, we propose initially pre-training on scenarios where eye-gaze data is more easily collected and then fine-tuning on more challenging datasets. This approach is similar to current mainstream multimodal pre-training models, where a robust model is trained on a broadly available dataset and then fine-tuned on other datasets to perform downstream tasks. The performance of EGMA in this work supports the feasibility of this approach.

Q1: "How does the quality and quantity of eye-gaze data affect the performance of the EGMA framework?"

We thank the reviewer for this great question. The quality of eye-gaze data is crucial for model performance. In this work, we first filter out interfering data such as blinks and saccades by using only the fixation data from the eye-tracking recordings. We then generate a 2D attention heatmap from these fixation points to further mitigate the influence of interfering data. Figure R1 in the attached PDF shows an example of denoising in MIMIC-EYE dataset. Additionally, in the design of the EGMA framework, we use the EGM module to further filter out potentially problematic regions. For catastrophic errors could happened, the MIMIC-EYE dataset provides a robust solution by involving multiple expert radiologists in data collection. This approach ensures overall data quality and allows for the compensation of errors made by one or two radiologists with correct data from other radiologists. Figure R2 in the attached PDF shows an example of this compensation.

Regarding the issue of the quantity of eye-gaze data, we conducted extensive comparisons in the ablation study of this work (Tab. 4 in main paper). Specifically, we evaluated the model performance with 1%, 5%, 10%, and 50% of the eye-gaze data. We found that with a small amount of data, such as 1% (approximately 37 samples), the model's performance improvement was limited, which is attributed to insufficient auxiliary information. As the proportion of eye-gaze data increased, the performance of EGMA improved progressively across the three datasets. This aligns with the expectation that providing more supervised information allows the model to learn better feature representations and consequently enhance performance on the datasets.

Q2: "Are there specific thresholds or optimal amounts of eye-gaze data required to achieve the best results?"

We thank the reviewer for this great question. For our EGMA framework, having more eye-gaze data is certainly beneficial, similar to other types of annotations. Additionally, EGMA is designed to handle situations where eye-gaze data may be limited, as it supports training with both eye-gaze and non-eye-gaze data. In our ablation experiments (Tab. 4 in main paper), we also investigated the impact of the proportion of eye-gaze data on model performance. Our goal was to demonstrate that even with limited eye-gaze data (e.g., using only 185 samples), EGMA's performance can still improve steadily, highlighting the framework's flexibility and effectiveness.

References:

[1] Stember, J.N., et al. (2020). Integrating eye tracking and speech recognition accurately annotates MR brain images for deep learning: proof of principle.

[2] Khosravan, N., et al. (2019). A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning.

[3] Ma, C., et al. (2023). Eye-gaze-guided vision transformer for rectifying shortcut learning.

[4] Men, Q., et al. (2023). Gaze-probe joint guidance with multi-task learning in obstetric ultrasound scanning.

[5] Wang, S., et al. (2022). Follow my eye: Using gaze to supervise computer-aided diagnosis.

[6] Ma, C., et al. (2023). Rectify vit shortcut learning by visual saliency.

W1: We greatly appreciate the reviewer's suggestion. Supervision information used for localization and segmentation tasks, such as bounding boxes and masks, is stronger than the labels used for classification tasks. Additionally, the amount of refined manual annotation required for localization and segmentation is relatively limited. Therefore, we believe that if EGMA performs well on classification tasks with relatively weak supervision and across multiple classification datasets, it is likely to also excel in localization and segmentation tasks. Furthermore, the visualization results in our work already provide preliminary evidence of EGMA's localization capability. Finally, thank the reviewer again for this suggestion. Conducting additional validation experiments is indeed a promising direction for our future work.

W2: We thank the reviewer for this great comment. Privacy concerns in eye-gaze data are often overlooked by researchers. Studies [1,2] have shown that eye-gaze data can reveal private information such as the collector's personality traits, age, gender, and mental state, making privacy protection crucial. One approach [3] involves adding noise to the original eye-gaze data to obscure some private information, but this can compromise the quality of the data. Another commonly used method [4], which is employed in EGMA, is to convert the original eye-gaze data into a two-dimensional attention heatmap. This heatmap retains rich visual information while mitigating the risk of privacy leaks compared to raw eye-gaze data. Therefore, the EGMA framework protects the privacy of the participants during the data preprocessing phase.

W4: We thank the reviewer for this great comment. In fact, when designing experiments for zero-shot classification and zero-shot image-text retrieval, EGMA took this issue into account. Experimental results show that after multimodal training on datasets with eye-gaze data, the model performs well on other datasets without eye-gaze data. Furthermore, with the advancement of eye-gaze data collection systems [5-8], collecting multimodal data with eye-gaze has become increasingly feasible. Therefore, we propose a strategy where pre-training is conducted in scenarios where eye-gaze data is more easily collected, followed by fine-tuning on more challenging scenarios. This approach is similar to mainstream multimodal pre-training models, where a strong pre-trained model is first trained on a widely available dataset and then fine-tuned on other datasets for downstream tasks. The performance of EGMA in this work supports the feasibility of this approach.

W3&Q1: We thank the reviewer for this great question. In fact, we have discussed this issue in the ablation study of our paper (Tab. 4 in main paper). Given the current scarcity of eye-gaze data, EGMA was designed to support training with both eye-gaze and non-eye-gaze data within the same batch. In Tab. 4, we conducted comparative experiments with four different proportions of eye-gaze data: 1%, 5%, 10%, and 50%, corresponding to approximately 37, 185, 370, and 1848 instances with eye-gaze data in the training set, respectively. It can be observed that with 1% of eye-gaze data, the model shows only slight improvement on the CheXpert5x200 dataset due to limited auxiliary information. As the proportion of eye-gaze data increases, the performance of EGMA gradually improves across the three datasets. This aligns with the expectation that providing more auxiliary information allows the model to learn better feature representations and thus enhance its performance on the datasets.

Q2: We thank the reviewer for this great question. Our EGMA model is designed based on the CLIP architecture and supports the co-training of both eye-gaze and non-eye-gaze data, which provides it with significant flexibility. Recently, a study has seamlessly integrated eye-tracking technology into ultrasound imaging [8], where ultrasound images, eye-gaze data, and diagnostic texts were collected. EGMA can be easily adapted to this dataset with minimal modifications, primarily involving the replacement of the encoder with one pre-trained on ultrasound images.

References:

[1] Kröger, J.L., et al. (2020). What does your gaze reveal about you? On the privacy implications of eye tracking.

[2] Katsini, C., et al. (2020). The role of eye gaze in security and privacy applications: Survey and future HCI research directions.

[3] Steil, J., et al. (2019). Privacy-aware eye tracking using differential privacy.

[4] Liu, A., et al. (2019). Differential privacy for eye-tracking data.

[5] Stember, J.N., et al. (2020). Integrating eye tracking and speech recognition accurately annotates MR brain images for deep learning: proof of principle.

[6] Khosravan, N., et al. (2019). A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning.

[7] Ma, C., et al. (2023). Eye-gaze-guided vision transformer for rectifying shortcut learning.

[8] Men, Q., et al. (2023). Gaze-probe joint guidance with multi-task learning in obstetric ultrasound scanning.

Q1&W1: "Please clarify how the Mean(Max())..." "Despite the authors’ best efforts, Fig. 2C and Fig. 2D...allows for gaze-free supervision)"

We thank the reviewer for this great question and apologize for any confusion caused by casual description in our paper. First, to clarify the role of fine-grained alignment, we have rewritten the Mean(Max()) operation in Sec. 3.2 of main paper as follows:

$$\hat{z}^I_k= \frac{1}{m} \sum^{m}_{j=1}[\underset{j}{max} {(x^{P2S}_k)} ], ~~ 1 \leqslant j \leqslant m ~~ (1)$$ $$\hat{z}^T_k= \frac{1}{n} \sum^{n}_{i=1}[\underset{i}{max} {(x^{S2P}_k)} ], ~~ 1 \leqslant i \leqslant n ~~ (2)$$

In the above equations, $x_k^{P2S} \in \mathbb{R}^{n \times m}$ and $x_k^{S2P} \in \mathbb{R}^{m \times n}$ are similarity matrices between local features within a single instance, and these matrices have been optimized with eye-tracking data as shown in Eq. 4 in main paper. We know that CLIP's contrastive loss can align features well within a batch by computing the similarity between global features of the samples. To continue using this contrastive loss on top of instance-level local features, we need the global similarity between the corresponding text and image for each instance. In this work, instead of directly using the global features output by the encoder as CLIP does, we first calculate local feature similarities and then derive global similarity from the local similarity matrix. Specifically, to compute the global similarity between an image and a diagnostic text within a batch, we first calculate the similarity matrix $x_k^{P2S}$ between the local features of image patches and sentence tokens using Eq. 3 in main paper. Then, we take the maximum value of each column of this matrix, meaning each sentence token gets a similarity score with its most related image patch. Finally, we average these values to obtain the global similarity refined by local similarity. The calculation of the fine-grained global similarity from diagnostic text to image within a batch is similar. This discussion and results will be incorporated into the revised version of the paper.

Q2&W2: "I would also like to understand the motivation for the EGM module..." "why the cross-modality mapping is necessary."

We thank the reviewer for this great question. In fact, the EGF and EGM modules proposed in our work are similar to the diagnostic process of radiologists. Many studies [1-5] have pointed out that radiologists perform a global search first and then conduct a detailed examination of the local areas when suspicious lesions are found. Firstly, the EGF module of EGMA corresponds to the global search by radiologists, aligning the local image patch features with the individual sentence features. Secondly, EGM uses the key values from the local similarity matrix computed in the first step as weights, focusing the alignment on certain important image patches and texts, akin to the detailed observation stage after identifying suspicious lesions. Thus, by mimicking the real diagnostic behavior of radiologists, our EGMA further enhances the model's multimodal processing capability and diagnostic accuracy. This discussion will be incorporated into the revised version of the paper.

L: "In this reviewer’s opinion, the framework proposed in this paper could be easily applied to non-medical areas as well as its design is domain-agnostic."

We greatly appreciate the reviewer's suggestion. In fact, thanks to the inherent flexibility of the EGMA model, it is well-suited for multimodal alignment tasks involving natural images, such as in human-robot interaction and control as well as for education/training purposes. Therefore, we believe this is a promising direction for future expansion. Once again, thank the reviewer for this valuable suggestion. This discussion will be incorporated into the revised version of the paper.

References:

[1] Nodine, C.F., et al. (1987). Using eye movements to study visual search and to improve tumor detection.

[2] Swensson, R.G.. (1980). A two-stage detection model applied to skilled visual search by radiologists.

[3] Kahneman, D.. (2003). A perspective on judgment and choice: Mapping bounded rationality.

[4] Kundel, H.L., et al. (2007). Holistic component of image perception in mammogram interpretation: gaze-tracking study.

[5] Drew, T., et al. (2013). Informatics in radiology: what can you see in a single glance and how might this guide visual search in medical images?

W1&Q1: We thank the reviewer for these great questions and apologize for any confusion caused by the lack of discussion in our paper. Indeed, noise and errors in eye-gaze data are as common as noise in images, and these can all affect the model's final performance. In this work, the errors in eye-gaze data primarily stem from two factors: involuntary saccades and microsaccades of the radiologists' eyes [1], and subjective fixation errors [2].

Human eye movements can be categorized into two main types: saccades and fixations [1]. Saccades are the rapid movements between different areas of the visual field and are generally considered noise data that do not involve cognitive processes. Fixations, on the other hand, are brief pauses of the eyes on a small area and are regarded as the primary cognitive behavior. Additionally, due to the structure of the eye, fixations are accompanied by microsaccades, causing the fixation point to drift within the fixation area. Thus, microsaccades are also a major source of noise in eye-gaze data. Fortunately, eye-tracking technology has evolved significantly over the past century, and noise reduction techniques for eye-gaze data have become highly advanced. Currently, all commercial eye-tracking devices come with preprocessing software that uses adaptive methods (e.g., [3]) to filter out noise, greatly facilitating the use of eye-tracking technology. In the attached PDF, Figure R1 shows an example of denoising process in MIMIC-EYE dataset.

Additionally, due to variations in radiologists' expertise and cognitive levels, fixation data may contain errors, such as the reviewer mentioned, "occasionally shift to locations not exactly on the objects."

For the minor shift error of fixations, where the duration of incorrect fixations is small compared to the overall duration, their values in the attention heatmap are low, thus having minimal impact on the mlce loss utilized in our work (Eq. 4 in main paper). In our EGMA, EGF and EGM modules (Sec. 3.2 and Sec. 3.3 in main paper) can also filter out this minor errors in the heatmap. For the significant fixation errors, MIMIC-EYE data has been cleaned to address this issue at the first time. Specifically, these data were first reviewed by professionals to exclude inconsistent diagnoses and fixation errors. Moreover, the MIMIC-EYE dataset includes eye-gaze data from six radiologists. Even if one or two radiologists' fixations are incorrect, the correct fixations from the other radiologists under the same semantic context can mitigate the adverse effects. In the attached PDF, Figure R2 shows an example of this compensation.

As the reviewer pointed out, effectively identifying and mitigating errors in radiologists' diagnostic behavior during model training is crucial, and this will be a focus of our future work. This discussion and results will be incorporated into the revised version of the paper.

W2: We thank the reviewer for this insightful question and apologize for any confusion caused by the lack of relevant discussion in our paper. In fact, numerous studies have investigated and proven the close relationship between radiologists' gaze behavior, image content, and diagnostic results. For example, studies [4-9] have found that radiologists fixate more on diseased areas than on healthy ones. Moreover, they discovered that novice radiologists, due to their lack of experience, repeatedly look at diseased areas, resulting in even more fixations in these areas than those of expert radiologists. Figure R3 in the attached PDF shows some comparison cases.

Additionally, the abnormal regions with more fixations have higher weights in the attention heatmap. Our EGMA model can leverage these weights, along with image content and text, to learn better disease diagnosis capabilities and feature representations. This discussion and results will be incorporated into the revised version of the paper.

W3: We are grateful for the reviewer’s attentive analysis and helpful feedback. We have revised the formulas as follows: (1) Change "COS" (Eq. 1 and Eq. 3 in main paper) to lowercase, as shown in Equations 1 and 2 below:

$$s_{k,l}^{I2T} = cos(z_k^I, z_l^T), \ s_{k,l}^{T2I} = cos(z_k^T, z_l^I) \quad 1 \leqslant l \leqslant b ~~ (1) \\$$ $$x_k^{S2P} = cos(S_k^j, P_k^i), \ x_k^{P2S} = cos(P_k^i, S_k^j) ~~ (2)$$

(2) Change "Mean/Max" (Sec. 3.2 in main paper) to the following Equations 3 and 4:

$$\hat{z}^I_k= \frac{1}{m} \sum^{m}_{j=1}[\underset{j}{max} {(x^{P2S}_k)} ], ~~ 1 \leqslant j \leqslant m ~~ (3)$$ $$\hat{z}^T_k= \frac{1}{n} \sum^{n}_{i=1}[\underset{i}{max} {(x^{S2P}_k)} ], ~~ 1 \leqslant i \leqslant n ~~ (4)$$

These revised formulas will be incorporated into the revised version of the paper.

References:

[1] Chamberlain, L.. (2007). Eye tracking methodology: theory and practice.

[2] Brady, A.P.. (2017). Error and discrepancy in radiology: inevitable or avoidable?

[3] Nyström, M. and Holmqvist, K.. (2010). An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data.

[4] Nodine, C.F., et al. (1996). Nature of expertise in searching mammograms for breast masses.

[5] Mallett, S., et al. (2010). Tracking eye gaze during interpretation of endoluminal three-dimensional CT colonography: visual perception of experienced and inexperienced readers.

[6] Voisin, S., et al. (2013). Investigating the association of eye gaze pattern and diagnostic error in mammography.

[7] Giovinco, N. A., et al. (2015). A passing glance? Differences in eye tracking and gaze patterns between trainees and experts reading plain film bunion radiographs.

[8] Van der Gijp, A., et al. (2017). How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology.

[9] Tourassi, G., et al. (2013). Investigating the link between radiologists' gaze, diagnostic decision, and image content.