Eye Fairness: A Large-Scale 3D Imaging Dataset for Equitable Eye Diseases Screening and Fair Identity Scaling

Question: Can the authors also provide further breakdown statistics of the intersectional groups, e.g. black females?

Response: The statistics of the intersectional groups (race + gender) are presented in the following table. We have added more statistics of intersectional groups to the revised manuscript.

Question: The dataset also contains some other attributes such as preferred language. I'm not very sure how this is related to eye diseases and fairness, e.g. do non-English speaking patients get lower AUC? But how does the eye imaging model perceive the speaking? Also, the authors did not evaluate the performance of different subgroups of preferred language and marital status.

Response: Our dataset is from the United States, where a patient's primary language can be potentially indicative of their socioeconomic status. This factor could impact the fairness of AI models. For instance, patients with lower incomes, who might seek hospital care later and often present with multiple confounding eye/systemic diseases, could influence the predictions of deep learning models. The research [2] has shown that non-English speakers often have diseases at more advanced stages when they seek medical help.

Following the suggestion, the experimental results of preferred language and marital status are reported in the first and second tables below, respectively. FIS demonstrates consistent superiority over the baseline model, showcasing enhanced generalizability across metrics such as AUC, Mean PSD, and Worst-case AUC. Notably, the AUC gap when compared to EfficientNet is narrower than that with EfficientNet+FIS. This indicates that FIS not only enhances the worst-case AUC but also improves the best-case AUC. The results, along with a relevant discussion, have been included in the revised manuscript.

References:

[1] Zong, Yongshuo, Yongxin Yang, and Timothy Hospedales. "MEDFAIR: Benchmarking Fairness for Medical Imaging." The Eleventh International Conference on Learning Representations. 2022.

[2] Halawa, Omar A., et al. "Race and ethnicity differences in disease severity and visual field progression among glaucoma patients." American journal of ophthalmology 242 (2022): 69-76.

Question: The authors didn't tune the hyper-parameters of the baseline methods but only used the default HPs, which leads to unfair comparisons. The baseline methods are not designed for medical imaging, so if applied to a different setting, the hyperparameters should be carefully tuned to get the best performance. Especially there're adversarial training method and self-supervised pretraining method that are very sensitive to the HPs.

Response: We have meticulously tuned the hyperparameters for all baseline models presented in this paper. It's important to note that without such tuning, the baseline approaches yield inferior results. To enhance clarity and provide a comprehensive understanding, we have included a detailed discussion on this aspect in the revised manuscript.

Question: The definition of performance-scaled disparity (PSD) is not clear. It says in the paper: "PSD metrics are calculated as the standard deviation of group performance or absolute maximum group performance difference divided by overall performance." Which one did the authors use? And what does Mean PSD and Max PSD mean?

Response: PSD, performance-scaled disparity, aims to use a single scaler to quantify group disparities based on the AUC metric. The PSD formula is defined as follows:

$$\text{Mean PSD} = \frac{ \text{STD}(\text{Group-wise AUCs}) } {\text{Overall AUC}}$$ $$\text{Max PSD} = \frac{ \text{Max}(\text{Group-wise AUCs}) - \text{Min}(\text{Group-wise AUCs}) } {\text{Overall AUC}}$$

We have revised the manuscript accordingly to improve the clarity.

Question: Also, regarding the metrics, authors can provide the worst-case AUC and the AUC gap between best-performing and worst-performing groups besides the current overall AUC and group-wise AUC. It would be clearer to directly look at the AUC gap to validate the effectiveness of the proposed methods. Additionally, what are the advantages of using PSD instead of the AUC gap? Consider an extreme case: for model 1, two groups have an AUC of 25% and 51% while for model 2, two groups have an AUC of 50% and 100%. According to the authors' definition,, but can you say model 2 is fairer than model 1 as it's the smaller the better? I know the AUC usually is higher than 50% but this is just an example and I think there are many similar cases in regular scenarios.

Response: We thank the reviewer for the comment. The table of the performance on Race for AMD detection is as follows. FIS continues to demonstrate superior performance in fairness metrics. Furthermore, as outlined in the definition of Max PSD, Max PSD is closely correlated with AUC gap that is part of Max PSD. We fully agree that the PSD metric has its own limitations in representing the model fairness in scenarios outlined by the reviewer. The advantage of PSD is that: when the AUC gap is the same for two models, PSD considers the model with a higher AUC performance as the fairer model, while the AUC gap itself in this case considers the two models are equally fair. Realizing the fact that each fairness metric may have its limitations, we also calculated other fairness metrics in our manuscript such as DPD and DEOdds.

Question: I think Table 1 in this paper is taken from Table 1 in [1] without reference as (1) the number of images of each dataset is not the original number but the number after preprocessing by [1] and (2) the so-called ADNI 1.5T is a subset of the large ADNI dataset [2] extracted by [1].

Response: We thank the reviewer for pointing this out. We have cited the paper of MedFair [1] and clarified that ADNI 1.5T is a subset of the large ADNI dataset extracted by [1] in the revision.

Question: At the time I wrote this review, the GitHub repo authors provided was empty.

Response: We apologize for the delay in making our dataset and code available. They were released for peer review on October 16th at https://github.com/anonymous4science/EyeFairness. To comply with IRB requirements, we meticulously reviewed our dataset including 30,000 patients to ensure the removal of any identifiable information, such as names, locations, and dates, before the data release.

Question: Minor: the current citation style makes reading difficult. The author should use (Deng et al. (2009)) instead of Deng et al. (2009), i.e. \citep instead of \cite.

Response: We have fixed our citation style in the revised manuscript.

Question: Discuss a pooling of other public datasets from centres with different scanners / from different countries

Response: As shown in Table 1 in the manuscript, for eye diseases, there are no public fairness datasets available that have comprehensive demographic identity attributes as in our dataset including age, gender, race, ethnicity, preferred language, and marital status. Our dataset is also the only public dataset with 3D eye imaging data measured by optical coherence tomography that enables 3D fairness learning modeling. For instance, the ODIR-2019 dataset only contains demographic identity attributes of age and gender, which is insufficient for comprehensive fairness learning. Another example is that the AMD-OCT dataset only contains the age attributes. Therefore, given the uniqueness of our dataset, which covers three major eye diseases, including age-related macular degeneration, diabetic retinopathy, and glaucoma, affecting 380 million patients worldwide, contains a large number of patients of 30,000 in total and six demographic attributes (age, gender, race, ethnicity, preferred language, and marital status) for fairness learning, there are no existing public datasets that can be used to pool with our dataset. In addition, we would like to stress that our dataset is from a prominent eye hospital located in a region with a large population with high diversities.

Question: Clarify the 2D/3D data split wrt. patients.

Response: As detailed in Section 5.1, our dataset utilizes a patient-level split. It comprises one 2D fundus image and a set of 3D OCT B-Scans for each patient. This information has been further clarified in the revised version of our manuscript.

Question: Expand the baselines.

Response: We are grateful for the reviewer's insightful comment. We have included the state-of-the-art fairness method, FairAdaBN [1], in our experiments. The results for AMD detection based on race are presented below. We tune the hyperparameters of FairAdaBN to ensure optimal performance and utilize its best-reported backbone network, ResNet. However, it's important to note that FairAdaBN's dependency on the backbone network architecture restricts its generalizability and ability to leverage more powerful pre-trained models, such as EfficientNet. We have updated the experiment in the revised manuscript accordingly.

Question: Correct the statement of non-existing 3D augmentation strategies.

Response: We appreciate the reviewer’s contribution to this discussion. Rather than delving into general 3D augmentation strategies, our discussion focus is on augmentation methods specifically pertinent to fairness. To our knowledge, there is a lack of established fairness methods demonstrating that algorithmic fairness can be improved through 3D augmentation. The application of current 3D augmentation techniques, such as cutmix, random crop, and mixup, raises concerns. These methods could inadvertently lead to inaccurate disease detection by obscuring or eliminating abnormal lesions in the 3D space. This discussion has been included in the revised manuscript.

References:

[1] Zikang Xu, Shang Zhao, Quan Quan, Qingsong Yao, and S. Kevin Zhou. 2023. FairAdaBN: Mitigating Unfairness with Adaptive Batch Normalization and Its Application to Dermatological Disease Classification. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part II. Springer-Verlag, Berlin, Heidelberg, 307–317.

Question: Minimal ablation analysis temperature scaling parameter, FIS group/individual scaling trade-offs actually not very consistent (Figure 2) / The main contribution of a Fair Identity Scaling (FIS) model with learnable group weights and past individual loss data, might have been analyzed in greater details as to the temperature scaling parameter (alongside fusion weight).

Response: As suggested by the reviewer, we have included a table below showcasing the experimental results for AMD detection with different temperature values, specifically focusing on the attribute of race in fundus images, with the hyperparameter c set to 0.5. This analysis has been incorporated into the revised version of our paper.

Question: Side-effects of fairness on adjacent demographic features not considered/Returning to fairness as a focus, the presented analysis does not appear to be concerned with the impact on fairness amongst other demographic features, when FIS is applied to a particular feature. For example, when FIS is applied on race (as in Tables 2 & 3), what is the effect on results stratified with other features such as gender, ethnicity, age etc.? This appears particularly relevant since the other demographics may be no less significant for the consideration of fairness/equity purposes.

Response: We thank the reviewer for the insightful comment. Focusing on a single attribute can make the cause-and-effect relationship clear. However, as the reviewer highlighted, examining the impact of applying FIS to one attribute on the performance metrics of other attributes is also valuable. To this end, we have conducted an analysis where FIS is specifically applied to race in the context of AMD detection, and we report its effects on gender and ethnicity performance. For our FIS model specifically addressing the fairness for race, the model fairness quantified by PSD values for gender and ethnicity attributes is also improved. It's important to note that the individual scaling in Equation (1) is attribute-independent and is designed to adaptively influence the significance of each sample during back-propagation. This approach could be a contributing factor to the observed improvements in fairness.

Question: Conceptually, the distinction between improvements in “general (AUC) performance” and “fairness” might have been further considered. In particular, from the results in Tables 2 to 7, FIS appears often capable of not only improving “fairness” (i.e. minimizing performance-scaled disparity), but instead often improves performance in all groups (and overall). As such, a natural question might be whether FIS might be used with arbitrary groupings of data, to improve classifier performance.

Response: The observed improvement in both fairness and performance can be attributed to the individual scaling mechanism in FIS, which considers the losses of samples within a mini-batch to determine their relative importance for back-propagation. This approach enables the training process to give priority to samples with larger losses, effectively balancing the learning focus across the dataset.

Following the suggestion, we present the experimental results for arbitrary groupings in AMD detection, as detailed in the table below. We observe that combining race, gender, and ethnicity enhances performance, although the resulting mean PSDs are not consistently optimal compared to other methods. This highlights the intricate and intriguing interplay between attributes. We plan to delve deeper into this aspect in our future research.

Question: The trend of hyperparameter in Figure 2 is not quite clear since it kind of alternates for both AUC and mean PSD. The authors might need to further discuss the choice of, which still seems rather empirical given the current visualization.

Response: We adhere to the Keep It Simple and Stupid (KISS) principle by setting the default hyperparameter c to 0.5, striking a balance between individual and group scaling to achieve a trade-off between effectiveness/accuracy and fairness. $c = 1$ represents exclusive group scaling, while $c = 0$ denotes solely individual scaling. As corroborated by the analysis in Figure 2, our choice of c at 0.5 aligns with empirical results, demonstrating that higher AUC coincides with lower mean PSD, compared to other c values.

Question: The experimental results section is currently flooded with numbers and quite hard to follow.

Response: We appreciate the reviewers' feedback on the comprehensiveness of our study, which encompasses a wide range of factors such as identity attributes, metrics, and methods, and includes extensive experiments across three eye diseases. Recognizing the importance of thoroughness in our experimental results, we have enhanced the readability of our experimental section by incorporating visualizations in the revised manuscript.