FairSeg: A Large-Scale Medical Image Segmentation Dataset for Fairness Learning Using Segment Anything Model with Fair Error-Bound Scaling

Thank you so much for your review and the insightful comments.

ICLR might not be the best place for this paper. Other medical journals or conferences would be more suitable.

Our primary contribution centers on fairness in machine learning, rather than on medical imaging itself. We regard medical imaging as a crucial area for studying fairness, given its human-centric nature and the potentially severe consequences of unfairness in medical deep learning systems in real-world scenarios.

There are many evaluation ways to assess the fairness problem. The selected metrics might not be the most suitable one. Please elaborate more on the motivation of baseline setting and evaluation.

We introduced a novel metric for assessing fairness in medical segmentation, called Equity-Scaled Segmentation Performance (ESSP). ESSP offers a more direct and clinician-friendly evaluation compared to traditional fairness metrics like DPD and DEOdds. Unlike DPD and DEOdds, which may overlook overall performance, leading to a situation where a model with uniformly lower performance across all demographics could falsely appear fairer. This misalignment is particularly problematic in safety-critical medical applications, which demand high accuracy. In contrast, our proposed ESSP addresses this issue. It evaluates not only the disparity across different demographic groups but also measures the extent to which overall segmentation accuracy is compromised to achieve a fairer model. A higher ESSP score indicates a model that achieves both fairness and high accuracy.

Alongside ESSP, according to the reviewer’s suggestion, we have also updated the DPD and DEOdds results in the supplementary material, providing a comprehensive fairness assessment of various segmentation approaches. As the table below illustrates, our FEBS (Fairness-Enhanced Biased Segmentation) approach outperforms previous methods in terms of DPD and DEOdds, highlighting its effectiveness in achieving fairer outcomes in medical imaging. From the table below, we show that our method achieves comparable DPD and DEodds performances when compared against other segmentation methods.

Some current works should be included to make the experiments sufficient. See: FairAdaBN: Mitigating unfairness with adaptive batch normalization and its application to dermatological disease classification.

FairAdaBN was originally proposed for classification tasks, not segmentation, and necessitated modifications to the backbone architectures of classification models. However, with the advent of recent large-scale segmentation models like Meta's SAM, these models require fine-tuning with pre-trained weights for optimal performance. Altering the architecture, as in the case of FairAdaBN, might hinder the loading of these pre-trained parameters, which are trained from extensive datasets, potentially leading to lower segmentation accuracy. Additionally, FairAdaBN was initially applied to a ResNet, and adapting it to Transformer-based segmentation models also presents challenges. Thus, we suggest that further investigation is needed before applying FairAdaBN to segmentation tasks. We have cited and discussed FairAdaBN in our related work section, and we aim to integrate the FairAdaBN approach into our future research.

Since most of the attributes are only for the patient level, why use the pixel-wise weights?

Patients from various demographic groups may exhibit different anatomical characteristics. For example, Black people often have a larger cup-to-disc ratio and cup area compared to other races. These anatomical differences can influence segmentation accuracy. To address this, we employ pixel-wise weights to accommodate these underlying anatomical variations within the fundus images.

We thank the reviewer for the positive and encouraging review.

Please add NSD which is suggested by metrics reloaded.

Dice and IoU are commonly used by previous research [1,2,3]. For the sake of future comparison and completeness, we will keep the Dice and IoU in the table, and add NSD as the additional metric. Furthermore, we have included the NSD results for race as table below. The table indicates that unfairness commonly exists in our proposed fair segmentation task, which further strengthens the contribution of our FairSeg dataset. We have added those results in the supplementary material.

Add nnUNet.

We have included nnUNet in our segmentation benchmarks, which are detailed in the supplementary material. The table provided below presents results focusing on racial disparities. From the table, it is evident that nnUNet demonstrates marginally better performance compared to SAMed and TransUNet. However, it still exhibits significant performance disparities across different racial groups. This suggests that algorithmic unfairness is a pervasive issue in our proposed cup-disc segmentation, regardless of the choice of segmentation architectures.

The dataset was released as npz format. Could you please also release the original format?

We will release the dataset with the original format in addition to the existing npz format of our database.

It would be great if you could release the trained models as well.

The trained checkpoints of our models have been released through our Github repository.

Where do you plan to host this benchmark? CodaLab could be a good platform.

We will co-host our benchmark/dataset using both Google Drive and CodaLab.

Reference:

[1] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023).

[2] Zhang, Kaidong, and Dong Liu. "Customized segment anything model for medical image segmentation." arXiv preprint arXiv:2304.13785 (2023).

[3] Chen, Jieneng, et al. "Transunet: Transformers make strong encoders for medical image segmentation." arXiv preprint arXiv:2102.04306 (2021).

Thank you very much for your review and the insightful comments.

There is no visualization comparison between different methods.

We have added a visualization comparison of segmentation results between our method and other competing approaches.

In equation (1), a parenthesis is missing in the formula.

We have addressed the formatting issue in the paper.

The authors proposed a new Dice loss with a novel Fair Error-Bound Scaling mechanism, however, there are experiment results to show the differences between the new dice loss and the common one.

As mentioned in Section 6.2 - “Training and Implementation Details”, the original SAMed and TransUNet are trained using cross entropy and the common dice losses. Hence, in Tables 1-5, the results between SAMed vs SAMed (Ours) / TransUNet vs. TransUNet (Ours) are the performance comparisons for the new proposed dice loss and the common one.

Thank you very much for the supportive comments and valuable suggestions!

The accuracy of the Nifty reg needs to be investigated since it might not be the SOTA for image registration.

In Section 4, we mentioned that “It’s noteworthy that this registration operation demonstrates considerable precision in real-world scenarios, as evident from our empirical observations that highlight an approximate 80% success rate in registrations.” During experiments, we have compared Niftyreg against the SOTA deep learning based approach [1] and a retinal image-based registration method [2]. We observed that NiftyReg is more robust than other registration methods. Upon further analysis, we have calculated that NiftyReg achieves an accuracy of roughly 82.4% in registration tasks. All the failed registration cases have been excluded by five professional clinician graders. Although there is a failure rate of about 20%, we are committed to ongoing exploration of state-of-the-art (SOTA) registration tools, including contemporary deep learning-based methods. Our aim is to enhance registration accuracy and, consequently, to release more images in our datasets during our future work.

Why validation set is not constructed/used in selecting models in training?

Given that our model is selected based on the last epoch of training. The finetuning of foundation models like SAM could be computationally expensive for both training and inference. This makes computing the validation accuracy every few epochs and selecting the checkpoints based on the accuracy is infeasible.

To better facilitate future research, we have released an extra 500 images as the validation set in our codebase and dataset.

Report Hausdorff distance and average surface distance along with Dice.

Please see the table below for Hausdorff distance (HD95) and average surface distance (ASD) for race. We have included such metrics across all sensitive attributes in the supplementary material. From table below, we observed that the disparity between different demographic groups commonly existed regardless of its evaluation metrics, which further strengthened the significance of our proposed FairSeg dataset.

The details of how standard deviation is computed need to be elaborated. Is it computed across the mean of for each group?

The computational cost of multiple runs with large segmentation foundation models is high. Previous studies [3, 4] typically evaluate these models only once, without standard deviation, rather than conducting multiple runs of evaluations.

How is the training/testing split performed? Is it just randomly sampled without considering sensitive attributes at patient level?

The training and testing split are just randomly sampled, and our sample distribution of different sensitive attributes reflects the real-world clinical patient distribution.

It would be helpful to discuss the importance of registration in preprocessing using NiftyReg.

As mentioned above and in Section 4/Figure 1, to obtain a large-scale fundus dataset with high-quality pixel-wise annotation, we need to use the OCT machine to generate the OCT fundus images and their corresponding cup-disc mask. However, OCT machines are fairly expensive and less prevalent in primary care, therefore, we propose to migrate those annotations from 3D OCT to 2D SLO fundus for potentially broader impact in early-stage glaucoma screening in the primary care domains. In order to transfer the annotations from 3D OCT fundus images to 2D SLO fundus images, we need to register the image to align the two imaging modalities by comparing the characteristic features between the two fundus imaging modalities of the same patient. The computed alignment matrix is then applied to the disc-cup masks of the OCT fundus images, aligning them to the SLO fundus images.

Reference:

[1] Hoopes, Andrew, et al. "Hypermorph: Amortized hyperparameter learning for image registration." Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings 27. Springer International Publishing, 2021.

[2] https://github.com/tobiaselze/oct_fundus_registration/tree/main

[3] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023).

[4] Zhang, Kaidong, and Dong Liu. "Customized segment anything model for medical image segmentation." arXiv preprint arXiv:2304.13785 (2023).