FairSkin: Fair Diffusion for Skin Disease Image Geneartion

This paper falls between a methodological paper and an application paper. For a methods paper, the individual methods (balanced sampling methods and losses) are either reused from other papers, or not substantial enough. If we consider the whole framework as a creative combination of those methods, there are still problems: There is hardly any comparison to work by other authors, and the whole analysis is conducted on a single dataset. For an applications paper I would also expect the use of more than one dataset, more (fairness) metrics and a deeper discussion of the clinical relevance of the findings. The very thing the paper tries to solve also very much impedes it's impact: The scarcity of darker skin types in the dataset leads to a small test set und thus results from which it is unclear whether they generalise well. I encourage the authors to think about using additional attributes to repeat and strenghten their analysis (e.g. using FairSkin on age/gender and seeing whether the results still hold).

A more detailed list of issues:

1. The main claim of the paper is that it improves fairness by balancing out the lack of high quality training data of darker skin-types in the dataset. But the results in Fig 4 show that the accuracy increases for 'Caucasian' and decreases for 'Asian' and 'African' compared to the vanilla approach. So either I misunderstand something fundamentally, or the validation/test set is way too small to be representative, or this paper achieves the opposite of what it tries to do.
2. Much of the analysis hinges on how the method does on the most underrepresented data group, which according to table 4, is 'African'. Given that the smallest disease subgroup only contains 12 examples and a 8:1:1 train/val/test split, some of the analysis is done on 1-2 datapoints. I would expect repetitions of results using different hold out sets to mitigate against this very low number.
3. A repetition of (part of) the analysis on a different dataset would make this paper much stronger. There is e.g. the scin dataset, which also contains Fitzpatrick labels: https://github.com/google-research-datasets/scin
4. A tabular ablation study would be really helpful to decide which of the components are actually making a difference. The current solution with FairSkin-XYC is hard to parse and read and I had to skip back and forth numerous times.
5. Many of the claims in the conclusion are weekly supported by the experimental results:

• A):"Our approach effectively enhances the overall quality of generated images while reducing disparities in image generation quality across different classes." The overall quality as measures by IS and FID scores stays practiacally the same, and is worse than the baseline CBDM. But I agree with the second half of the statement.
• B) "we address the challenge of balancing fairness and performance in image generation and downstream classifier tasks" There is the compound ESSP score, but for example table 1 lacks an accuracy score altogether. There is also little discussion on how the balance the two objectives.
• C) ..." tasks when dealing with long-tailed datasets [plural!] through FairSkin." But the paper only deals with one single dataset, and apparently only with a fixed datasplit within that dataset. The results would be much stronger if the framework was tested on different datasets too, even synthetic ones, or at least multiple ways to hold out data in the splits. CBDM
• D) "Furthermore, it improves the fairness of downstream classifiers when using generated images for data augmentation" While the paper demonstrates an improvement in the Demographic Parity metric, it actually decreases downstream performance (fig 4) of the skin type classes with less images in the original training data. It makes me wonder whether DP is the appropriate fairness metric in this case. The paper lacks a discussion of why DP was chosen as the only fairness metric. Why did you e.g. not report equalized odds? Consider reporting more metrics and more results on a skin type and disease stratified level (in the appendix).
• E) "Our comprehensive experiments demonstrate significant performance improvements across various tasks" The validation accuracy stays practically the same. DP and FID variance improve, but again 8D) is this even meaningful when it seems to decrease performance of classes with least training data?
• F) "Highlighting the practicality and effectiveness of our methods in real-world applications." There is no discussion in the paper on how this could be practical or effective in real-world applications. Such a claim would at the very least require use test-data from a different dataset. Or an analysis on most-confounded disease classes to look at potential real-world harm of misclassifications.

6. I would have appreciated a deeper analysis stratified by disease to see how the method fares for diseases with more or less training data (for individual/all skin type classes)
7. There is very little discussion in the introduction and elsewhere about fairness in ML literature.

A general remark:

• The paper indiscriminately uses the words racial/ethnic/demographic classes when referring to the above labels. Race, ethnicity, and demographic, are a) not synonyms and b) not covered by the Fitzpatrick scale. The Fitzpatrick scale measures skin type (how much tanning occurs under UV exposure).

None of the ideas presented in the methods is particularly exciting or novel.

The work fails to cite many recent works that tackled the problem of fairness in dermatology images.

There is no comparison with other competing methods (e.g. [5,6]).

This work synthesizes medical images, which, even though they look realistic, are not grounded in a real underlying physical manifestation of skin lesion pathology. Therefore, there is a risk of harm in using these images for serious medical applications. Medical experts have not assessed the quality of the synthesized images. The authors should clearly mark the synthesized images as such in all the figures.

• The paper's selection of three racial categories and five specific skin diseases lacks adequate justification. A more robust approach would involve a fine-grained control of skin tone, potentially incorporating established skin tone scales such as the Fitzpatrick or Monk scales.
• The evaluation protocol's reliance on FID and Inception-based scores may not be appropriate in this context. While monitoring the generation performance of synthetic data is essential, utilizing these metrics as quality indicators is questionable, mainly because the Inception Score is based on ImageNet statistics, which may not be relevant to the datasets employed in this study.
• The paper raises concerns regarding dataset imbalance. However, it employs standard accuracy rather than a balanced accuracy metric, which could lead to skewed performance assessments, especially in imbalanced datasets. Balanced accuracy metric would provide a fairer evaluation of the model’s performance across underrepresented classes.
• The experimental protocol could also be enhanced by including additional datasets beyond Fitzpatrick17. Expanding the dataset selection would improve the generalizability of the findings. See some suggestions in the section below.

• Several existing studies apply diffusion models to fairness-related challenges in skin lesion imaging, and some even use the same Fitzpatrick17k dataset. The paper should compare the proposed generation method with these closely related works and clarify why it is a valuable addition to this research area.

  [1] Sagers, L.W., et al., Improving dermatology classifiers across populations using images generated by large diffusion models, NeurIPS Workshop, 2022.

  [2] Sagers, L.W., et al., Augmenting medical image classifiers with synthetic data from latent diffusion models, 2023.

  [3] Wang, J., et al., From Majority to Minority: A Diffusion-based Augmentation for Underrepresented Groups in Skin Lesion Analysis, MICCAI ISIC Workshop, 2024.

  [4] Ktena, I., et al., Generative models improve fairness of medical classifiers under distribution shifts, Nature Medicine, 2024.

• The proposed reweighting and resampling techniques are standard practices for handling imbalanced datasets and could be considered intuitive; thus, the novelty of the proposed method may be limited.

• Despite the class diversity loss, the generated images still lack diversity, which might limit their utility in augmenting datasets for varied skin lesion types.

• While demographic parity (DP) is a commonly used metric, it may not fully capture the nuances of fairness, as equal positive rates across groups do not necessarily indicate equitable outcomes. True positive rates across groups might provide a more accurate assessment of fairness in clinical applications. Both fairness metrics used in this study are DP-based, and the work should consider additional, informative metrics.