FairFedMed: Achieving Equity in Medical Federated Learning via FairLoRA

• I don't (yet?) see how / why the main algorithm presented here should be a correct federated implementation of truncated SVD, as is claimed. This concerns in particular the global aggregation part, i.e. eq. (3): If $A \approx U_A S_A V_A^T$ and $B \approx U_B S_B V_B^T$ then it does not generally follow that $A + B \approx (U_A + U_B) (S_A + S_B) (V_A + V_B)^T$? Or am I missing something crucial here? (Intuitively, even if "similar" singular components are identified at the different sites - unclear why that would necessarily be the case - why should these also be in the same order?)
  • Also, the authors essentially propose a new method for performing federated SVD here, on which there is already research which is neither acknowledged nor discussed [1-3].
  • Finally, the whole premise of FL is that data remain private per site. This is not necessarily the case if full SVD updates are shared, hence the various proposals for privacy-preserving federated SVD. While I would imagine that this isn't a realistic concern here due to the low truncation order, an argument to this effect should at least be provided.
• No details beyond the pure data content are available about the new FairFedMed dataset. What is the source of the data? How were these recordings obtained? Do the authors have the rights to publish this dataset? Was consent obtained from the human subjects and / or the rights holders? Under which license is the dataset provided?
  • Also, the authors simply write that they "divide all subjects into three separate sites" - does that mean that these are not real sites? If not, what makes these 'synthetic' sites in any way similar to real sites, i.e., exhibiting realistic data shifts etc.? In other words, how is this different from an arbitrary medical imaging dataset that is randomly split into three parts?
• It remained unclear to me which model parts exactly the different compared methods are tuning, partially because the whole modeling setup remained unclear to me - it is simply not described at all. I can try to deduce some things from Fig. 3 but it's really not very clear. Which piece of information is fed into which model component in order to pass from the input image to the output prediction? What even is the output of the pipeline? And, again, which parts of this pipeline are tuned by the different methods? I only gathered from the tables that apparently FairLoRA only tunes a (soft) prompt? This is not at all apparent from the remainder of the text.
• Given that the whole premise of the paper and approach is that different models should be learned for different demographic groups, I would strongly suggest adding at least some discussion of situations in which this might be desirable or not. Cf. the extensive recent debates on race adjustments in clinical risk models [4-5]. Also see [6] on the risks of modeling based on coarse demographic data. This also raises questions of causality: is demographic group membership really a cause of the prediction target, or is it just a correlate of an underlying cause that is (as of yet) unmeasured (but maybe should be) [7]? This will have implications for generalizability across domains. Learning group-specific models also hugely increases the risk of learning demographic shortcuts and thereby reinforcing existing (e.g. underdiagnosis) biases [8]. Compare this to e.g. equal opportunity regularization, which can sometimes mitigate label biases [9].
• The authors claim that "Despite the urgent need for fairness in healthcare, research on group fairness in federated learning for medical applications is limited, revealing a significant gap in this field." However, there is actually quite a bit of research in this area that is not acknowledged, cf. e.g. the various refs in [10].

References

• [1] Chai et al., Practical Lossless Federated Singular Vector Decomposition over Billion-Scale Data
• [2] Hardtebrodt et al., Federated singular value decomposition for high-dimensional data
• [3] Grammenos et al., Federated Principal Component Analysis
• [4] Zink et al., Race adjustments in clinical algorithms can help correct for racial disparities in data quality
• [5] Diao et al., Implications of Race Adjustment in Lung-Function Equations
• [6] Movva et al., Coarse race data conceals disparities in clinical risk score performance
• [7] Petersen et al., The path toward equal performance in medical machine learning
• [8] Yang et al., The limits of fair medical imaging AI in real-world generalization
• [9] Blum and Stangl, Recovering from Biased Data: Can Fairness Constraints Improve Accuracy?
• [10] Chen et al., Algorithmic fairness in artificial intelligence for medicine and healthcare

• It is not clear how many sites/institutions are involved in this dataset.
• The annotation and processing protocols should be discussed.
• The attributes-specific singular value is transferred during the communication. This may raise privacy concerns.
• There are six attributes, but only three of them are evaluated in the experiments.

1. The rationale for the proposed method lacks clarity. Specifically, it is unclear how decomposing the singular value matrix S into distinct matrices for each demographic group during low-rank adaptation can enhance fairness. The authors should clarify what fairness notions their method can improve and provide theoretical backing for their design choices.

2. While the authors assert that achieving fairness in federated learning with medical images presents unique challenges, they do not specify these challenges. A discussion comparing the complexities of fair federated learning in medical contexts to general settings is absent. Given existing research on fairness in federated learning, such as [1] and [2], it would be beneficial to evaluate the applicability of these studies to medical settings, as the lack of this comparison leaves the motivation for the proposed method ambiguous.

3. The claim that existing state-of-the-art federated learning methods excel with natural images but struggle with medical images is made without a corresponding strategy in the proposed method to address the specific characteristics and challenges of medical data. The authors should clarify how their approach is tailored to meet these unique needs.

4. The authors claimed that in the medical field, cross-institution demographic differences are common due to geographic heterogeneity. However, the cross-site distribution in the proposed dataset, according to Figure 2, doesn't show clear difference among different site. This hinders the utility of the proposed benchmark for site heterogeneity.

5. The creation of the proposed dataset is not clearly described, and many details are missing. Where do these data come from - a single site or multiple sites? What are the criteria to split the dataset into three sites? If all the data involved comes from a single site and is synthesized from three sites, It will be far from the real-world setting as data from different sites can be extremely different as they may use various devices and protocols, and the patient distribution varies a lot.

6. The authors mentioned previous fair federated learning research [1,2]. However, these studies were never included as the baseline. Therefore, there's no insight into how medical applications are different from general settings and whether or not previous methods work for medical applications. Moreover, another trivial baseline missing is using fairness training strategies in clients, such as regularization, resampling, etc.

7. Site fairness is not discussed in the manuscript. However, this could be a problem in the medical field as data disparity between medical sites can be significant. The reason that there's no much performance difference between sites could be that the dataset involved comes from single site.

8. It is counterintuitive that the proposed federated learning method outperforms single-site training configuration. There ought to be more discussion.

9. Standard deviation or statistic testing is not presented to show the significance of performance differences.

References

[1] Ezzeldin, Y. H., Yan, S., He, C., Ferrara, E., & Avestimehr, A. S. (2023, June). Fairfed: Enabling group fairness in federated learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 6, pp. 7494-7502).

[2] Badar, M., Sikdar, S., Nejdl, W., & Fisichella, M. (2024, March). FairTrade: Achieving Pareto-Optimal Trade-Offs between Balanced Accuracy and Fairness in Federated Learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 10, pp. 10962-10970).

The paper’s limitations include FairFedMed’s focus on glaucoma, limited fairness metrics, high computational costs, need for deeper ablation studies, lack of longitudinal fairness analysis, and limited interpretability insights for real-world deployment.