HeteroSFL: Split Federated Learning with heterogeneous clients and non-IID data

When comparing the training time, the federated learning methods (e.g., FedAvg) should be used as a baseline in the experiment. The main motivation for developing SFL methods is that the classical FL training schemes need to deploy the entire model on clients, which leads to high computation latency. However, the drawback of SFL is that the clients need to transmit the activations and receive the gradients in each training round, which may result in a much higher communication overhead than that of the FL methods. Besides, most of the FL methods (e.g., FedAvg) can iterate the local update multiple times before aggregation. This reduces the communication frequency and thus further reduces the training time. Given the above, the reviewer supposes that low upload or download rates will form a bottleneck for the SFL methods.

1. There are several important baselines that this paper should compare with. Since this paper says that they focus on compressing the communication and tackling the NonIID data, communication compression strategies in SFL should naturally be compared. The typical works include [a]. [a] Jianyu Wang, Hang Qi, Ankit Singh Rawat, Sashank Reddi, Sagar Waghmare, Felix X Yu, and GauriJoshi. Fedlite: A scalable approach for federated learning on resource-constrained clients. preprint arXiv:2201.11865, 2022

2. The theoretical convergence is not guaranteed.

1. The novelty of this paper is not sufficient. The paper seems to be a bundle of multiple existing techniques. To be specific, the authors integrate bottleneck technique by bottleneck compression by Eshratifar et al. (2019); Ayad et al. (2021), logit calibration by (Zhang et al, 2022), subnetwork aggregation in HeteroFL (Diao et al, 2021), knowledge distillation between sub-network by (Ilhan et al, 2023). Given this, the contribution seems to be rather incremental as all of these techniques have been applied in SFL or FL setting. The only novel contribution is bi-directional knowledge transfer improving upon single direction knowledge transfer proposed by (Ilhan et al, 2023). However, the bi-directional knowledge distillation (or mutual learning) has also been explored in federated learning before (Shen et al. 2020).

Shen T, Zhang J, Jia X, et al. Federated mutual learning[J]. arXiv preprint arXiv:2006.16765, 2020.

2. The experiment is not comprehensive.

   • The authors only compare their methods with three baselines, SR and TOP-K sparsity, and IdBL. However, the authors do not compare other methods with those baselines that consider heterogeneous training settings, e.g., ScaleFL (Ilhan et al, 2023), and HeteroFL (Diao et al, 2021). Note that these methods are model-agnostic, which means they can be easily applied to IdBL with bottleneck layers in the model. To ensure fair competition, I suggest the authors to compare those two methods in two groups of experiment, with the first group assuming the presence of logit calibration, and the second group without it, as both of these two methods do not apply this technique in their vanilla version.
   • The experiment is done only in small datasets CIFAR10, and CIFAR100.
   • The authors did not do an ablation study. The methods bundle a lot of components. It is essentially to see the impact of each component.
   • The baselines in CIFAR10 task do not reach comparable accuracy, which at least reaches 80% even in Non-IID setting.
   • The authors should compare their methods under different Non-IID settings. Please consider doing experiment setting $\lambda= 0.5$, $\lambda= 1$, and IID setting.

3. Privacy concern.

   • The BL method needs to send the compressed representation to the server, which poses serious threats of data leakage. As FL is mostly concerned with privacy protection, sending activation alone is too risky.
   • (major) The clients in the high-end group need to be aware of R, i.e., the set of underrepresented classes. However, in the FL setting, knowing the statistical distribution of data of others alone leaks privacy.

4. Minor: the paper itself is not self-contained. The authors integrate logit calibration by (Zhang et al, 2022), but do not show how it calibrates the label heterogeneity.