FedGSE:Gradient-based Sub-model Extraction for Resource-constrained Federated Learning

• There are numerous typos in the writing, and the authors should check more before they submit their work. Meanwhile, authors are encouraged to improve the writing and organization.
• The related work dicussion is poor. Authors are recommended to read a recently published paper [1] to learn the related organization of related works. Authors are encouraged to learn how to cite the related papers correctly.
• The assumption of this work is too ideal. Firstly, the acquisition of a benchmark dataset at the server side is hard and usually viewed as an ideal assumption. Secondly, the label distribution of each client is also a a "secret" for each client, which may be not shared to the server due to privacy consideration.
• The evaluation details are not clear and correct enough. Firstly, all experiments should be implemented on the same type of GPUs to ensure a fair comparison. Secondly, the Non-iid setting is relatively poor. Authors should first state clearly that data are partitioned via Dirichlet distribution. Authors should follow the mainstream seletion for s (e.g. 0.1-1.0).

[1] Xue J, Liu M, Sun S, et al. FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated Learning with Bayesian Inference-Based Adaptive Dropout[C]//2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2023: 489-500.

General

Claim of Novelty: The authors assert that their method is pioneering in enhancing Federated Learning training by approximating the update gradients between the global model and sub-models. This claim hinges on the novelty of their strategy, which zeroes in on the optimization direction of approaching the global model's effectiveness when extracting sub-models. While they have provided a thorough review of related works, including static and dynamic allocation strategies for sub-model extraction, a significant oversight is the lack of comparison with other resource-constrained federated learning methods, especially those based on Knowledge Distillation (KD).It would be beneficial for the authors to expand their discussion and benchmarking to include KD-based methods and other strategies that address resource constraints in federated learning.

** Scalability Concerns:** The paper addresses a highly practical scenario of resource-constrained FL. However, the datasets used for validation, such as CIFAR-10/100, may not be sufficiently large or complex to truly test the scalability of the proposed method. For a method targeting resource-constrained environments, it's crucial to demonstrate its efficacy on high-dimensional data, such as face recognition datasets like CelebA or ImageNet.Suggestion: The authors should consider conducting experiments on larger and more complex datasets to truly validate the scalability and robustness of their approach.

Potential Limitations: The authors have acknowledged some limitations of their method, such as the need for constructing a public dataset on the server side and the challenges associated with executing the backward progress on the server side. While it's commendable that they've recognized these issues, it's essential to delve deeper into potential solutions or workarounds for these challenges.

About Algorithm

Gradient Representation and Assumptions: The paper introduces the gradient of a neuron with the equation $g_{l,i} = \sum_{k=0}^{K-1} \sum_{v=0}^{V-1} |\frac{\partial f(w; x, y)}{\partial h_{l,i}(k, v)}|$ where $f(w; x, y)$ represents the cross-entropy loss of the deep neural network for the input $(x, y)$. The dimension length of the feature map is represented by $K$ and $V$, i.e., $h_{l,i} \in R^{K×V}$. However, the paper does not provide a clear justification or empirical evidence for choosing this gradient representation. It would be beneficial to have a more detailed discussion.

Proof: The proofs of Lemma 1 and Theorem 1 share Appendix B, but it is oversimplified, and not clear to see its pertinence. Moreover, how can eq (5) be derived from eq (4) with a non-linear relationship?

I am struggling to understand in the paper what happens in the case of device heterogeneity - it is not explicitly described in the paper nor seems to be discussed sufficiently. I also lack visibility on what happens in case a node drops out from the computation or struggles to keep up with the load. Finally, it would be great to understand if the scheme introduces any bias towards the end result based on the proportional client datasets in the global scheme?