FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extraction

Response to [W1]. Thanks for your question. We train ResNet-18 using CIFAR-10, which is partitioned into 100 clients using a Dirichlet distribution with a parameter of 0.3. In each communication round, we sample 10 clients, and these clients locally train the model for four epochs with a batch size of 20.

According to the numerical results in Table 1 and Figure 3 in the manuscript, all baselines can achieve a global accuracy of 65% or higher, and various submodels can realize a global accuracy of 60%. Therefore, the table below presents the number of communication rounds, overall computation, and communication costs when the average global accuracy reaches 65% and all submodels should achieve an accuracy higher than 60%:

Notes to the table: (i) we evaluate the result every 20 rounds, so the value of rounds can be divided by 20. (ii) Model 1/64 of FedRolex never achieves an accuracy above 60%. So we present its result of the 640th round, where FedRolex achieves an average global accuracy of 65% for the first time.

The table demonstrates the superiority of the proposed FIARSE: it requires the fewest communication rounds, and the least communication and computation overhead among the selected methods.

Response to [W2]. Thanks for your question. It is possible that our proposed method may eventually extract a constant submodel for a given size. Figure 1(b)(d) in supplementary material shows the differences gradually diminish between two models of $(t-50)$-th and $t$-th communication rounds, where $t \in \{50, 100, \dots\}$. However, it does not mean the performance is limited; instead, the proposed FIARSE finds optimal model architectures for various model sizes. This is because all model parameters are explored/trained, as shown in Figure 1(a)(c), demonstrating the number of parameters that different model sizes have explored.

The following tables present the test accuracy of HeteroFL and the proposed FIARSE on both global and local datasets after 1000 rounds of training: (Note: The numerical cells are in the form of "global/local")

• Figure 1(a)(b): {1/64, 1/16, 1/4, 1.0} with 25 clients each

• Figure 1(c)(d): {0.04, 0.16, 0.36, 0.64} with 25 clients each

Response to [W3]. Thanks for your suggestion. We conducted empirical studies to train a ResNet with CIFAR-10 and CIFAR-100 using 100 clients, while other settings are consistent with those described in the response of [W1]. The tables below compare the proposed FIARSE and FjORD.

• CIFAR-10:

• CIFAR-100:

In Table 1 of the supplementary material, we compare FjORD and other baselines comprehensively. We also conduct one more client heterogeneity setting (i.e., {0.04, 0.16, 0.36, 0.64, 1.0} with 20 clients each) and present the numerical results in Table 2.

Response to [W1]. Thanks for your question. While we acknowledge the limitations of unstructured pruning in the Conclusion, we also highlight significant industry progress in Appendix A, supported by notable evidence [1,2]. For instance, Nvidia GPUs support unstructured pruning through the NVIDIA Ampere Architecture and NVIDIA TensorRT.

The rest of the response demonstrates the advantages of FIARSE in terms of computation and communication overhead. We train ResNet-18 using CIFAR-10, which is partitioned into 100 clients using Dirichlet distribution with a parameter 0.3. In each communication round, we sample 10 clients, and these clients locally train the model for 4 epochs with a batch size of 20.

According to the numerical results presented in Table 1 and Figure 3 in the manuscript, all baselines can achieve a global accuracy of 65% or higher, and various submodels can realize a global accuracy of 60%. Therefore, the table below presents the number of communication rounds, overall computation, and communication costs when the average global accuracy reaches 65% and all submodels should achieve an accuracy higher than 60%:

Notes to the table: (i) we evaluate the result every 20 rounds, so the value of rounds can be divided by 20. (ii) Model 1/64 of FedRolex never achieves an accuracy above 60%. So we present its result of the 640th round, where FedRolex achieves an average global accuracy of 65% for the first time.

The table demonstrates the superiority of the proposed FIARSE: it requires the fewest communication rounds, and the least communication and computation overhead among the selected methods.

References: [1] How Well Do Sparse ImageNet Models Transfer?, CVPR'22 [2] Sparse Fine-tuning for Inference Acceleration of Large Language Models, 2023

Response to [W2]. Thanks for your suggestion. We conduct empirical studies to train a ResNet with CIFAR-10 using 100 clients, while only 10 clients are selected to train the model in each communication round. The tables below make a comparison between NeFL, FjORD, and the proposed FIARSE.

Among these three methods, the proposed FIARSE significantly outperforms another two baselines.

Response to [W3/Q2]. Thanks for your suggestion and for bringing interesting work. Due to the space limit, we moved our related work to Appendix A and discussed the similarities and differences compared to the previous works. For example, this work achieves model sparsification in federated learning, but different from the previous studies [1,2], we consider each client to possess heterogeneous computation capacities. We understand that paper organization is not a common practice, and we will insert a brief summarization of related work between Introduction and Preliminary.

IST [3] focuses on model parallelism in federated learning, where each node independently trains a submodel, and the server assembles all submodels into a global network. In contrast, our proposed FIARSE explores submodel extractions for different model sizes, which can be efficiently trained to achieve remarkable performance. These two works address different problems. From a theoretical perspective, IST analyzes the convergence while training with various submodels. Similar to the conclusion drawn from Theorem 2 of IST, the proposed FIARSE converges to a neighborhood of a stationary point. IST confirms that the bias cannot be eliminated, indicating that the model keeps fluctuating within a constant area. In contrast, our theoretical analysis demonstrates the possibility that FIARSE can minimize this neighborhood area to zero.

References: [1] Sparse Random Networks for Communication-Efficient Federated Learning, ICLR'23 [2] DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training, ICML'22 [3] Towards a Better Theoretical Understanding of Independent Subnetwork Training, ICML'24

Response to [Q1]. In our opinion, a static model never explores other parameters, while a dynamic model investigates new parts of the model. According to FjORD, the model architecture is fixed at the beginning of training, meaning the submodel never actively explores new parts of the model. Therefore, we consider FjORD to be a static strategy. We will clarify this classification in the Introduction.

Response to [W4]. Thanks for your suggestion. We have updated the experimental figure in the supplementary materials of the rebuttal. I have taken into account the suggestions from Reviewer inPJ and made the necessary updates accordingly, i.e., we will replace Figure 3 in the manuscript with Figure 2 in the supplementary material.

Response to [W1/Q1]. Thanks for your suggestion. We train ResNet-18 for 800 rounds using CIFAR-10, which is partitioned into 100 clients in a pathological scenario, where each client holds two classes. All these clients are equally distributed into four computation capacities {1/64, 1/16, 1/4, 1.0}. In each communication round, we sample 10 clients, and these clients locally train the model for four epochs with a batch size of 20. The following table presents the test accuracy of HeteroFL and the proposed FIARSE on both global and local datasets: (Note: The numerical cells are in the form of "global/local")

Response to [W2] Thanks for your suggestion. Unlike the experimental setting described in the Response to [W1/Q1], we partition CIFAR-10 into 100 clients using a Dirichlet distribution with a parameter of 0.3.

According to the numerical results presented in Table 1 and Figure 3 in the manuscript, all baselines can achieve a global accuracy of 65% or higher, and various submodels can realize a global accuracy of 60%. Therefore, the table below presents the number of communication rounds, overall computation, and communication costs when the average global accuracy reaches 65% and all submodels should achieve an accuracy higher than 60%:

Note to the table: (i) we evaluate the result every 20 rounds, so the value of rounds can be divided by 20. (ii) Model 1/64 of FedRolex never achieves an accuracy above 60%. So we present its result of the 640th round, where FedRolex achieves an average global accuracy of 65% for the first time.

Response to [W3/Q3]. Thanks for your suggestion. Different from the experimental setting described in the response to [W1], we partition CIFAR-10 into 100 clients using Dirichlet distribution with a parameter 0.3. Moreover, all these clients are divided into four computation capacities {1/64, 1/16, 1/4, 1.0} with the size of {40, 30, 20, 10}, respectively. The following table presents the test accuracy of HeteroFL and the proposed FIARSE on both global and local datasets: (Note: The numerical cells are in the form of "global/local")

Response to [Q2]. Thanks for your question. The aggregation of a specific parameter depends on the number of clients training in that parameter. For example, if five clients train a parameter, it is updated with their average value. In this sense, equal weighting is used when aggregating local updates. Our method is applicable to the case where clients carry on different weights, and the aggregation rules can follow Pruning-Greedy [1].

Reference: [1] Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction, NeurIPS'23

Response to [W1]. Thanks for your question. Given an arbitrary neural network, it is impossible to state the optimal architectures for different sizes so that all these submodels can achieve their best performance after federated learning. Therefore, we should explore different submodel combinations to find out the best one where all submodels can perform as outstanding as possible. The proposed FIARSE achieves the exploration by means of importance-aware extraction, where we assume the magnitude of model parameters reflects their importance and extract a submodel for a given size with the largest parameters.

In our opinion, your concern focuses on how many parameters are explored by a small model. To elaborate on this, we conduct more experiments to demonstrate the number of parameters that different model sizes have explored at $t$-th round and the model differences between two models of $(t-50)$-th and $t$-th communication rounds, where $t \in \{50, 100, \dots\}$. In specific, we train ResNet-18 on CIFAR-10, which is partitioned across 100 clients using Dirichlet distribution with a parameter $\alpha=0.3$, and ten clients are selected for training at each round. In the supplementary material, Figure 1 presents the numerical results, and the following paragraph will be added to Appendix D.4:

Ablation study: Submodel Exploration. Figure 1 includes two client heterogeneity settings, i.e., Figure 1(a)(b) is {1/64, 1/16, 1/4, 1.0} with 25 clients each, and Figure 1(c)(d) is {0.04, 0.16, 0.36, 0.64} with 25 clients each. Figure 1(a)(c) shows two phenomena: (i) all model sizes will gradually slow down their exploration speeds; (ii) even if the largest model size is smaller than the full model size, the number of untrained parameters will eventually go to zero, meaning that none of the parameters are ignored or deactivated. According to Figure 1(b)(d), the extracted submodel for a given size will gradually stabilize, indicating that a suitable submodel architecture has been found. Moreover, a submodel requiring a larger size makes it easier to obtain a stable architecture.

Response to [W2]. Thank you for raising an interesting aspect regarding fairness. In our work, model size is decided by client computation capacity. Hence, smaller models can be trained with more data, whereas some parameters in larger models may be trained on limited and biased data. Such limited and biased data could result in unfairness for clients with greater computational capacities, as the extract submodels may be unnecessarily large. However, these clients could extract smaller submodels to achieve a trade-off between model performance and fairness, which is an intriguing area for further investigation. We believe that the importance-based extraction does not necessarily amplify biases because the clients with large capacities opt not to use large but unfair models. Additionally, designing new local training approaches is another potential solution to address unfairness issues in our work. We will include this discussion in our paper and consider it for future study.

Response to [W3]. In this work, we introduce three kinds of thresholding strategies, i.e., model-wise, layer-wise, and sharding-wise. We introduce the model-wise strategy in the main body (Line 156 -- 159), while another two are discussed in Appendix D.1 (Line 1019 -- 1026). In the appendix of the manuscript, Tables 6 and 7 show the empirical results under different client heterogeneity settings, and both tables train a ResNet-18 on the CIFAR-100 dataset:

• Four different model sizes with 25 clients each:

• Five different model sizes with 20 clients each:

It is noted that layer-wise strategy divides ResNet-18 into four parts in accordance with the predefined convolutional layers, where each convolutional layer includes two residual blocks. As a result, this strategy can also be regarded as a sharding-wise strategy. Based on the numerical results, we witness that model-wise strategy outperforms layer-wise strategy. More detailed analysis is provided in the Appendix D.4 (Line 1060 -- 1067).

Response to [W4]. Thanks for your suggestions. We updated Figure 3, which is shown as Figure 2 in the supplementary materials. All these experiments were conducted under three different random seeds. We will update our experimental figures accordingly, including those in the appendix.

Dear Reviewer inPJ,

Thank you once again for your commitment to reviewing our paper and assisting us in improving our work. We would like to remind you that the discussion window will close in less than 24 hours, and we eagerly await your feedback.

We have provided detailed explanations for each of your concerns. We would greatly appreciate it if you could review our responses and let us know if they fully or partially address your concerns. Any additional comments you may have would also be highly appreciated.

Best,

Authors

Dear Reviewer inPJ,

Thank you for your feedback. We are pleased that our rebuttal has addressed your concerns and sincerely appreciate your positive rating.

Best,

Authors