SparsyFed: Sparse Adaptive Federated Learning

Dear Reviewer 75sH,

Thank you for your detailed and constructive comments. Below, we address each of your concerns in detail:

1. Supporting claims:
   • We acknowledge that some results and explanations may have seemed unclear. To address this, we have revised the Introduction and Evaluation sections, particularly those related to sparsity consensus (Section 5.3), and expanded the ablation studies of our algorithm in Sections 5.4 and 5.5. Specifically, the weight reparameterization (5.4) plays a critical role, as it inherently induces sparsity in the model while maintaining sparsity resilience during training. This approach ensures that clients train on the same subspace of the global model, improving consensus on the sparse masks. For example, weight reparametrization allows SparsyFed to reach convergence 7-8x times faster, as shown in Fig. 1.
2. Introduction:
   • To clarify *SparsyFed'*s technical contributions, we have restructured the introduction to emphasize the specific mechanisms we used. These include weight reparameterization and sparse activation pruning. We also expand upon how our layer-wise activation pruning at the end of each round significantly differs from previous works.
     • Unlike previous methods, which applied fixed global sparsity across layers, our layer-wise approach prunes based on each layer's proportion of total parameters. This removes more capacity from dense layers while preserving parameter-efficient ones.
     • Pruning models at the end of each round allows full flexibility for local optimization. By starting from a dense model in the first round, SparsyFed also provides robust initialization for crucial layers, such as embeddings or class projectors. By contrast, earlier works which pruned from the first step had to exclude layers to preserve performance.
   • We emphasize that SparsyFed only introduces one new hyperparameter, the $\beta$ term in our weight reparameterization, making it suitable for tuning under the time-intensive Federated Learning paradigm.
3. Algorithm clarity:
   • To enhance the clarity of Algorithm 2, we have extended the explanations for its key steps, in ‘Pruning Activations During Local Training’ and 'Pruning Activations During Local Training’, and introduced a pipeline diagram to visually represent the workflow (New Fig.1).
   • Specifically:
     • At line 7, the GetLayer function extracts the weights and activations for the current layer.
     • At line 8, the extracted weights are used to determine the pruning levels for pruning the model.
     • At line 9, the activations are pruned using the defined pruning thresholds.
     • At line 10, the SetLayer function updates the sparse weights and activations in the model.
     • At line 11, the sparse activations and weights are used to compute gradients, ensuring that sparsity is preserved throughout training.
   • We want to emphasize that we do not use sparse activations to obtain a sparse model, instead we use them to improve computational efficiency. This has been clarified in Section 5.5. The efficiency benefits of using sparse activations have been cataloged in previous works in a centralized environment[1] and in a federated context[2]. In [1] they registered 5.6x speed-up for the forward pass and a 6.3x sped-up for the backward pass on appropriate hardware.

We hope these revisions address your concerns and clarify the key contributions of our work. Thank you for your valuable feedback, which has helped us improve the quality and presentation of our manuscript.

Sincerely,

The Authors

[1] Raihan, Md Aamir, and Tor M Aamodt. "Sparse weight activation training." In Advances in Neural Information Processing Systems, December 2020.

[2] Qiu, Xinchi, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. "Zerofl: Efficient on-device training for federated learning with local sparsity." In International Conference on Learning Representations, 2021.

Dear Reviewer Ay19,

We greatly appreciate your thorough review and insightful suggestions. Below, we address each of your comments in detail:

1. Comprehensive survey on adaptive sparse training methods:
   • We acknowledge the need for a more detailed comparison of existing methods. To address this, we revisited the evaluation in Section 5 to ensure that all research questions posed in the initial guidelines were thoroughly answered. For example, we have provided comprehensive ablation studies on the effectiveness of weight reparametrization to stabilize the sparse training and the effectiveness of activation pruning for reducing the total computation cost of the training. Additionally, we expanded the discussion on the importance of consensus and sparsity pattern dynamics in Section 5.3, highlighted the significance of the weight reparameterization method in Section 5.4, and emphasized its effective decoupling with activation pruning in Section 5.5.
2. Global model sparsity:
   • In the revised manuscript, in Section 3 we clarified that the server maintains a sparse model for all rounds after the initial round. At each training round, all clients receive the same (sparse) global model, train it locally on their data, and send back sparse local updates. The sparsity of the global model after aggregation will depend on the outer optimizer and the sparsity pattern across clients; however, our weight reparametrization in SparsyFed tends to create consensus on the sparse masks of the clients, limiting interference during aggregation and allowing us to maintain a higher degree of sparsity than the baselines (see the right-side of Fig. 2).
   • This behaviour is illustrated in Fig. 1(b), where we have updated the caption for greater clarity.
3. Model plasticity:
   • A clear definition of model plasticity has been added to the Introduction section. Specifically, plasticity refers to the model’s ability to adapt to new data while retaining previously learned knowledge. Studies in supervised and reinforcement learning have shown that reduced plasticity can lead to overfitting to limited data and hinder generalization[2,3]. This definition has been contextualized to the contrast between fixed-mask approaches which limit plasticity, used by previous works, and the dynamic approach of SparsyFed. As an extreme example, in a multi-task or continual learning setting, neural networks can be iteratively pruned to build task-specific sub-networks[1], which, as an algorithm, requires dynamic masking.
4. Expanding baselines:
   • We appreciate the suggestion to include additional baselines. After reviewing the works in [4, 5, 6], we discuss their relevance to our work:
     • FedSparsify [4]: Our approach begins at the target sparsity level. In contrast, FedSparsify gradually increases sparsity over training rounds, meaning it spends a significant chunk of training communicating the dense model and thus lacks the crucial communication-reduction benefits of SparsyFed. For example, we obtain a 10x reduction in total up and down-link communication costs while the best performing FedSparsify method obtains a 3-4x reduction~(See Table 1.a and Table 1.b of [4]).
     • RA-Fed [5]: This work focuses on training a dense global model in resource-constrained settings rather than a sparse model. Thus, it deviates from the goals of our work, which are (a) to train a sparse model and (b) to reduce communication costs during federated training. As such, we do not consider it a relevant baseline.
     • Zhou et al. [6]: The goal of this work is not to propose a new competitive sparsification method; rather, it proposes a framework for analyzing the rate of convergence of sparse training methods. We thank the reviewer for pointing out the paper to us, and we are looking into how we could apply a similar analysis to our work. However, it does not represent an experimental baseline for SparsyFed.

We hope these revisions address your concerns comprehensively. Thank you again for your constructive feedback, which has greatly improved the quality of our work. We are happy to answer any further questions or follow any requests, your advice is crucial for improving the quality of our work.

Sincerely,

The Authors

[1] Mallya, Arun and Lazebnik, Svetlana, "PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning”

[2] Lyle, Clare, Mark Rowland, and Will Dabney. “Understanding and preventing capacity loss in reinforcement learning”

[3] Lyle, Clare, et al. “Understanding plasticity in neural networks”

[4] Stripelis, Dimitris, et al. "Federated progressive sparsification (purge, merge, tune)+.”

[5] Wang, Yangyang, et al. "Theoretical convergence guaranteed resource-adaptive federated learning with mixed heterogeneity.”

[6] Zhou, Hanhan, et al. "Every parameter matters: Ensuring the convergence of federated learning with dynamic heterogeneous models reduction."

Dear Reviewer yVEF,

We sincerely thank you for your thoughtful feedback and for highlighting both the strengths and weaknesses of our work. Below, we address your comments and outline the improvements we have made or plan to implement based on your valuable input:

1. Clarity of the Algorithm
   • We revisited the introduction to clarify the novelty and technical aspects of the proposed algorithm. We also added a graphical pipeline to show how our method works in new Fig. 1.
   • We have revised Section 3, particularly steps 6–10 of the algorithm, to enhance the clarity of each step of the algorithm. Specific updates were made in the subsections to explain the reasoning behind the steps, and how they contribute to the overall design of the algorithm. We added the missing definitions of the SetLayer and GetLayer functions.
   • Regarding Alg. 2, in step 8, we compute per-layer sparsity levels for each layer. Such per-layer sparsity levels reflect the per-layer sparsity of weights (the fraction of zero values in the weights of the specific layer). In step 9 of Alg.2, activations are pruned layer-wise via TopK operation using the per-layer sparsity level computed at step 8. This approach allows for a more tailored pruning strategy that aligns with the information retention characteristics of each layer.
   • We clarify here that the model weights are pruned applying the target sparsity level to the entire neural network, leading to a non-uniform level of sparsity among the layers of the network.
2. Weight Reparameterization and Sparsity
   • To clarify why we used a weight reparametrization method and what is the intuition beyond his effectiveness we updated the evaluation section. Specifically, in Section 5.4, we explain how the chosen weight reparameterization aims to train a sparse model helping the resilience of pruning. The consequences of this can be observed in the sparsity pattern of the model, which we detail in Section 5.3. Finally, in Section 5.5, we discuss the impact of activation pruning.
3. Computation Reduction
   • The computational reduction achieved through the use of sparse matrices during training and its associated benefits have been discussed in previous works like [1] in a centralized environment, and in [2] in a federated context. One of our main contributions lies in the application of a per-layer sparsity to the activations. Previous approaches such as SWAT [1] apply a fixed sparsity level across the activations of all layers. In contrast, we use a layer-wise sparsity for activations (step 8 of Alg. 2), instead of the target sparsity. Such per-layer sparsity levels (applied to the activations) reflect the per-layer sparsity of weights (the fraction of zero values in the weights of the specific layer).
4. Baseline Comparisons
   • We implemented a naive federated version of Powerpropagation as a baseline for comparison. In the original Powerpropagation paper, pruning was applied at the end of centralized training. We trained the model in a federated setting on various data heterogeneity partitions, then pruned and evaluated it at the end of the training. As evident from the reported results, SparsyFed significantly outperforms its naive counterpart. A detailed description of the naive federated Powerprogation will be provided in the appendix, SectionE.1.
   • Regarding the selection of G.3 over G.4 from the appendix: The performance of the two FLASH variants (G.3 and G.4) were similar in the reported results of the original paper, with the former obtaining better performance in some settings. We decided to use G3 since it uses fewer hyperparameters, and, in this sense, it is more similar to our method, which only requires one sparsity-related hyperparameter.
5. Sensitivity Analysis
   • We recognize the importance of understanding how client participation, number of clients, and data heterogeneity affect our method. We integrated in the appendix an experiment on CIFAR-10 with a large federation of clients, and a few local samples for each participant. We used a low level of participation rate of 0.1% which translates to 10 clients per round. Our results show how our method outperforms other baselines in this setting. A more detailed description, along with the results, has been included in the appendix in Section E.6.

We hope these responses address your concerns thoroughly. Your feedback has been invaluable in guiding improvements to our work, and we are committed to ensuring our revisions meet the highest standards. Thank you again for your constructive critique.

Sincerely,

The Authors

[1] Raihan, Md Aamir, and Tor M Aamodt. "Sparse weight activation training." In Advances in Neural Information Processing Systems, December 2020.

[2] Qiu, Xinchi, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. "Zerofl: Efficient on-device training for federated learning with local sparsity." In ICLR, 2021.

Dear Reviewer LGTa,

Thank you for your detailed comments and for highlighting the practical benefits of SparsyFed. Your feedback has been extremely helpful, and we have carefully addressed your concerns as follows:

1. Pipeline Clarity
   • To improve clarity, we have added a diagram illustrating the pipeline of our proposed method. It is a visual overview of the key steps of our training pipeline, with a particular focus on the sparse training pipeline on the client. To enhance the overall clarity of the manuscript we decided to put in the introduction (new Fig.1) to give a clear and immediate view of the work we are proposing.
2. Algorithm Clarification
   • In Algorithm 2, we retained both Line 9 (layer-wise pruning of activations) and Line 10 (updating activation values globally) to highlight their distinct roles. Line 9 involves operations applied at the layer level, while Line 10 updates the entire model. This distinction ensures clarity and prevents misinterpretation. We have revised Section 3, particularly steps 6–10 of the algorithm, to improve the clarity of each step. Specific updates were made in the subsections Sparsity-Inducing Weight Reparameterization and Pruning Activations During Local Training to elaborate on the rationale behind these steps and their contribution to the overall design of the algorithm.
3. Hyperparameter Tuning Efforts
   • We understand your concern that hyperparameter tuning should not be emphasized as the primary contribution. However, we believe that our focus on reducing hyperparameter complexity, while not resolving all related challenges in FedOpt, is an often-overlooked aspect in other works. This simplification ensures practicality and usability, particularly in federated learning scenarios. Our design avoids introducing additional hyperparameters, which can exacerbate tuning difficulties, and instead uses only a single hyperparameter (sparsity target) to balance performance and adaptability.
4. Sparse Communication Relevance
   • While sparse communication is not the primary focus of our research, it is a critical consideration in federated learning due to communication being a major bottleneck in such systems. Reducing communication costs during training can significantly improve overall efficiency, as both the model and local updates must be transmitted between clients and the server in every round.
   • SparsyFed achieves substantial reductions in both uplink and downlink communication, as illustrated in Fig. 2(a). This reduction translates into noticeable speedups of the overall training and lower bandwidth requirements, particularly beneficial for cross-device FL environments.
5. Consensus on Sparsity Patterns
   • We have expanded the discussion in Section 5.3 to explain why SparsyFed facilitates better consensus on sparsity patterns across clients. This improvement came from the weight reparameterization technique, which enables the model to intrinsically focus training on a consistent subset of weights. This alignment enhances convergence and performance across heterogeneous client data distributions.
6. Theoretical Guarantees
   • As this work focuses on practical implementations and experimental validation, we did not include a theoretical analysis of convergence properties. Such an analysis would be outside the scope of this paper, but we acknowledge its importance and consider it a potential direction for future work.

We hope these responses address your concerns comprehensively. Thank you again for your constructive feedback, which has helped us refine our paper.

Sincerely,

The Authors

Dear Reviewer wQKr,

Thank you for your insightful feedback and for recognizing the strengths of our experimental evaluation. Your detailed critique has provided us with valuable guidance for improving our work. Below, we address your comments point by point:

1. Clarifying Results and Captions
   • We clarified that Fig. 2 (right) compares the sparsity levels achieved by the global model after each round of training. These values are measured after aggregating local updates on the server. Due to the non-exact overlap of the local updates, the global model tends to gain density after aggregation. A smaller density gain indicates that the updates share most weights in their binary masks, reflecting a better consensus on the masks among clients.
   • Additional plots with varying target sparsity levels have been added to the appendix to provide a broader perspective, fig. 6 in section E.3.
   • Section 5.3.1 has been rewritten to better elaborate on the concept of mask consensus. Improved consensus results from the weight reparameterization technique, which encourages clients to focus on the same subset of weights. This leads to marked mask stability without excessive weight growth, resulting in convergence to a shared mask.
2. Activation Pruning Performance
   • When comparing SparsyFed with and without activation pruning, performance degradation was noticeable only at extreme sparsity levels. In such cases, some layers tend to become extremely sparse to meet the overall target sparsity. Pruning the activations of these specific layers can significantly reduce the information flow between layers, thereby impacting overall performance. It is important to note that this phenomenon occurs only at very high sparsity levels. At lower, yet significant, sparsity levels, our approach shows negligible differences in performance.
   • Section 5.5 and Table 2 have been updated to clarify this phenomenon. Experiments for α = 0.1 have been included in Table 2 in the updated submission.
3. Reparameterization Method
   • We selected reparameterization methods based on the intuition that an inherently sparse model would perform better when a layer-tailored sparsification of activations is applied. These methods were chosen for their ability to induce inherent sparsity during training while mitigating performance degradation caused by sparsity. Section 5.4 has been updated to clarify our rationale and explain how these methods were integrated into SparsyFed, as well as why we adopted the reparameterization proposed in Powerpropagation.
   • Regarding Fig. 4, the reparameterization methods shown were selected from state-of-the-art approaches to evaluate their ability to induce sparsity naturally and their impact on performance under high sparsity levels.
4. Theoretical Analysis of Costs
   • The computational reduction achieved through the use of sparse matrices during training and its associated benefits have been discussed in previous works like [1] in a centralized environment, and in [2] in a federated context. Our main contribution lies in recognizing the varying nature of the layers in a neural network, where different layers retain different amounts of information. Unlike previous approaches, which applied the same level of sparsity across all layers, we address this issue by using the layer-wise sparsity, as computed in step 8, instead of the target sparsity. This approach allows for a more tailored pruning strategy that aligns with the information retention characteristics of each layer.
5. Convergence Analysis
   • While theoretical convergence bounds are beyond the current scope of the paper, we have clarified why SparsyFed demonstrates better performance compared to state-of-the-art methods. Specifically, as shown in Fig. 2(b), our method achieves high sparsity almost immediately, resulting in reduced downlink communication (server to client) after the initial round. Additionally, the weight reparameterization technique helps clients focus on a consistent subset of weights, leading to a faster convergency, as it’s possible to see in Fig. 2(a) SparsyFed reaches better accuracy results compared for example to Top-K, with an overall similar communication cost.

We hope these responses address your concerns thoroughly. We greatly appreciate your constructive feedback and have used it to refine both the clarity and depth of our paper. Your critique has been instrumental in strengthening our work, and we look forward to any further suggestions you may have.

Sincerely,

The Authors

[1] Raihan, Md Aamir, and Tor M Aamodt. "Sparse weight activation training." In Advances in Neural Information Processing Systems, December 2020.

[2] Qiu, Xinchi, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. "Zerofl: Efficient on-device training for federated learning with local sparsity." In International Conference on Learning Representations, 2021.