Local Superior Soups: A Catalyst for Reducing Communication Rounds in Federated Learning with Pre-trained Model

1. Addressing Novelty Concerns: Thank you for highlighting the importance of novelty in our work. The core novelty of our paper lies in the improved and efficient model soup method into federated learning (FL) with pre-trained models to drastically reduce communication rounds. This approach, particularly in the context of pre-training FL, represents a novel contribution to the field. Our experimental results across various datasets, model architectures, and FL non-iid settings demonstrate its effectiveness, which we believe is a significant contribution in FL research.

2. Computational Efficiency of the Diversity Term: We appreciate your concern about the computational efficiency of the diversity term. To address this, we conducted an empirical evaluation of the computational overhead. Our findings indicate that the diversity term results in only a marginal increase in runtime (about 5%) for a typical model pool size of 4.

3. Ablation Study and Figure 5 Observation: The results shown in Figure 5, where LSS without diversity and affinity terms still outperforms baselines, underscore the effectiveness of our connecting-preserving random model interpolation module. This module, a key component of our method, significantly contributes to the overall performance by enhancing the connectivity between local low-loss regions. The absence of the diversity and affinity terms in this scenario highlights the standalone impact of our interpolation technique, reinforcing the value it brings to the local model training process in FL.

4. Error Loss Landscape and Further Explanations: In response to your suggestion for further explanations, we provide data in Appendix E.2 on the flatness quantification of loss landscape to further show how our method enhances the local training performance and is located in a common low-loss region. This addition should provide a clearer understanding of the underlying mechanisms of our approach.

Your feedback is helpful in refining our work, and we hope these revisions and explanations address your concerns satisfactorily.

1. Length of Section 3.2 and Typos: We appreciate your observation regarding the length of Section 3.2. In response, we have succinctly revised this section to better highlight our unique contributions while referencing prior works for context. And we also correct the typos and refine our explanations to be more precise. We also provide the empirical studies in Appendix E.2 for verifying the limited gains achievable by increasing the number of local steps arbitrarily.

2. Empirical Validation of Hypotheses: To address the concern about the lack of empirical validation, we have included new empirical studies in the Appendix E.2. These studies test our hypotheses on communication rounds reduction and provide evidence supporting our assertions on improving smoothness (reducing $\beta$).

3. Detailed Description of Random Interpolation: In line with your suggestion, we have added an algorithmic comparison of the Random Interpolation subroutine in the Section 3.3.1 and the Appendix E.2. This addition aims to clarify our methodology and its unique benefits, particularly in ensuring model alignment within low-loss valleys.

4. Client Scale in Experimental Setup: Our algorithm primarily focuses on the cross-silo federated learning setting, hence the number of clients in our experiments is usually set to be within 100. In the appendix, we have included experiments to demonstrate the effectiveness of our approach with 50 clients on the CIFAR10 dataset.

5. Comparative Analysis of Computational Overhead: To ensure a fair comparison between FedAvg and LSS, we have added a comparative analysis of computational overheads in the Appendix E.2. This includes a table comparing the computational costs across different methods. While our method requires more memory during training, compared to other soups-based methods, our overall computation time is faster within a communication round.

6. Local Epochs and Model Performance: Regarding the fixed number of local steps, additional experiments have been conducted to show the optimal number. The results, now included in the Appendix E.2, demonstrate that increasing the local epochs beyond 8 leads to diminishing returns in performance, validating our initial choice.

7. Performance Comparison with Other Model Soup Methods and Explanations: To address your query on LSS's superior performance, we have included a detailed analysis in Section 1. This analysis contrasts LSS with other model soup methods, emphasizing the importance of our proposed three key modules: random model interpolation, diversity and affinity regularization term. We improve the inefficient training approach of the previous soup-based model, which involved training a large number of models and then filtering them. Instead, we have adopted the random model interpolation method to maintain the connectivity of various candidate models during the training process. Moreover, we incorporate two regularization terms to increase model diversity and enhance connectivity between models to reduce the time for training unnecessary models, thereby improving training efficiency.

8. Clarifications and Typo Corrections: We have redefined ambiguous variables in Algorithm 1 for clarity and corrected the identified typographical errors, including the expression "connecting preserving" and capitalization in Section 3.

9. Additional FL Baseline Performance Data: Thank you for your suggestion to include additional FL baselines in Figure 3. However, our primary objective with Figure 3 is to emphasize the fast convergence achieved by our LSS method. The inclusion of FedAvg as a comparison baseline is intended to clearly demonstrate this aspect. Adding more baselines, while potentially informative, might not substantially contribute to this specific narrative focus.

Your thorough review has been invaluable in enhancing the quality and clarity of our work. We are grateful for your time and effort, and we remain open to any further suggestions or discussions.

1. Memory and Computational Costs of Our Method: We acknowledge your concern about the memory and computational costs of our algorithm. In the revised manuscript, we have included a detailed analysis in the Appendix E.2 that compares the memory footprint of our method with other existing approaches. This analysis demonstrates how our approach effectively balances memory usage with computational efficiency.

2. Inclusion of Large Pre-trained Models in Experiments: In the Appendix E.2 of the initial version, we have already included experiments that involve fine-tuning large models like Vision Transformer (ViT) with LORA. We will consider incorporating larger pre-trained models in the future work.

3. Use of More Complex Datasets: The included experiments using the DomainNet dataset are more challenging than CIFAR-100, featuring higher resolution images and a larger variety of categories. This inclusion should provide a more comprehensive assessment of our algorithm's performance and applicability.

We are grateful for your valuable suggestions. If further details or clarifications are needed on any of these points, please do not hesitate to let us know.

1. Presentation and Connectivity of Section 3 (Theory): We thank you for your insightful feedback on the presentation of Section 3. In the revised manuscript, we have streamlined the section to reduce verbosity and enhance its connection with the rest of the paper. We have also moved some detailed analyses to Appendix B.2 to maintain the flow while ensuring the availability of in-depth information.

2. Relevance and Validation of Proposition 3.1: We acknowledge your observation regarding the lack of validation of Proposition 3.1. To address this, we have included an empirical verification in Appendix E.2 concerning FL methods for reducing communication rounds, which cannot be achieved simply by increasing local epochs.

3. Clarification on Additional Error Term in Convergence Rate: Thanks for your correction. In the revised version, Section 3.2 has been updated to provide a clearer explanation and the correct error term index.

4. LSS Smoothness: In response, we have included empirical studies in Appendix E.2. These studies demonstrate the effect of LSS on the improved smoothness and provide evidence to support our theoretical claims.

5. Comparison with Stochastic Weight Averaging (SWA) and Random Interpolation: To clarify the differences with SWA and the concept of random interpolation, we have revised Section 3.3. This includes a more straightforward and less complicated description of our methodology, differentiating it from existing techniques like SWA.

6. Experimental Setup and Rationale for Different Local Update Steps: To address this, we have added new empirical evidence in Appendix E.2, showcasing the reasoning behind choosing different local steps for FedAvg.

We deeply appreciate the time and effort you have put into reviewing our work. Your feedback has been instrumental in enhancing the clarity and rigor of our manuscript. If any part of our response requires further elaboration or discussion, please let us know.