Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

Thank you for your constructive feedback on our manuscript. We appreciate your positive comments on the effectiveness and soundness of our proposed method, as well as the clarity of our writing.

Expermental Scope Clarification. We would like to clarify that the current set of experiments is sufficient to support our claims and demonstrate the effectiveness and generalizability of our proposed method. Additionally, other reviewers have acknowledged the comprehensiveness of our experiments. Reviewer 2F5f noted, "The proposed algorithm is tested on a variety of datasets and types of distribution shifts," and Reviewer 3kGS commented, "This paper conducts extensive experiments to illustrate the effectiveness of LSS."

However, we also recognize that extending our experiments could further enhance our experimental scope, particularly with initial explorations on large language models (LLMs).

We have addressed your concerns as follows:

1. Issue: Additional Evaluations on Large-Scale Models besides Image Classification

Action Taken. We evaluated our proposed method, LLSS, against FedAvg on a large language model for a multilingual instruction tuning task to demonstrate the generalizability of LSS. Additional details can be found in the Appendix.

Experiment Detail. Evaluation of Large Language Models for Multilingual Instruction Tuning. Setup. We follow the setup of Fed-Aya [1], which involves four iterative steps as common federated learning: server-to-client model downloading, local model training, client-to-server model uploading, and global model aggregation. For instruction tuning, we use the parameter-efficient fine-tuning technique, LoRA applied to the Llama2-7b model.

Dataset. We use the Aya dataset [1], a multilingual instruction tuning dataset with annotations from contributors worldwide. Our experiments include 6 high-resource languages (English, Spanish, French, Russian, Portuguese, Chinese) and 2 low-resource languages (standard Arabic, Telugu). The dataset is filtered to include contributors with at least 100 annotations, resulting in 38 clients with a total of 25k data samples.

Model. The model used for our experiments is the Llama2-7b, fine-tuned using the LoRA technique. We evaluate the effectiveness of the training methods using an in-domain evaluation metric termed Ref-GPT4, where GPT-4o rates the generated responses against ground-truth responses. The score given by GPT-Ref ranges from 0 to 10. We adopt the same prompt template used in FedLLM-Bench [1]. The implementation of applying our method to LoRA is the same as that used in the ViT experiments (see Fig.4 described earlier).

Result. In Figure 9 in the Author Rebuttal Attached File, our method, LSS, when applied to large language models for instruction tuning, achieves higher scores than the common FedAvg. This suggests that LSS is a promising approach for improving performance and convergence in federated learning settings for large language models, in addition to its success in image classification. Exploring the use of our method in a diverse set of complex LLM tasks is an interesting direction for future research.

[1] Ye, Rui, et al. "FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models." arXiv preprint arXiv:2406.04845 (2024).

2. Issue: Additional Evaluations on More Pre-trained Models

Clarification We have added evaluations on various pre-trained models, including ResNet18, ResNet50, ViT, and LLMs (llama2-7b). We believe our experimental evaluation covers a diverse range of model architectures.

We sincerely appreciate the time and effort you have invested in reviewing our work and providing such valuable feedback. We hope that our rebuttal has thoroughly addressed your comments and concerns.

As the author-reviewer discussion period nears its conclusion on August 13th, we would like to kindly invite you to contact us if there are any additional questions or points that you believe require further discussion. Your insights have been crucial in refining our submission, and we remain fully committed to addressing any remaining issues.

If you find that our responses and improvements adequately address your concerns, we would be deeply grateful if you could consider raising your score. Thank you once again for your thoughtful review.

We appreciate your thorough review and valuable feedback on our paper. Thank you for your positive comments on the comprehensiveness of our experiments and for recognizing the key contribution of our proposed two metrics (i.e., affinity and diversity) in quantifying model quality for model selection in model interpolation.

Below are our responses to the key points raised:

1. Comparison with Similar Methods

Response. We have discussed the differences between our method and the most relevant existing methods (model averaging) in the Method and Experiment section (see Line 202 to 220 and Line 300 to 302). Our proposed LSS consists of three novel key components: random model interpolation, diversity regularization, and affinity regularization, which is fundamentally different from FedProx.

Clarification. Compared with FedProx, our LSS contains random model interpolation and diversity regularization two novel components that help reduce the communication cost, the only similarity between LSS and FedProx is the presence of a regularization term: ours leverages affinity regularization, while FedProx uses proximal regularization. However, they differ significantly in their motivation, implementation, and focus.

(a) Motivation: The affinity term in LSS is designed to ensure that the expansion of low-loss regions remains close to the shared initialization point when incorporating diversity regularization. This increases the likelihood of overlapping connected regions (see Line 81 to 91), thereby enhancing performance and convergence by leveraging pre-trained models and significantly reducing communication rounds. In contrast, FedProx aims to improve general convergence and performance in the presence of statistical heterogeneity.

(b) Implementation: In LSS, the affinity term minimizes the distance of local models from the initialized model parameters (i.e., the pre-trained model parameters), rather than the global model (see Line 81 to 83, Algorithm 1 Line 2 and Require Description). In contrast, FedProx directly incorporates global model weights to regulate client loss.

(c) Focus: FedProx primarily introduces a regularization term to address general convergence and performance issues arising from statistical heterogeneity. In contrast, our method focuses on leveraging pre-trained models to achieve significant communication round savings. We approach this from a novel perspective, using model interpolation techniques to enhance FL convergence and performance. Additionally, we introduce an affinity regularization term to improve model quality during interpolation. Our affinity term is intrinsically linked to the diversity term and model interpolation process (see Line 81 to 94, and Line 252 to 255), making it an integral and non-isolated part of our approach.

Added Experiment. To clearly illustrate the distinct advantages of LSS, we have included comparative results with FedProx similar to those in Figure 3 in the Author Rebuttal Attached File. Fig.8. This comparison highlights the benefits of our approach in terms of communication efficiency and performance.

2. Mathematical Detail of Random Interpolation: The subsection 3.3.1 lacks mathematical detail to fully understand the interpolation process.

Response: We will revise subsection 3.3.1 to provide a more detailed mathematical explanation of the random interpolation process.

3. Communication Overhead: Potential significant communication overhead due to the requirement for clients to receive the interpolated model pool.

Clarification: We would like to clarify that our LSS method does not lead to additional communication overhead. Since we use the same model interpolation operation after model selection as model soups, the local model is merged from the interpolated model pool before communication. This ensures that our method maintains the same level of overhead as basic methods like FedAvg. In our revised version, we will introduce more background on model soups in Section 1 to improve clarity regarding communication overhead in parameters.

4. Theoretical Connection

Response. Theorem 3.1 aims to provide a convergence guarantee when our proposed regularization terms are added, not to derive these terms directly from the theorem. In revision, we will clarify the motivation and role of the regularization terms at the beginning of Theorem 3.1.

5. Reference and Statement Support

Response. We would like to clarify that we have already included references to support our statements in Section 3 (see Line 203). Although the limitation is not explicitly stated in the references, it can be clearly derived from the algorithm process. In revision, we will expand this part by adding more related work to better support our claims.

6. Experiment Scope: The results presented suggest LSS performs well in initial training rounds. It is unclear if LSS is advantageous throughout the training process.

Response: Yes, as indicated in Figure 3, our LSS performs well not only during the initial training rounds but also throughout the entire training process. Additionally, we would like to highlight that LSS is designed to reduce communication rounds when using pre-trained models. Training with 'continued post-initialization' is not the focus of our problem setting. Specifically, we aim to emphasize that LSS is tailored to minimize communication rounds in the context of pre-trained models. However, the design of our method should be compatible with existing algorithms for post-initialization training. Examining the effects of post-initialization training falls beyond the scope of our current paper, we will explore this topic in our future research work.

We believe these revisions significantly enhance the clarity of our paper. Thank you again for your insightful comments and suggestions.

Thank you for your thoughtful and constructive feedback. We appreciate that you found our figure illustrations clear and our experimental scope comprehensive, covering diverse datasets and types of distribution shifts. We also value your suggestions for improving our manuscript.

We have addressed your concerns as follows:

1. Notation Clarity

Revision:

• We will revise Algorithm 1, changing the interpolated model number notation from "n" to "N" (Line 4 in Algorithm 1) to distinguish it from the sample number.
• In Section 3, we will provide different notation and description for distribution and dataset to prevent any potential confusion. Change Line 136 to 137 into "For each client, we have access to $n$training data points in the form of `$(\mathcal{X}i, \mathcal{Y}i) = {(x{j}^{i}, y{j}^{i})}_{j=1}^{n}$, where $y_{j}^{i}`$ denotes the target label for input $x_{j}^{i}$. "
• We will add a reference note in Section 3.1 directing readers to the notation table in Appendix A for convenient access to corresponding notations and descriptions.""

2. Question: Could you explain intuitively why LSS is expected to perform well in most of the cases?

Response. We acknowledge that high affinity and high diversity do not necessarily guarantee an aggregated model with low loss due to the complexity of the high-dimensional loss landscape. However, our empirical evaluations show that using pre-trained model initialization, combined with our proposed random interpolation and the terms affinity and diversity, positively impacts performance and convergence across various types of distribution shifts.

Clarification. Our method performs well in most cases when using overparameterized pre-trained models. Typically, training with a pre-trained model results in small initial gradient dissimilarity, and overparameterization ensures that the optimal parameters are close to the initialization. See Section 3 (see Line 181 to 183) for more detailed explanations. In summary, small gradient dissimilarity among clients and a small parameter distance from initialization to optima, combined with leveraging our introduced diversity and affinity terms, make our method work well in most cases when training with pre-trained models.

3. Limitation: The method is tested only with pre-trained models. It is unclear how it would perform with randomly initialized models.

Response. Our LSS approach is also effective in improving performance and convergence when training models with random initialization. However, it does not achieve the impressive few-round near-optimal performance seen when training with pre-trained models.

Added Experiment. In response to your query, we have added performance results comparing the training with randomly initialized and pre-trained parameters in the Author Rebuttal Attached File Table 11.

Thank you again for your valuable feedback. We believe that these revisions significantly strengthen our manuscript, and we appreciate your feedback in making these improvements.

We sincerely appreciate the time and effort you have invested in reviewing our work and providing such valuable feedback. We hope that our rebuttal has thoroughly addressed your comments and concerns.

As the author-reviewer discussion period nears its conclusion on August 13th, we would like to kindly invite you to contact us if there are any additional questions or points that you believe require further discussion. Your insights have been crucial in refining our submission, and we remain fully committed to addressing any remaining issues.

If you find that our responses and improvements adequately address your concerns, we would be deeply grateful if you could consider raising your score. Thank you once again for your thoughtful review.