Soft-consensual Federated Learning for Data Heterogeneity via Multiple Paths

Thank you for your professional review. Below are our responses to your comments:

• Response to Weakness (a): Yes, the multi-path method introduces additional optimization for the extra path. However, we adopt the data adaptation strategy to reduce this additional computational and communication overhead. Moreover, the proposed method improves the performance of federated training and provides a novel perspective for federated learning, which is of exploratory significance. Future work can focus on further reducing the additional computational cost to mitigate this issue.

• Response to Weakness (b): Balancing different paths does introduce an additional hyperparameter. However, we can simply set it to 0.1 to avoid complicated hyperparameter tuning. We will include a hyperparameter analysis in the appendix. Below is the hyperparameter analysis result on the CIFAR-100 dataset.

• Response to Weakness (c): The swapping strategy indeed carries a potential risk of additional privacy leakage, but this risk is under a controllable level. First, due to the use of data adaptation, the swapping strategy does not require transmitting the full model; thus, the amount of leaked information is limited. Second, since each swap is randomly assigned by the server, it is difficult for clients to infer gradients from the two received models. Furthermore, explicit information such as raw data or label distributions is not directly revealed. Additionally, many existing approaches focus on privacy preservation in federated learning (e.g., differential privacy), and we provide relevant discussions on these methods in the paper.

• Response to Questions (a): Yes, the proposed multi-path method requires two networks to simulate two solution paths, which introduces additional parameters during training. However, during model inference, the network architecture and the number of parameters are identical to those of the original model without multi-path learning. Multi-path learning serves only as a training-time enhancement, effectively adding an auxiliary network in parallel to the original network. During training, the two networks optimize from different starting points, and a shared projection ensures consistency in the solution. At inference time, the auxiliary components are discarded, and only the original network is retained, thus no additional parameters are introduced in the final deployed model.

• Response to Questions (b): As addressed in our response to weakness (b), we provide the hyperparameter analysis in the appendix. In general, maintaining a balance between the two solution paths is essential for achieving correct multi-path learning outcomes. Moreover, under the swapping strategy, the shared projection is passed along with the local preference path, meaning this path has higher compatibility with the projection. Slightly increasing its weight helps stabilize the multi-path learning process.

• Response to Questions (c): Figure 9 shows the model’s accuracy (ACC) on local data before and after local training. Each communication round index on the horizontal axis represents two time points: the time points after model aggregation and after local training. As shown in the figure, compared to single-path learning, multi-path learning requires reaching a soft consensus between paths, making it harder to fit local data—this is reflected in the lower ACC after local training. On the other hand, because the model actively seeks a soft consensus between local and global features during training, more local knowledge is preserved during aggregation across clients. This figure illustrates that multi-path learning achieves a better balance between global and local information in federated learning, leading to an improved global model.

• Response to Questions (d): "Binding" means that the shared projection and the main path form a complete and paired solution. During the swapping strategy, this pair of models is assigned together to the same client, hence we refer to them as " Binding." "Swapping" refers to the specific execution step of the swapping strategy.

Thank you for your professional review. We will respond to each of your comments as follows:

• Response to Weakness (1): We will summarize all symbols used in the paper in a table, which will be added to the appendix. Additionally, each symbol will be explicitly explained at its first appearance in the text. | Symbol | Meaning | |--------|--------| | $\mathcal{D}_p$ | The dataset of the $p$-th client, which is a subset of the global data | | $\|\cdot\|$ | The number of elements in a set | | $\hat{\theta}$ | Global model parameters | | $\theta_p$ | Model parameters on the $p$-th client | | $\mathbf{X}^i_p$ | The $i$-th data sample of the $p$-th client | | $\mathcal{L}_p^{(l)}$ | The loss function for the local solution path on the $p$-th client | | $\mathcal{L}_p^{(g)}$ | The loss function for the global solution path on the $p$-th client | | $\gamma$ | The path balancing hyperparameter, used to balance the influence between local and global paths | | $\Phi_p^{(l)}(\cdot)$ | The mapping of the local solution path on the $p$-th client | | $\Phi_p^{(g)}(\cdot)$ | The mapping of the global solution path on the $p$-th client | | $\Psi_p(\cdot)$ | The shared projection on the $p$-th client | | $\phi^r_p$ | The model used by the $p$-th client in the $r$-th communication round to approximate $\Phi^{(l)}_p(\cdot)$ | | $\psi_p$ | The model used by the $p$-th client to approximate the projection $\Psi_p(\cdot)$ | | $p^{r'}$ | A random sample of client indices that participated in the previous round of training | | $\Omega_p(\cdot)$ | The task network on the $p$-th client | | $\tau_p^{(l),i}$ | The adapted data generated on the $p$-th client of the $i$-th data sample through the local path mapping | | $\tau_p^{(g),i}$ | The adapted data generated on the $p$-th client of the $i$-th data sample through the global path mapping |

• Response to Weakness (2): Data adaptation, as an auxiliary strategy of the multi-path method, helps reduce communication overhead. It employs adapters to transform data into a representation space recognizable by the task network, and applies the multi-path method to obtain better solutions for the adapters. In this way, only the adapters need to be exchanged during communication, rather than the entire model.

• Response to Weakness (3): We will add a hyperparameter analysis in the appendix. The following table shows the impact of different path balancing hyperparameters on the final performance of the global federated model on the CIFAR-100 dataset. Typically, we set the hyperparameter to 0.1, which aims to balance the two paths as much as possible, with the local preference path serving as the main path and the global preference path as the auxiliary. This is because the shared projection is transmitted along with the local preference path, making it natural to treat the local preference path as the primary one. Equations (6) and (7) describe the initial states of the local preference path models at the beginning of each local training round, which constitutes a formal description of the swapping strategy, indicating that at the start of each local training, the local preference path is initialized with a randomly selected client’s model.

• Response to Questions (2): These equations are indeed customized modifications of the multi-path scheme within the federated learning setting. The swapping strategy is a mechanism designed to maintain balance between the two paths in federated learning, preventing one path from becoming overly dominant. Equations (3), (4), and (5) describe the initial state of the model on client p without the swapping strategy, while Equations (6) and (7) describe the initial state on client p when the swapping strategy is applied.

• Response to Questions (3): As mentioned in our response to Weakness (3), we have added the experimental results for this hyperparameter analysis.

• Response to Questions (4): Indeed, when only the multi-path method is used without the swapping strategy, performance degrades. This precisely highlights the importance of correct multi-path construction for achieving good results. Without the swapping strategy, one path corresponds to a local model trained repeatedly on local data without communication, making it prone to overfitting on that local data. Consequently, during multi-path learning, this model would provide misleading or inappropriate guidance to the global model. Since our goal is to obtain a generalizable model that performs well across all clients, overly strong local bias is harmful to generalization. This is exactly why we adopt the swapping strategy. With swapping, we can still introduce local information that differs from the global model, using a model derived from additional local data as the starting point for one optimization path, thereby enabling a proper multi-path construction. In summary, providing an incorrect path can degrade performance even when using a multi-path framework. Hence, additional strategies are necessary.

After reading the response, most of my concerns have been solved. I maintain my score.

Thank you very much for your questions and professional comments. Below are our responses:

• Response to Weakness (1): In our experiments, we found that if the swapping strategy is not adopted and the multi-path solving approach is used alone, one of the solution paths would continuously receive local data, leading to overfitting. This significantly degrades the performance of the multi-path method, as demonstrated in our ablation studies: without the swapping strategy to weaken the local-preference path, this path becomes overly dominant and unbalanced, which is an incorrect usage of multi-path learning and results in performance degradation. Therefore, we adopt the swapping strategy to balance the influence of local data on the model. The multi-path method requires two solution paths that optimize from different initial states toward the final models, which can be regarded as two distinct solution paths. If the initial state is close to a suitable value (i.e., derived from actual data rather than random initialization), it reduces the optimization difficulty of the model. Hence, we use the swapping strategy to obtain reasonable initial values from other clients for constructing the multi-path setup. Since this path originates from a model trained on a specific client, we distinguish it from the initial state derived from the global model by referring to it as a local feature preference. This terminology is introduced solely to differentiate the two solution paths. The term "local feature preference" indicates that, unlike the global model, this model has only been exposed to limited data, preserving unique information that has not yet been overwritten or forgotten by other data. This provides a unique interpretation of the current client's data, and this path is expected to reach consensus jointly with the global model, helping to avoid the mediocre solutions that may arise from relying solely on the global model. In summary, the swapping strategy is employed to ensure that, while maintaining multiple solution paths, the starting points of these paths are meaningful and appropriate.

• Response to Weakness (2): Yes, it does. Therefore, we provide experimental results without data adaptation, in which the task network is fully optimized using the multi-path method, achieving better performance. Below is a comparison between the full multi-path version and the method described in the main text. The detailed experiment will be updated in the paper.

• Response to Weakness (3): Here, $L_p$ denotes the loss function required for training the task on a client. For the image classification experiments in this paper, we use cross-entropy as the loss function.

The response addresses my concern, so I will raise my score.

Thank you very much for your professional comments. We will respond to each of your concerns as follows:

• Response to Weakness (i): The goal of our proposed method is to seek a better global model in federated learning, and the use of image data is merely a way to validate the effectiveness of our method under federated settings. A suitable federated approach should not be restricted to a specific data type. Our original intention is to develop a method for obtaining a better global model under federated requirements, and thus we use image data as a case for validation. Similar to image classification, many tasks fundamentally involve learning data embeddings and modeling target probability distributions; hence, our method can be easily extended to tasks on other datasets. We have conducted additional experiments on the Vehicle Sensor dataset, a vehicle dataset containing 23 groups of sensor data, where each sensor group serves as the data for one client. This experiment will be included in the appendix. The following table presents the results of our method compared with several baselines: | Method | FedAvg | FedProx | SCAFFOLD | FedPVR | FedUCS | FedMP | |----------|--------|---------|----------|------------|--------|-------------| | ACC | 86.33 | 86.12 | 87.14 | 87.71 | 87.53 | 87.93 |

• Response to Weakness (ii): We will add a line plot showing the impact of the path balancing hyperparameter on the performance of the global model. Below is the table for the hyperparameter analysis experiment in CIFAR-100 dataset. | $\gamma$ | 0.01 | 0.05 | 0.1 | 0.5 | 1 | 5 | 10 | 50 | 100 | |--------------|-------|-------|-------|-------|-------|-------|-------|-------|-------| | $\alpha$=0.3 | 60.76 | 60.46 | 61.43 | 60.17 | 59.22 | 2.57 | 23.92 | 1.00 | 1.00 | | $\alpha$=0.5 | 61.35 | 61.13 | 62.04 | 60.91 | 60.18 | 9.19 | 16.33 | 1.00 | 1.00 | | $\alpha$=1.0 | 63.10 | 62.94 | 62.95 | 62.89 | 62.76 | 19.21 | 14.50 | 1.00 | 1.00 |

• Response to Weakness (iii): We will revise or supplement the relevant descriptions based on your feedback.

• Response to Questions (i): As mentioned in our response to Weakness(i), the objective of our proposed method is to improve the global model in federated learning, and the use of image data is only one way to verify the effectiveness of our method under federated settings. A proper federated method should not be limited to specific data types. Therefore, we have added an experiment on an additional data type. As shown in the above table, our method still achieves a certain level of performance improvement.

• Response to Questions (ii): Here we would like to clarify that, for the multi-path method, the inference structure of the original model does not need to be changed. Therefore, during the final testing phase, the model used has the same architecture as the original model, and thus the number of parameters remains unchanged. The multi-path method only requires parallel auxiliary network structures during training to simulate different solution paths, which introduces additional auxiliary parameters. However, after training is completed, these auxiliary parameters are no longer involved in the model inference process. Hence, the improvement in federated model performance is not due to an increase in the number of parameters.

• Response to Questions (iii): We connect an adapter between the task network, and the data to transform the raw data into a format suitable for the task network. For multi-path solving, we also attach a shared projection adapter after the task network to convert its output into the desired target output. In this case, the multi-path solving process is transformed from optimizing the entire model to optimizing only the adapters. As a result, only the adapter parameters need to be communicated, reducing communication overhead.

• Response to Questions (iv): We believe the scenario you mentioned does not qualify as multi-path learning. The key of multi-path learning lies in having some distinct paths that negotiate and reach a consensus solution. Simply training under the same loss function without generating a consensus solution does not constitute multi-path learning. In our proposed multi-path construction, two parallel networks feed into a shared projection module, which maps their outputs into the target space (i.e., the desired output). This projection can be viewed as a consensus reached through the collaboration of the two parallel networks, with each network representing a separate solution path. Only approaches that involve such a negotiation process can be considered the multi-path learning.

• Response to Questions (v): The multi-path scheme is a general idea for finding consensus between two solution paths. How to concretely implement such a multi-path solution within the federated learning framework requires further exploration. In this paper, we combine a global-preference path with a local-preference path to achieve a soft consensus, which addresses the conflict between global and local knowledges in federated learning. Therefore, this constitutes a federated learning method based on multi-path learning.