Enhancing Personal Decentralized Federated Learning through Model Decoupling

We greatly appreciate your suggestions and comments, which will make our work better than this version. Thanks for your positive comments, and we address the weaknesses and minor comments below.

W1: What do the "Grid" and "Exp" represent?

Thank you for your suggestion and we added the visualization of topologies in Appendix B.3 in the revision. The explanations of Grid and Exp are as follows:

• Grid: Grid topology refers to a network configuration where clients are arranged in a two-dimensional grid-like pattern. That means clients are organized in rows and columns, forming a rectangular or square grid structure. In a 100-client network, the exponential neighbors of client 10 are client 11(10+1), client 9(10-1), client 19(10+9), client 1(10-9).
• Exp: Exponential(Exp) topology refers to a network configuration where clients are arranged in an exponential manner. In a 100-client network, the exponential neighbors of client 10 are client 11(10+$2^0$), client 12(10+$2^1$), client 14(10+$2^2$), client 18(10+$2^3$), client 26(10+$2^4$), client 42(10+$2^5$), client 74(10+$2^6$).

W2: The value and the influence of $\lambda$ in Fig.3.

Thank you for your suggestion and we have added more analysis of the spectral gap $(1-\lambda)$ of different communication topologies to the revision. From the relationship between the spectral gap $(1-\lambda)$ and the participation number of clients $m$ ($m=100$ in our experiment), as follows, we can see that the convergence bounds of different topologies are ranked as follows: Fully-connected > Exponential > Grid > Ring, which is consistent with our experiment results.

Q1: Incorporation of SAM optimizer with other baseline methods.

We add more experiments about the incorporation of SAM optimizer with other baseline methods on CIFAR-10 as follows.

From the experimental results, SAM could similarly enhance the performance of the partial model personalized methods, but whether it could enhance the full model personalized methods is doubtful. For the partial model personalized methods FedPer, FedRep, FedBABU and DFedMDC, SAM adds proper perturbation in the direction of the shared models’ gradient to make the shared part more robust and flat, which will benefit the average progress for each client. For the full model personalized methods FedAvg, FedRod, Ditto and DFedAvgM, pursuing the full model flatteness will decrease the classifier’s adaptation to local distribution.

Thanks for your comments and we reply to them below. Please reconsider the contributions of our work.

W1: The doubts about Theorem 1 and Theorem 2.

• Only the $\mathcal{O}$ expression. $\mathcal{O}$ is the final results after assuming the learning rates $\eta_u$ and $\eta_v$. Similar results can be found in [1-3]. Due to the limited space, the initial convergence analysis has been presented in Formula (19) and Formula (33).
• The right-hand side of Eqs. (3)-(4) will goes to 0 as the number of rounds $T$ goes to infinity.
  • The gradient that goes to 0 is one of the optimality conditions in non-convex optimization. As an optimization problem, we consider minimizing problem: $\min _{x \in \mathbb{R}^d} f(x)$ and find the global minima $x^*$, that satisfies $f(x) \geq f(x^*)$. In non-convex optimization, the minima can be defined by $\nabla f\left(x^{\star}\right)=0$. So we prove the convergence from the gradients going to 0, which satisfies $\| \nabla f(x^T)\| \leq \epsilon$. More details about non-convex optimization can be referred to [4,5].
  • We assume that $\eta_u= \mathcal{O}(1/L_uK_u\sqrt{T})$ and $\eta_v= \mathcal{O}(1/L_vK_v\sqrt{T})$, which means that the learning rate $\eta_u$ and $\eta_v$ tend to 0 as the number of rounds $T$ goes to infinity, and the gradients eventually tend to 0 correspondently. We deduce that the reviewer wants to say that the proposed methods will converge to a variance under the constant learning rate. Actually, our methods converge to a variance domain with the sublinear order convergence speed.
  • Moreover, in a non-iid scenario, each client will achieve convergence and their gradient will approximate to 0 after sufficient large communication rounds. However, this does not mean that all clients have the same solution. For PFL, every client will own its unique solution $(u^*,v_i^*)$, the $v_i^*$ of which is different from each other.

W2: The convergence speed is monotonically related to the spectral gap.

• The convergence speed is monotonically related to the spectral gap. The convergence speed is related to the statistical heterogeneity $\delta$, the smoothness $L_u$, $L_v$, $L_{vu}$ of loss functions, the local epochs $K_u$ and the communication topology (1 − $\lambda$) in the theoretical results. Due to the limited space, we only present the final results in Formula (3) and Formula (4). Please check the details in Formula (20-21) and Formula (34-36).
• Whether Fully-connected topology is the optimal topology according to the theoretical results.

  • Theoretically, Fully-connected topology is the optimal topology among the comparison topologies and the comparison of the spectral gap of each topology is as follows ($m$ refers to the partition number of clients):

    Graph Topology	Spectral Gap $1-\lambda$
    Fully-connected	1
    Disconnected	0
    Ring	$\approx 16\pi^2/3m^2$
    Grid	$\mathcal{O}(1/(mlog_2(m)))$
    Exponential	$2/(1 + log_2(m))$

    It is clear to see that the convergence bounds of different topologies are ranked as follows: Fully-connected > Exponential > Grid > Ring. But it contains more communication cost compared with other topologies. So it is a trade-off between the convergence and communication cost in the real world. We have added the comparison to the revision.

  • Empirically, we conduct experiments on different topologies (i.e. Ring, Grid, Exponential and Fully-connected）in Fig 3, which suggests the personalized performance of different topologies is ranked as follows: Fully-connected > Exponential > Grid > Ring.

W3: The convergence results are not related to the number of workers.

Refer to the table above, the effects of the participation workers $m$ have been involved in the spectral gap $(1-\lambda)$ of communication topologies. We have added it to the revision.

[1] Decentralized federated averaging, ICML2022.

[2] Improving the Model Consistency of Decentralized Federated Learning, ICML2023.

[3]Federated Learning with Partial Model Personalization, ICML2022.

[4] Non-convex Optimization for Machine Learning.

[5] First-order methods in optimization.

M1: What's the used hyperparameter for baselines? Are the baselines’ results well fine-tuned?

The baselines’ results have been well-finetuned and the hyperparameter of the baselines can be found in Appendix B.3 (MORE DETAILS ABOUT BASELINES). We show the finetuned processing as follows.

• Setting: For all methods, we keep the experiment setting as the same as follows. We perform 500 communication rounds with 100 clients. The client sampling radio is 0.1 in CFL, while each client communicates with 10 neighbors in DFL. The batch size is 128. The learning rate is 0.1 with 0.99 exponential decay. The local epochs are set to 5 for full model personalized methods and the shared models of the partial model personalized methods. The local optimization is SGD with momentum 0.9 and 5e-4 weighted decay.
• Finetuning: For FedAvg, FedPer, FedBABU, Fed-RoD, DFedAvgM, there is no more hyperparameters. We show the finetuning progress for FedRep, FedSAM, Ditto, Dis-PFL on CIFAR-10 as follow:

  • FedRep: We finetune the local epochs for the personalized parts $K_v$ and set $K_v=10$ in our experiments.

    $K_v$	1	4	7	10	15
    Dir-0.3	88.50	90.71	90.81	91.09	90.79
    Pat-2	80.85	83.93	84.17	84.50	84.27

    Theoretically, multiple local epochs will speed up convergence but may be prone to over-fitting. It is a trade-off between convergence and generalization. The difference of the best epochs for the personal part demonstrates the difference between CFL and DFL.

  • FedSAM: The extra hyperparameter in FedSAM is the perturbation radius $\rho$ and we set $\rho = 0.7$ in our experiments.

    $\rho$	0.01	0.1	0.3	0.5	0.7	0.9
    Dir-0.3	78.78	79.08	79.20	79.61	80.02	79.52
    Pat-2	83.26	83.69	84.11	84.57	84.99	83.48

    With a larger $\rho$, the flatter models that benefit the aggregation average can increase the global generalization. But for PFL, it is a trade-off between the personalization and the generalization.

  • Ditto: The extra hyperparameter in Ditto is the interpolated coefficient $\lambda$ and we set $\lambda = 0.75$ in our experiments.

    $\lambda$	0.1	0.4	0.75	1.25	2
    Dir-0.3	64.77	70.63	73.51	72.73	69.50
    Pat-2	80.10	83.64	84.96	84.78	76.71

    $\lambda$ offers a trade-off between the personalization and the generalization. When $\lambda$ is set to 0, Ditto is reduced to training local models; as $\lambda$ grows large, it recovers the global model objective.

  • Dis-PFL:We finetuned the sparsity ratio in Dis-PFL and set it as 0.5 in our experiments.

    Sparsity Ratio	0.2	0.4	0.5	0.6	0.8
    Dir-0.3	80.17	82.63	82.72	82.56	81.94
    Pat-2	87.77	88.07	88.19	87.99	87.82

    The sparsity ratio is a trade-off between generalization and personalization. A higher sparsity ratio may bring more generalization benefits when the sparsity is small. But with the sparsity ratio increasing, less information exchange will lead to performance degradation.

M2: What is the definition of $\sigma$ in Theorem 2?

$\sigma$ is a variate associate with the gradient perturbation $\rho$ in SAM. We define $\sigma^2 =\frac{\rho^2}{K_u}+\frac{\sigma^2_u+\delta^2}{L_u^2}$ in Formula (24-26). And when assuming $\rho = \mathcal{O}(\frac{1}{\sqrt{T}})$, $\mathcal{O}(\sigma^2)= \mathcal{O}(\frac{1}{K_uT}+\frac{\sigma^2_u+\delta^2}{L_u^2})$.

Thanks for your comments and we reply to them below. Please reconsider the contributions of our work.

W1: Clarify the main novelty of this method.

We clarify more details about our background, motivation and contribution. Please check them in the top official comment.

W2: Why introduce the SAM?

• Intuitively, SAM adds proper perturbation in the gradient direction to make the shared part become more robust and flatter, which will benefit the model average between clients.
• Theoretically, we give the effect on bound after adding the operation of SAM in Remark 2, where assuming $\rho = \mathcal{O}(\frac{1}{\sqrt{T}})$. Due to the limited space, we have presented the bound before assuming $\rho = \mathcal{O}(\frac{1}{\sqrt{T}})$ in Formula (34) in Appendix C.2.2 on Page 26 as follows:
  • before using $\rho = \mathcal{O}(\frac{1}{\sqrt{T}})$: $\begin{aligned} &\frac{1}{T} \sum_{i=1}^T\left(\frac{1}{L_u} \mathbb{E}\left[\Delta_{\bar{u}}^t\right]+\frac{1}{L_v} \mathbb{E}\left[\Delta_v^t\right]\right) \leq \mathcal{O}\left(\frac{F\left(\bar{u}^1, V^1\right)-F^*}{\sqrt{T}}+\frac{\sigma_v^2\left(L_v+1\right)}{L_v^2 \sqrt{T}}+\frac{L_u^2 \rho^2+\sigma_u^2+\delta^2}{L_u K_u \sqrt{T}}\right. \left.+\frac{L_{v u}^2}{(1-\lambda)^2 \sqrt{T}}\left(\frac{\rho^2}{K_u}+\frac{\sigma_u^2+\delta^2}{L_u^2}\right)+\frac{L_u}{(1-\lambda)^2 T}\left(\frac{\rho^2}{K_u}+\frac{\sigma_u^2+\delta^2}{L_u^2}\right)\right) .\end{aligned}$
  • after using $\rho = \mathcal{O}(\frac{1}{\sqrt{T}})$: $\begin{aligned} &\frac{1}{T} \sum_{i=1}^T\left(\frac{1}{L_u} \mathbb{E}\left[\Delta_{\bar{u}}^t\right]+\frac{1}{L_v} \mathbb{E}\left[\Delta_v^t\right]\right) \leq \mathcal{O}\left(\frac{F\left(\bar{u}^1, V^1\right)-F^*}{\sqrt{T}} +\frac{\sigma_v^2\left(L_v+1\right)}{L_v^2 \sqrt{T}} +\frac{\sigma_u^2+\delta^2}{L_u K_u \sqrt{T}}\right. \left.+\frac{L_{v u}^2(\sigma_u^2+\delta^2)}{L_u^2(1-\lambda)^2 \sqrt{T}} +\frac{\sigma_u^2+\delta^2}{L_u(1-\lambda)^2 T}+ \frac{L_u}{K_u\sqrt{T^3}}+\frac{L_{vu}^2}{K_u(1-\lambda)^2\sqrt{T^3}}+\frac{L_u}{K_u(1-\lambda)^2T^2}\right) .\end{aligned}$
• Empirically, the comparison between DFedSMDC and DFedMDC in different heterogeneity and different topologies can significantly demonstrate the effectiveness in enhancing the consistency of the shared models in decentralized PFL.

W3: $V^{t}$ not $V^{t+1}$ in the theorem.

Thanks for your suggestion and we have polished it in the revision paper.

W4: "Why do all baselines achieve better performance under larger heterogeneity? "

What we focus on are the PFL problems, not the FL problems. In PFL tasks, the higher heterogeneity of data distribution means it owns fewer classes of data locally, which makes the classification task easier and clients will achieve better performance. For example, in the Pathological-2 setting, the local task is a binary classification task, which is easier than the five classification tasks in the Pathological-5 setting, so the average test performance in Pathological-2 is better than that in Pathological-5. The same phenomenon can be seen in most PFL works [1, 2, 3, 4].

W5: "The test performance will get a great margin with the participation of clients decreasing.’’

We have updated more experiments when the client number is 10 and 5 in Fig 5 in the revision paper. The test performance still improves with the participation of clients decreasing. And we clarify our opinion theoretically and empirically as follows:

• Theoretically, the convergence bound is related to the topology, which is associated with the participation of clients. The relationship between the topology and the participation clients $m$ is as below:

  Graph Topology	Spectral Gap 1-$\lambda$
  Fully-connected	1
  Disconnected	0
  Ring	$\approx 16\pi^2/3m^2$
  Grid	$\mathcal{O}(1/(mlog_2(m)))$
  Exponential Graph	$2/(1 + log_2(m))$

  When the participation clients $m$ decrease, the spectral gap increases and the convergence bound is tighter.

• Empirically, with the participation of clients decreasing, the number of local data increases. More data will help to achieve a better performance.

• When the client number is 1, the problem will become a classical ML problem based on a centralized data assumption, not an FL problem based on a decentralized data setting. We aim to collaboratively train the personalized models without transmitting data.

[1]On bridging generic and personalized federated learning for image classification. ICLR2022.

[2]Fedbabu: Towards enhanced representation for federated image classification. ICLR2022.

[3]Personalized federated learning with feature alignment and classifier collaboration. ICLR2023.

[4]FedProto: federated prototype learning across heterogeneous clients AAAI2022.

Dear Reviewer A4M3,

We sincerely appreciate the time and effort you have invested in reviewing our work. As the deadline for the discussion period is approaching, we kindly request that you inform us of any remaining questions you may have.

We are confident that our response has adequately addressed your concerns, and we would be grateful for your feedback. Should you require further clarification, we would be delighted to answer any additional questions and provide more information.

Best wishes,

The Authors

Thanks for your comments and we reply to them below. Please reconsider the contributions of our work.

W1: The novelty of our work.

We give more details about our background, motivations and contributions. Please check them in the top official comment box.

W2: Add the most recent decentralized federated learning method for a comprehensive comparison.

Thanks for your suggestion. We provide a comparison with DFedSAM in the following and we have added this comparison in the revision.

| Dataset   |   | Cifar-10 | |   |   | Cifar-100 | |   |  
|:-------------:|:-------|:------------:|:-----:|:-----:|:-------:|:-------------:|:-----|:------|
| Algorithm | Dir-0.1 | Dir-0.3     | Pat-2 | Pat-5 | Dir-0.1 | Dir-0.3       | Pat-5 | Pat-10 |      
| DFedSAM|84.96| 77.36|90.14|83.05|58.21|47.80|74.25|67.34| | DFedAvgM | 87.39   | 82.60       | 90.72 | 84.69 | 59.76   | 54.98         | 76.70 | 71.08 | | DFedMDC| 88.85   | 86.50       | 91.26 | 86.85 | 66.26   | 57.66         | 78.78 | 72.19 |       
| DFedSMDC  | 91.08 | 87.67|92.20|88.34|67.03|58.73|80.82|74.50|

From the comparison between DFedAvgM and DFedSAM we can see that adding the gradient perturbation to the full model will increase the model consistency but hurt the personalized performance. On the contrary, adding the perturbation to the shared representation parts in DFedSMDC outperforms DFedMDC clearly, which shows that both improving the shared representation parts consistency and keeping the private classifiers locally are both the key points to facilitate the personalized performance in decentralized PFL.

W3: compares the proposed method's convergence rate with the state-of-the-art (SOTA) approaches.

Compared with the SOTA bounds $\mathcal{O}\Big(\frac{1}{\sqrt{T}}+\frac{\sigma_l^2 + K \sigma_g^2}{K \sqrt{T}}+\frac{\sigma_l^2 + K\sigma_g^2+ KB^2}{K(1-\lambda)^2T^{3/2}}\Big)$ ( is the upper bound of the gradient) of existing work DFedAvg and $\mathcal{O}\Big(\frac{1}{\sqrt{KT}}+\frac{K(\sigma_g^2+\sigma_l^2)}{T}+\frac{\sigma_g^2+\sigma_l^2}{K^{1/2}(1-\lambda)^2T^{3/2}}\Big)$ of DFedSAM in decentralized works, our algorithms reflect the impact of the L-smoothness and the gradient variance of the shared model $u$ and personalized model $v$ on convergence rate. On the other hand, compared with the SOTA bound of FedAlt in personalized works, our algorithms reflect the impact of the communication topology $\lambda$ (the value of that increases when connectivity is more sparse, which has been studied by existing work [1].)

Therefore, the main contribution of our convergence analysis is the first to analyze the impact of model decoupling and the decentralized networks on the convergence rate instead of delivering the SOTA bound compared with others. Meanwhile, this theoretical analysis can be mutually verified with experimental results.

[1] Topology-aware Generalization of Decentralized SGD. ICML2023.

Dear Reviewer iEM9,

We sincerely appreciate the time and effort you have invested in reviewing our work. As the deadline for the discussion period is approaching, we kindly request that you inform us of any remaining questions you may have.

We are confident that our response has adequately addressed your concerns, and we would be grateful for your feedback. Should you require further clarification, we would be delighted to answer any additional questions and provide more information.

Best wishes,

The Authors