Improving Generalization in Federated Learning with Model-Data Mutual Information Regularization: A Posterior Inference Approach

We would like to thank for the comments, and our response can be found in the following.

Comment 1-The writing is sometimes difficult to follow:

1. Due to the space limit of the main text, we placed the theoretical proof, more detailed analysis, and Algorithm 1 in the Appendix of our submission. To improve readability, we will include these core steps of theoretical analysis and Algorithm 1 in the main text in the final version of this paper, where there is one additional page allowed.

2. Line 10 of Algorithm 1 represents the local SGLD update, which has been reflected in Figure 1 as the “SGLD for sampling” step. In the modified version, we will also reflect the global sliding average step (i.e., Line 16 of Algorithm 1) in Figure 1.

Comment 2-A Comparison of Computational Cost:

Please see our response to common Comment 1. Here, we first provide a complexity analysis concerning the computation time, storage, and communication for our method compared to other Bayesian methods. Subsequently, our experiments examine the relationship between computation time and model dimension within a single communication round. It is shown that the computation time of our algorithm is significantly superior to that of FedEP.

Comment 3-Additional Experiments on ResNet-18:

Following this suggestion, we have evaluated the performance of our FedMDMI by using ResNet18 on the CIFAR-10 and CIFAR-100 datasets, with the following observations.

Table 13 shows that our FedMDMI still outperforms other comparison methods in terms of the generalization performance, with the ResNet-18 architecture. Here, we replace the batch norm with group norm and set the number of clients to 20, such that the 10% and 5% sampling rates correspond to only two and one client participating in training per communication round, respectively. This may highlight the robustness of our FedMDMI to some possible model-architecture changes and its ability to adapt to various models.

Comment 4-Additional datasets provided by TensorFlow Federated and LEAF:

Please note that in the original manuscript, we used LEAF to generate the heterogeneous Shakespeare dataset, with each client associated with a speaking role comprising a few lines. Following this advice, we have further used TensorFlow Federated to generate the Stack Overflow dataset. Similar to [D1], due to the limited graphics memory and rebuttal time, we utilize only a sample of 200 clients from the original dataset, with the following observations.

As shown in Table 14, on the Stack Overflow dataset, our FedMDMI continues to outperform the majority of other optimization algorithms designed to address the data heterogeneity issue. This can be attributed to our proposed model-data mutual information regularization that enhances generalization.

[D1] G. Cheng et al., "Federated asymptotics: a model to compare federated learning algorithms," in Proc. AISTATS, 2023.

Comment 5-Further Clarification on Fig. 3:

i) Figures 3(a) and 3(b) provide an effective visual representation of model calibration. In a well-calibrated model, the difference between confidence and accuracy should be close to zero for each bin. Our FedMDMI (black curve) shows a closer alignment to zero across most bins, indicating superior calibration. This is further supported by Table 3, where our FedMDMI exhibits the lowest expected calibration error in most cases, particularly on the CIFAR-100 dataset.

ii) Figures 3(c) and 3(d) show the convergence behavior of different algorithms. Here, we would like to show that though our FedMDMI does not converge the fastest, especially compared to the traditional optimization-based algorithm SCAFFOLD, it still slightly outperforms the other Bayesian inference-based baselines. In fact, we have also given an explanation on this observation in Line 358 of the original manuscript: "One potential explanation is our use of stochastic gradient Langevin dynamics (SGLD) to approximate the posterior, which often suffers from a slow convergence rate due to the variance introduced by the stochastic gradient."

iii) Figures 3(c) and 3(d) also show that the final convergence of our FedMDMI achieves higher accuracy. This was further demonstrated in Table 2. Since the CIFAR-10 dataset is relatively simple, the baselines can easily achieve high accuracy, making our improvement less significant. However, for the more complex datasets like CIFAR-100 and Shakespeare, our FedMDMI shows a more significant improvement.

The volatile learning curve mentioned by the reviewer may be due to the fact that it was plotted averaged over 5 random seeds and drawn by sampling 25 points apart, resulting in a curve that is not completely smooth. In addition, we will show here the standard deviation of the accuracy on each convergence curve when it is close to convergence (i.e., 2000-4000 rounds), as follows.

For CIFAR10 on Dir (0.2)-L: FedAvg (0.44%), FedPA (0.74%), FedEP (0.34%), FedBE (0.51%), FALD (1.17%), FedMDMI (0.28%).

For CIFAR100 on Dir (0.2)-L: FedAvg (0.58%), FedPA (1.19%), FedEP (0.85%), FedBE (0.69%), FALD (0.96%), FedMDMI (0.79%).

Such experimental results show that, in most cases, our FedMDMI is less volatile during the convergence stage compared to other posterior inference-based algorithms.

Comment 6-Discussion on Computational Complexity:

Please also see our response to common Comment 1.

For the batch size, in the original manuscript it was set to the default value of 50 for CIFAR-10 and CIFAR-100, following the same setting as in the other baselines. We have also conducted additional experiments to demonstrate the effect of batch size on performance. The results are shown in Table 12. Our method demonstrates greater robustness across different batch sizes compared to other baselines.

Dear Reviewer hP14,

Many thanks for the time and effort that you have dedicated to reviewing our paper and providing these insightful comments, which will further help enhance the quality of the final version of our manuscript.

We would like to thank for the comments, and our response can be found in the following.

Comment 1-Further clarification on Our Motivation:

We agree with the reviewer that in centralized learning, posterior inference is proposed to provide a more reliable assessment of model uncertainty than point estimation. This uncertainty estimation is also crucial in the safety-critical applications of federated learning, such as autonomous driving, healthcare, and finance. Consequently, there has been some recent literature, such as FedPA and FedEP, considering the introduction of posterior inference into the federated setup.

However, we found that these Bayesian inference-based federated learning algorithms do not adequately address the data heterogeneity issue. Therefore, we are motivated to propose a mutual information regularization to enforce the global posterior to learn essential information from the heterogeneous local data, thereby improving the generalization capability. To optimize this regularization, we further employ a series of techniques, including the global mutual information decomposition, PAC-Bayesian conclusions, and stochastic gradient Langevin dynamics (SGLD) sampling.

Comment 2-Additional Experiments on Scarce Local Data:

Thank you for your comments. We have conducted additional experiments to analyze the impact of the number of local data samples. Given that the total number of data samples in CIFAR-10 and CIFAR-100 is fixed, we controlled the number of samples on each client by adjusting the number of clients. With fewer data samples on a client, the local model is more prone to overfitting. The results are shown in Table 11. Our FedMDMI maintains superior performance even with scarce local data, demonstrating that our MI regularization-based posterior estimation effectively alleviates the overfitting caused by data scarcity.

Comment 3-The difference between mutual information terms, I(w; S) and I(w; D):

In Eq. (20), $I(w; S)$ relates to the participating generalization error, i.e., the difference between the empirical and expected risk for the participating clients. This serves as the regularized term that can be estimated in our FedMDMI. On the other hand, $I(w; D)$ relates to the participation gap, i.e., the difference in the expected risk between the participating and non-participating clients. This term, however, cannot be estimated due to the unavailability of the non-participating clients.

Here, we will illustrate the difference with a simple example. Consider an extreme case with 10 clients, each holding only one label of the MNIST dataset. In this scenario, we restrict federated training on the first 9 clients, which contain train dataset labeled 0-8. During the test phase, the error measured on test sets from labels 0-8 is referred to as the participating generalization error $I(w; S)$. The error measured on the data from the $10$-th client, which holds label 9, is termed as the participation gap $I(w; D)$.

Thus, $I(w;D)$ and $I(w;S)$ do not represent a compromise or antagonistic relationship. By optimizing $I(w;S)$, we can improve the performance of the learned model on the distribution seen during training. Consequently, if the distribution of clients that have never participated is the same as the distribution seen by trained clients, then $I(w;D)$ degenerates to $I(w;S)$. In other words, our FedMDMI can only guarantee improved generalization on the distribution seen during training, but not on the out-of-distribution data from non-participating clients. This could motivate an interesting future direction: reducing $I(w;D)$ through techniques like transfer learning or domain generalization.

Comment 4-What are the conditions under which it is possible to decompose the posterior of the global model into the product of posteriors of the local models?

The decomposition of the global posterior into the product of posteriors of the local models is based on two assumptions:

1. The global likelihood is conditionally independent given $w$, i.e., $p(S_1,\dots,S_m | w) = \prod_{i=1}^m p(S_i | w)$.
2. The ratio of the global prior to the product of client priors, $\frac{p(w)}{\prod_{i=1}^m p_i(w)}$, is considered a constant based on some prior assumptions, such as Gaussian priors or uniform priors.

These assumptions are widely recognized and significant across various fields, including in PoE [C1] and (EP)-MCMC [C2, C3], and Bayesian FL [C4].

[C1] Hinton, Geoffrey E., "Training products of experts by minimizing contrastive divergence," Neural computation 14(8): 1771-1800, 2002.

[C2] Neiswanger, Willie, Chong Wang, and Eric P. Xing, "Asymptotically exact, embarrassingly parallel MCMC," in Proc. UAI, 2014.

[C3] Wang, Xiangyu, and David B. Dunson, "Parallelizing MCMC via Weierstrass sampler," arXiv preprint arXiv:1312.4605, 2013.

[C4] Al-Shedivat, Maruan, et al., "Federated learning via posterior averaging: A new perspective and practical algorithms," in Proc. ICLR, 2021.

Comment 5-In the conclusion, you claim to show that the optimal posterior is the Gibbs posterior, but as I understand, you used this result from previous research?

We agree with the reviewer that this is not our contribution. To avoid confusion, we emphasized in the original manuscript that "this conclusion is well-established in the field of PAC-Bayesian learning" in the Introduction (Line 64). We also stated that, "In order for our paper to be self-contained, we re-state the proof from [12, 44] here for the optimal posterior" in the Proof (Line 988). To avoid overclaiming our contribution, we will further emphasize that the result of the optimal posterior being the Gibbs posterior stems from previous research.

We would like to thank for the comments, and our response can be found in the following.

Comment 1-Discussion on tightness of the upper bound:

First, we are constrained in practice to only leverage local data $S_i$ at individual client $i$ under the FL settings. Alternatively, based on the Global Model-Data MI Decomposition in Proposition 4.1, we have: $I(w ; S) = \sum_{i=1}^m \left[ I\left(w ; S_i\right) - I\left(S_i ; S^{i-1}\right) \right] \le \sum_{i=1}^m I\left(w ; S_i\right).$ Here, we can upper bound $I(w ; S)$ by the sum of local MI terms $I(w ; S_i)$ that can be estimated locally by these clients. By doing so, we also introduce an estimation error, i.e., the data correlation $I\left(S_i ; S^{i-1}\right)$ between the individual clients, which determines the tightness of the proposed upper bound $\sum_{i=1}^m I\left(w ; S_i\right)$. This data correlation $I\left(S_i ; S^{i-1}\right)$ between the individual clients is related to the clients' data generation and collection, which is intractable to estimate and optimize.

Then, each local MI term $I(w; S_i)$ can be further expressed as:

= \mathbb{E}\_{p(S_i)}[\operatorname{KL}(p(w \mid S_i) \| r(w))]-\operatorname{KL}[p_i(w) \| r(w)] \leq \mathbb{E}\_{p(S_i)}[\operatorname{KL}(p(w \mid S_i) \| r(w))]\],$$ where $p_i(w) \triangleq \mathbb{E}_{p(S_i)}[p(w | S_i)]$ denotes the oracle prior and $r(w)$ is an arbitrary prior distribution that is used to approximate $p_i(w)$. This may incur another estimation error, which is the difference between the oracle prior $p_i(w)$ and the prior $r(w)$ we actually use, i.e., $\operatorname{KL}\left[p_i(w) \| r(w)\right]$. As indicated in Line 240, [B1] emphasizes that to achieve a small KL divergence, the prior must, in essence, predict the posterior. Specifically, [B1] employs a distribution of pre-trained models derived from a portion of the untrained data as prior $r(w)$. In our work, we propose using the global model in the previous round as the mean $\mu$ of Gaussian prior and the uncertainty introduced by all clients in the previous round as the covariance $\Sigma^{-1}$ in the prior, as shown in Eq. (18) and Line 6 in Algorithm 1 of our FedMDMI. This incorporates the global data information and potentially helps predict the local priors based on the global posterior decomposition. [B1] G. Dziugaite _et al._, "On the role of data in PAC-Bayes bounds," in _Proc. ICML_, 2021. **Comment 2-Additional baselines:** Please see our response to common Comment 2. **Comment 3-Minor typos:** Thank you for pointing out the small issue with collapsed Lines 142-143, and we will fix it. Additionally, in Figure 2, the ordinate represents the percentage error. Specifically, the left figure indicates that the training error is within the range of [0, 1%], while the right figure indicates that the test error is within the range of [0, 20%]. **Comment 4-Relationship between Overfitness and Privacy of Training Data:** The MI term $I(w;S)$ regularizes the fitness of a local model. In fact, the hyperparameter $\alpha$ controls the degree of the MI regularization and embodies a trade-off between privacy and fitting (or trade-off between generalization and fitting). As highlighted in Line 275, the proposed MI regularizer $I(w;S)$ directly quantifies the extent to which the model memorizes data. A larger value of $I(w;S)$ implies that the model fits the data well, thus signifying the fitting. Conversely, a smaller $I(w;S)$ indicates that the model avoids memorizing excessive data details, thereby promoting generalization and, from another perspective, privacy protection. In fact, some work [B2] demonstrates that in certain cases, knowing the previous model and the gradient update from a client can allow one to infer a training example held by that user. Therefore, for stronger privacy protection, differential privacy can be used to encrypt these models or gradient updates. Our work shows that regularizing the MI term can provide a form of differential privacy protection as a byproduct. [B2] P. Kairouz _et al._, Advances and open problems in federated learning, Foundations and Trends in machine learning, 2021. **Comment 5- Further Clarification on Factor $\delta$ :** Here, we would like to explore and offer further insights, from the perspective of FL, on how minimizing mutual information can help mitigate bias resulted from the data heterogeneity. In FL, we denote the bias factor as $\delta$ that affects the generation of local heterogeneous data (such as the difference in clients' locations, preferences, or habits). If we can effectively eliminate the influence of the bias factor $\delta$, the entire dataset would become homogeneous (similar), thereby reducing bias among local posteriors. Indeed, directly eliminating the impact of bias factor is impractical. But leveraging Markov chain $\delta \rightarrow S \rightarrow w$ allows us to directly infer that $I(w; \delta) \le I(w; S)$, where the inequality holds due to the **Data Processing Inequality (DPI)**. As a consequence, we can diminish $I(w; S)$ to constrain $I(w; \delta)$, thereby rendering the model $w$ insensitive to the bias factor $\delta$ from the diverse clients. Furthermore, similar analyses are prevalent in other contexts. For instance, in centered learning, literature [B3] (Section 2.2 on Page 6) assumes that nuisances affect the observed data, and these nuisances are mitigated through information bottleneck regularization (another well-known mutual information). [B3] A. Achille _et al._, "Emergence of invariance and disentanglement in deep representations," Journal of Machine Learning Research, 2018. **Comment 6-Comparison between MCMC and Variational Approximation (VA):** Please see our response to common Comment 3.

We would like to thank for the comments, and our response is as follows.

Comment 1-Further Clarification on Fig.3: Fig. 3 visualizes uncertainty calibration and convergence, with quantitative results in Tables 2 and 3 highlighting improvements of our FedMDMI.

i) Figs. 3(a) and 3(b) provide an effective visual representation of model calibration. In a well-calibrated model, the difference between confidence and accuracy should be close to zero for each bin. Our FedMDMI (black curve) shows a closer alignment to zero across most bins, indicating a superior calibration. This is further supported by Table 3, where our FedMDMI exhibits the lowest expected calibration error in most cases, particularly on CIFAR-100 dataset.

ii) Figs. 3(c) and (d) illustrate convergence behavior. Although FedMDMI converges slower than SCAFFOLD, it slightly outperforms other Bayesian baselines. This is explained in Line 358 of the manuscript, attributing the slower convergence to the variance from stochastic gradient Langevin dynamics (SGLD).

iii) Figs. 3(c) and (d) also show that the final convergence of our FedMDMI achieves higher accuracy. This was further demonstrated in Table 2. Since CIFAR-10 dataset is relatively simple, the baselines can easily achieve high accuracy, making our improvement less significant. However, for the more complex datasets like CIFAR-100 and Shakespeare, our FedMDMI shows a more significant improvement.

iv) Regarding our response to Comment 6, the computational and storage complexity of our FedMDMI is lower than that of most other Bayesian FL algorithms.

Comment 2-Performance differences in Tables 2 and 3: Please note that results presented in Tables 2 and 3 are averaged over five random seeds, as indicated in the caption of both tables.

Comment 3- Degree of heterogeneity and effectiveness: Yes, we agree with the reviewer. As the degree of heterogeneity increases, performance of all baselines decreases, but our method's performance decreases less significantly. This indicates that our FedMDMI is more robust to increasing heterogeneity compared to the baseline.

Specifically, for CIFAR-10, as the degree of heterogeneity increases from Dir(0.7)-H to Dir(0.2)-H, performance of our FedMDMI and FedAvg decreases by 0.2% and 0.63%, respectively. Similarly, for CIFAR-100, performance of our FedMDMI and FedAvg decreases by 1.01% and 2.03%, respectively. This trend holds in most cases, indicating that our proposed posterior inference approach based on model-data mutual information regularization can effectively alleviate the impact of data heterogeneity.

Comment 4-Discussion of $\alpha$: The Lagrange multiplier $\alpha$ in Eq. (7) balances fitting ($L_{S_i}(w)$) and generalization ($I(w; S_i)$). Fig. 2 illustrates that as $\alpha$ increases, the gap between test errors across varying data heterogeneity decreases, indicating the efficacy of our proposed regularization in addressing data heterogeneity. Specifically, increasing $\alpha$ leads to a gradual rise in train error and an initial drop followed by an increase in test error, suggesting that lower complexity (larger $\alpha$) may cause underfitting, while higher complexity (smaller $\alpha$) risks overfitting. Thus, larger $\alpha$ mitigates the bias from data heterogeneity.

Comment 5-Additional Baselines: Please see common comment 2.

Comment 6-Complexity Analysis: Please see common comment 1.

Comment 7-Comparison of MCMC and VA: Please see common comment 3.

Comment 8-Effectiveness of MI regularization: As discussed in the Related Work section, Model-Data Mutual Information (MDMI) regularization is a well-established concept in centralized learning, supported by both theoretical analyses and extensive experimental verification. For instance, [A1] demonstrates that MDMI regularization outperforms L2-norm regularization and dropout through Fisher information matrix estimation.

For FL, we propose a federated posterior inference method to estimate the global mutual information via global MDMI Decomposition. Our experiments indicate that MDMI regularization is equally effective in FL.

[A1] Z. Wang et al., "Pac-Bayes information bottleneck." in Proc. ICLR, 2022.

Comment 9-Difference between mutual information terms: In Eq. (20), $I(w; S)$ relates to the participating generalization error, i.e., the difference between the empirical and expected risk for participating clients. This serves as the regularized term that can be estimated in our FedMDMI. On the other hand, $I(w; D)$ relates to the participation gap, i.e., difference in the expected risk between participating and non-participating clients. This term, however, cannot be estimated due to unavailability of the non-participating clients.

Here, we will illustrate the difference with a simple example. Consider an extreme case with 10 clients, each holding only one label of MNIST dataset. In this case, we restrict federated training on the first 9 clients, which contain train dataset labeled 0-8. During the test phase, the error measured on test sets from labels 0-8 is referred to as the participating generalization error $I(w; S)$. The error measured on data from the $10$-th client, which holds label 9, is termed as the participation gap $I(w; D)$.

Thus, $I(w;D)$ and $I(w;S)$ do not represent a compromise or antagonistic relationship. By optimizing $I(w;S)$, we can improve performance of the learned model on the distribution seen during training. Consequently, if the distribution of clients that have never participated is the same as the distribution seen by trained clients, then $I(w;D)$ degenerates to $I(w;S)$. In other words, our FedMDMI can only guarantee improved generalization on the distribution seen during training, but not on the out-of-distribution data from non-participating clients. This could motivate an interesting future direction: reducing $I(w;D)$ through techniques like transfer learning or domain generalization.

Dear Reviewer 4nGu,

Thank you for the updated review. We have provided our response to each of these concerns in the following.

Comment 1-Further Clarification on Fig. 3: Regarding the plots in Figs. 3(a) and 3(b), as noted by the reviewer, the lines all lie very close together and very far from zero. This observation actually highlights a key challenge in the FL settings: the scarcity of data samples at each client increases the risk of local model overfitting, leading to overconfident decisions and poor uncertainty estimation in the aggregated global model. This also underscores the necessity of incorporating Bayesian inference into FL. As further illustrated in Table 3, the posterior-based approach demonstrates a significantly better performance in uncertainty estimation as compared to the point-based approach.

Besides, following this suggestion, we will also include the shaded error bars representing variance in all plots of Fig. 3 in the final version of our manuscript.

Comment 2-Performance differences in Table 3: Following this suggestion, we will include the variance of the ECE values in Table 3, as follows.

Table 1: ECE (with the variance of the ECE values) under various setting.

Comment 3- Degree of heterogeneity and effectiveness: First, we would like to clarify that most of the Bayesian-based comparison methods, including FedBE, FedPA, and FedEP, are specifically designed to address the data heterogeneity issue in FL. We apologize for not clarifying this point earlier due to the word limit on the first-round response.

Besides, following the reviewer's suggestion, we have also conducted additional experiments (iid and Dir (0.1)), observing that as the degree of heterogeneity increases, the generalization performance of all the comparison algorithms declines. However, the generalization performance of our FedMDMI decreases to a lesser extent in the most cases. This indicates that our FedMDMI is more robust to the increasing heterogeneity as compared to the other baselines. Since we cannot display images here, we present these results in the following table form for now, and we will convert it to line plots in the final version of our manuscript.

We have not yet reached a conclusion on how data heterogeneity affects the ECE. It is clear that uncertainty estimation is influenced by local sample size, and smaller sample sizes tend to lead to overconfident decisions. On the other hand, data heterogeneity has a more pronounced effect on accuracy.

Table 2: Test Accuracy (%) v.s. Data heterogeneity.

Table 3: ECE v.s. Data heterogeneity.

Comment 1-Complexity analysis: We did provide the complexity analysis in terms of the computation and memory for a single communication round. To calculate the total time consumption required to reach the convergence, we then have to multiply the computation time per round with the total number of communication rounds $T$ needed for convergence. In the following, we begin by providing a theoretical analysis of the total number of communication rounds required for convergence.

i) Theoretical convergence analysis. For our proposed FedMDMI, as shown in Lines 230-232 of our manuscript, and leveraging the insights from [A3], we provide the convergence rate of our FedMDMI under the $\mu$-strongly convex objective as: $\mathcal{O}\left((1-\gamma \mu /8)^{KT} + {1/m} + d\right)$, where $m$ is the number of clients, $K$ is the number of local updates, $d$ is dimensions of the model, and $T$ is the number of communication rounds. Thus, the convergence rate scales linearly with the model dimension $d$. Reference [A4] shows that the convergence rate of FedAvg on the $\mu$-strongly convex objective is: $\mathcal{O}\left(\mu\exp({-\frac{\mu}{16(1+B^2)}T}) +{1/(mKT)}\right)$, where $B$ is the gradient dissimilarity. Notably, neither FedPA nor FedEP presented the convergence rate in their original papers.

ii) Empirical convergence behavior. From the convergence curves illustrated in our manuscript (i.e., Figs. 3(c) and 3(d)), it can be empirically observed that the number of communication rounds required for convergence of our FedMDMI is less than that of FedPA and FedEP.

Then, we try to respond to the reviewer's concern on variational approximation (VA) methods being generally faster than sampling-based approaches in real-world problems. It is however observed in our rebuttal that FedEP, which uses the expectation propagation to obtain a VA of the global posterior, consumes more time per communication round and requires a greater number of communication rounds compared to our FedMDMI. The reasons are explained in the following.

i) Complexity comparison per communication round. As stated in our first-round response to the common comment 1, FedMDMI only requires generating and computing the Gaussian noise at each round, while the sequence of global model $w_t$ converges to Gibbs posterior with sufficiently large $t$. In contrast, FedEP requires approximating the covariance as the inverse Hessian at each round, which introduces an additional $\mathcal{O}(d^3)$ time complexity and $\mathcal{O}(d^2)$ memory complexity.

ii) Total communication rounds. In fact, convergence analysis for the federated variational approximation is still an open question, with no established results at present. So we try to give an intuitive explanation on why VA may converge slower than MCMC in FL. While VA may outperform sampling-based methods in terms of the convergence in centralized learning or real-world cases, the situation is different in FL. The main drawback of VA is that it typically results in biased posterior estimates for complex posterior distributions. This issue is further exacerbated in FL, where data heterogeneity across clients already contributes to biased local posteriors, making the use of VA even more challenging. Consequently, using VA in this context can result in an unstable and slow convergence of the aggregated global posterior.