Covariances for Free: Exploiting Mean Distributions for Federated Learning with Pre-trained Models

We thank the reviewer for their kind works about our work. We greatly appreciate your recognition that our contribution represents a valuable step for training-free federated learning. We are also grateful for your acknowledgment of the considerable effort we put into our research, including the extensive experiments validating the effectiveness of our proposed method. Below, we address the questions and comments you raised:

The algorithm necessitates the transmission of n_{k,c} to the server, which introduces certain privacy concerns. Although other methods also require this information, it would be beneficial if the authors could discuss potential techniques to address or mitigate this issue.

We agree with the reviewer that sharing the class statistics introduces certain privacy concerns. Following this suggestion (and the concern raised by the Reviewer jBiQ) we decided to investigate methodologies to mitigate these privacy concerns. We propose perturbing class-wise statistics with different types and intensities of noise before transmitting them to the global server and evaluate the performance of FedCOF. Specifically, we perturb the class-wise statistics as follows: $ñ_{k,c}= \max(n_{k,c} + \sigma^{\text{noise}}_{\epsilon},0).$

Here $\sigma^{\text{noise}}_{\epsilon}$ represents noise with intensity parametrized by $\epsilon$. The $\max$ operator ensure non-negative values in client statistics.

We vary the intensity of $\epsilon$ and the type of noise applied, including Uniform, Gaussian and Laplacian noise. Our findings demonstrate that the performance of FedCOF is robust to the noise types with varying intensities. These results suggest that perturbing statistics can mitigate privacy concerns stemming from the exposure of client class-wise frequencies. We added this empirical analysis in the Supplementary Material (Appendix L).

As modern pre-trained models tend to be generative models (e.g., GPT), it would be interesting to explore the possibility of extending the proposed methods to handle generative models by initializing the decoding heads accordingly.

We thank the reviewer for raising this interesting question. In principle, we think that initializing the linear layer that maps tokens to logits in autoregressive generative models would be possible. However, it is not clear whether the evaluated training-free appraoches proposed for federated learning including FedCOF would generalize to generative, autogressive settings. We think this could be an interesting direction for a future work.

We greatly appreciate the acknowledgment that we address a relevant and interesting problem, and we are pleased by the recognition that our proposed unbiased estimator for estimating class covariances in federated learning is sound, novel, and capable of achieving substantial improvement over existing training-free methods. Below, we address specific concerns raised by the reviewer.

The impact of the iid assumption on realistic scenarios is evaluated empirically. It would be great to quantify how heterogeneous distributions impact the estimator theoretically, e.g., under the assumptions that local distributions are Gaussians with different mean or covariance.

Many thanks to the reviewer for this insightful question. To theoretically quantify how heterogenous distributions impact our estimator $\hat{\Sigma}$, we derive a general bias formula independent of the i.i.d assumption. By treating each client as a random sample from distinct population distributions with mean $\mu_k$ and covariance $\Sigma_k$, we repeat the calculus for proving Proposition 2 in Appendix C. After some numerical steps, the general bias formula we derive is: $\text{Bias}(\hat{\Sigma}) = \text{E}[\hat{\Sigma}] - \Sigma = \frac{1}{K-1} \sum_{k=1}^K (\Sigma_k - \Sigma) + \frac{1}{K-1}\left(\sum_{k=1}^K n_k (\mu_k -\mu )(\mu_k -\mu)^\top \right),$ where $n_k$ is the number of samples assigned to a client , $\mu$ and $\Sigma$ are the global mean and covariance of all the features independently by the client assignement. Here as previously done, we are focusing on a single class.

If each client population covariance $\Sigma_k$ equals to the global covariance $\Sigma$, and the client mean $\mu_k$ matches the global mean $\mu$, the bias is zero, making the estimator unbiased. However, the bias formula shows that if a class distribution within a client differs from the global distribution of the same class, the estimator introduces a systematic bias. This situation can arise in the feature-shift setting, in which each client is characterized by a different domain. Quantifying this bias can open future directions for designing estimators that account for highly heterogenous distribution. In the Supplementary Material (Appendix J), we provide the mathematical derivation to arrive at this general formula for the bias of our estimator and this discussion.

The paper focuses on label shifts, i.e., a heterogeneous distribution of classes. It is unclear how the method performs in case of feature shift, i.e., a heterogeneous distribution of features, e.g., via locally different covariance structures [1].

Now, we empirically evaluate the performance of FedCOF in the feature-shift setting to understand how much the bias affect the performance in this setting. For this setting we use the DomainNet dataset, with six different domains. Following the work [1] suggested by the reviewer, we consider six clients where each client has i.i.d. data from one of the six domains.

Fed3R achieves better overall performance then FedCOF, likely due to its use of real class covariance, avoiding the bias addition that FedCOF introduces. However, FedCOF achieves comparable results while significantly reducing communication costs. FedNCM perform worse than FedCOF, at the same communication budget. When we increase the number of means sampled from each client, the performance of our approach improves. This is due to the fact that our method suffer with low number of clients (only 6 in this experiments) and sampling multiple means helps, as mentioned in the main paper. In Supplementary Material (Appendix K), we add these experiments and discussion.

The dimension of the feature space could impact the accuracy of the estimator.

We already analyzed how different feature space dimensionalities can affect performance (512 for SqueezeNet and ResNet18, 1280 for MobileNetV2 and 768 for ViT-B/16) but these are also dependent on the quality of the features and thus depends on the network architecture. We agree that very high-dimensional features can affect estimator accuracy due to low-rank, high-dimensional covariance estimates from limited samples (L273-284). We mitigate this issue by using covariance shrinkage regularization which adds a multiple of an identity matrix to the covariance estimate. Hoewever, we have not yet found a way to construct a representative example with gaussians to illustrate the impact of the dimensionality. We will expand the discussion on high dimensional synthetic features and plan to provide an example with synthetic dataset in the final version.

For consistency I suggest to use $\widehat{\Sigma}$ instead of $\widehat{S}$ in Eq. 10.

In the final version of the paper, we will fix this.

Why is this approach better in terms of accuracy that Fed3R?

We would like to highlight that the proposed method FedCOF has a different classifier compared to Fed3R which uses a ridge regression classifier. We discussed this in section 4.3. The classifier initialization of Fed3R uses $G$ obtained from Equation 10, which considers both the within- and between- class scatter matrices. We propose a different classifier initialization using Equation 11, which uses only within-class scatter matrices.

We empirically motivate this by analyzing the impact of within class scatter matrices in Figure 4. Using a centralized setting, we showed that classification performance can be improved by removing the between-class covariances from Equation 10. So, in FedCOF we initialize the classifier using $G$ from Equation 11 which uses only the estimated within-class covariances and thus performs better than Fed3R in most settings. Thus, the improvement in performance of FedCOF compared to Fed3R is due to the different classifier initialization.

Regarding the covariance approximation, FedCOF-Oracle uses true class covariances while FedCOF uses the proposed estimator using client means. While both employ the same formula for classifier initialization (equation 11), FedCOF-Oracle generally outperforms FedCOF because the rank of our estimated covariance is limited by the number of clients per class (see Section 4.2 "Impact of the Number of Clients"), resulting in a lower-rank approximation compared to the true class covariance.

Strong pre-trained feature extractor with a classical end-to-end federated learning baseline.

We perform experiments with ResNet-18 instead of ResNet50 due to limited time and compute resources and show how the classical federated learning approaches perform. We observe significant improvement on using FedCOF and finetuning with FedAdam compared to simply using FedAdam. We discuss this in details in Appendix H (Experiments with ResNe18, see Table 4). We also had the comparison with FedAdam in Figure 6 of our paper on three datasets using SqueezeNet architecture where we show that FedCOF+FedAdam outperforms FedAdam.

Please compare your approach also to distributed training of linear models (using standard FedAvg).

We now compare with our approach with the training-based federated linear probing (where we perform FedAvg and learn only the classifier weights of models) and show in table below that FedCOF is more robust and communication-efficient compared to federated linear probing across several datasets. We discuss this in details in Appendix H (Comparison of training-free methods with linear probing, Table 3).

Since ridge regression might not always be ideal approach given a fixed feature extractor, I wonder whether a kernel ridge regression could be applicable.

We agree that using a fixed feature extractor and employing ridge regression may not be optimal since the features might not be well separated. Kernel ridge regression could address this by implcitly mapping features to a higher dimensional space and thus improving separability. However, as noted this would increase communication costs quadratically with the number of data points and employing the Nystrom method, could mitigate these additional communication costs and we will certainly consider this possibility for future work.

Currently we see the following challenges:

1. Selecting kernel hyperparameters (e.g $\sigma$ for the kernel RBF), is dataset dependent, determining a fixed $\sigma$ across diverse dataset is very challenging.
2. High-dimensional feature spaces in Kernel Ridge Regression amplify the need for careful shrinkage tuning to stabilize smaller eigenvalues, as mentioned in our paper. An automatic shrinkage estimation technique may help in this scenario.
3. The Nystrom method requires selecting $m$ samples from each client, where $m<n$, with $n$ being the total number of data points. How to choose an appropriate $m$ and determining which $m$ samples to select is not obvious in a highly heterogeneous federated setting.

We thank the reviewer for appreciating the writing quality of our manuscript, the clarity on our theoretical derivation, and for recognizing that our approach demonstrates decent empirical performance for both classifier initialization and federated fine-tuning. Below we reply to the specific comments made.

The steps of Fed-COF + Fine-tuning can be made clearer. The description around line 357 is difficult to follow.

Regarding FedCOF+Fine-tuning, we have more discussions in section 5.2 in the paper. In L357, we discuss how FedCOF classifier initialization can be used in multiple rounds before any finetuning starts. We consider the realistic setting when all the clients are not available at the same time and only a fraction of clients comes in each round. While FedNCM assumes the availability of all clients in one round for classifier initialization, we follow the multi-step approach proposed by Fed3R. We now clarify and update the discussion on FedCOF multiple rounds (L357-367).

The choice of ridge regularization parameter seems important for the classification performance. Can authors give more empirical suggestions on how lambda should change with different numbers of clients/means per client?

Following Fed3R, we use the ridge regularization parameter to ensure that the matrix $G$ is invertible. This is only for numerical stability purposes. We performed an ablation to measure the impact of $\lambda$ and observe that the performance does not vary much (we notice a deviation of 0.1 on CIFAR100 and 0.25 on CUB200 by varying $\lambda$ from 0.001 to 1). In cases when the matrix $G$ is low-rank, such as in Fed-3R, the $\lambda$ parameter is helpful since it makes $G$ invertible. However, in our method we are not so dependent on $\lambda$ since we already use covariance shrinkage to obtain full-rank estimates of the covariance matrix (see equation 8) and as a consequence, we obtain an invertible matrix $G$ (see Equation 11).

What is the difference between Fed-COF oracle and Fed3R? In Table 2, it is surprising to see Fed3R sometimes achieves lower accuracy with more communication. The authors should provide more explanations for the phenomenon

We would like to highlight that the proposed method FedCOF has a different classifier compared to Fed3R which uses a ridge regression classifier. We discussed this in section 4.3. The classifier initialization of Fed3R uses $G$ obtained from Equation 10, which considers both the within- and between- class scatter matrices. We propose a different classifier initialization using Equation 11, which uses only within-class scatter matrices.

We empirically motivate this by analzing the impact of within class scatter matrices in Figure 4. Using a centralized setting, we showed that classification performance can be improved by removing the between-class covariances from Equation 10. So, in FedCOF we initialize the classifier using $G$ from Equation 11 which uses only the estimated within-class covariances and thus performs better than Fed3R in most settings. The FedCOF oracle uses the same classifier initialization as FedCOF but using the real covariances shared by clients. Thus, the difference between FedCOF-oracle and Fed3R is due to the different classifier initialization.

We thank the Reviewer for their feedback and are encouraged that, despite the extremely low overall score, the Reviewer acknowledges that our approach makes a timely contribution to federated learning with pre-trained model, achieving strong perfomance with minimal additional overhead when compared to the state-of-the-art competitors FedNCM and Fed3R. However, we are surprised by the strong reject recommendation, as it does not seem to fully align with the comments and concerns raised in the review. We are confident that we can adequately address the Reviewer’s concerns in this rebuttal. Below, we provide detailed responses to the issues raised.

Related Works - the use of fixed classifiers.

We thank the reviewer for suggesting these related papers. While we already discussed [3] in our related work, we have now added the discussion on the impact of freezing classifiers in federated settings with appropriate references [1,2,4] in the updated version of our paper (see L097-100)

Pretrained models are not always advantageous in federated settings.

In the original manuscript, we discussed several recent published works on federated learning with pre-trained models (Nguyen et al., 2023; Tan et al., 2022b; Chen et al., 2022; Qu et al., 2022; Shysheya et al., 2022; Legate et al., 2023a; Fanı̀ et al., 2024). All of these works show that using pre-trained models significantly benefit federated learning in highly non-iid settings using different federated optimization methods across several datasets (CIFAR-10, CIFAR-100, Stack Overflow, FEMNIST, Reddit, Flowers, CUB, Stanford cars, EuroSAT-Sub, iNaturalist-Users-120K). Our findings further substantiate these observations (see our comparison below with random initialization). As suggested by the Reviewer, we now refer to findings from FedFN [5] which show that in some settings using a pre-trained ResNet-18 model on the CIFAR-10 dataset negatively impacts the learning of the global model. We have added this discussion in L105-107.

Preliminaries:

We have clarified the following in the revised manuscript:

• L117: $D_k$ refers to the local dataset.
• L127: We clarified the loss function. Here, we do not provide details about how the loss is calculated because this is a general federated learning framework which can use different loss functions and employ different ways of computing the loss.
• L129-130: We now clarify in L131 that the proposed method do not involve any training or local model updates.

Privacy concerns on sending classwise statistics: The algorithm sends the class-wise frequency of the data held by clients to the central server. In fact, there are many previous FL papers that have communicated class frequency information and provided justifications. Citing these studies would strengthen the discussion, but this type of content is entirely missing.

We thank the reviewer for pointing out this. Our approach, like other methods we cited (e.g FedNCM (Legate et al., 2023) and CCVR (Luo et al. 2021)), requires transmitting class-wise frequencies from clients to global server. We agree with the reviewer that this could raise privacy concerns, as sharing class-wise statistics may expose client class distribution. We have updated the paper to explicitly discuss this issue.

Motivated by the Reviewer suggestion (and the suggestion of the Reviewer 6242), we decided to investigate methodologies to mitigate these privacy concerns. We propose perturbing class-wise statistics with different types and intensities of noise before transmitting them to the global server and evaluate the performance of FedCOF. Specifically, we perturb the class-wise statistics as follows: $ñ_{k,c}= \max(n_{k,c} + \sigma^{\text{noise}}_{\epsilon},0).$

Here $\sigma^{\text{noise}}_{\epsilon}$ represents noise with intensity parametrized by $\epsilon$. The $\max$ operator ensure non-negative values in client statistics.

We vary the intensity of $\epsilon$ and the type of noise applied. We consider Uniform, Gaussian and Laplacian noise. Our findings demonstrate that the performance of FedCOF is robust to the noise type with varying intensities. These results suggest that perturbing statistics can mitigate privacy concerns stemming from the exposure of client class-wise frequencies. We added this empirical analysis in the Supplementary Material (Appendix L).

Concern on Incremental Contribution: The problem may seem incremental, as it combines existing methods' strengths, but it addresses the practical challenge of balancing communication cost and performance in FL.

We thank the Reviewer for recognizing that we address the practical challenge of balancing communication cost and performance in FL. However, we respectfully disagree with the assertion that our proposed approach may appear incremental. In this work we introduce several novel contributions to training-free approaches for Federated Learning with pre-trained models. Firstly, we propose an unbiased estimator for class covariances (Proposition 2) that requires only client means, and we mathematically prove this result in the Supplementary Material (Appendix C). Secondly, we establish a connection between the Ridge Regression solution and class feature covariances (Proposition 3), and we again mathematically prove this result in the Supplementary Material (Appendix D). The only aspect that may appear incremental is our use of class covariances to initialize a Ridge Regression classifier. However, even in this case, we propose a different classifier initialization than Fed3R by demonstrating that using between-class scatter matrices decrease performance of the standard Ridge Regression classifier (see Equation 11). On the basis of these empirical observations, we remove these relationships for classifier initialization and demonstrate improved performance over Fed3R.

These contributions are not incremental but are grounded in a careful analysis of existing literature. We provide a novel, mathematically sound, and practically effective solution to address the challenge of training-free methods for pre-trained models in Federated Learning.

It would be beneficial for the authors to include experimental comparisons between using a pretrained model and a randomly initialized model.

To clarify the impact of using pre-trained models, we conducted additional experiments using a randomly initialized model. Specifically, we conduct these experiments employing SqueezeNet on CIFAR-10 and CIFAR-100. These experiments were conducted in a highly heterogeneous setting in which client data was assigned using a Dirichlet distribution with $\alpha=0.1$, following standard practice. We discuss these details in Appendix H (Impact of using pre-trained models, Table 5) in the revised version of paper. For convenience we report these results in the following table:

Our results demonstrate that federated training with a pre-trained SqueezeNet model significantly outperforms a randomly initialized model when using standard methods on CIFAR-10 and CIFAR-100.

If the characteristics of the training data used for the pretrained model(e.g. ImageNet) are significantly different from the test data(e.g. SVHN) targeted by the global model, using a pretrained model could potentially be detrimental.

The assumption that the pre-training data is relevant to the target dataset is a limitation of all existing works using pre-trained models in federated learning (Nguyen et al., 2023; Tan et al., 2022b; Chen et al., 2022; Qu et al., 2022; Shysheya et al., 2022; Legate et al., 2023a; Fanı̀ et al., 2024) and across other domains. Similar to most existing works, we use weights pre-trained on ImageNet-1k. We also mention in the limitations section of our paper (L537-539), “our method assumes the existence of a pre-trained network. If the domain shift with the client data is sufficiently large, this is expected to impact the performance.”

The proposed algorithm sends class frequency information from each client, but in the case of missing classes, this would simply convey a value of 0. Could the authors explain how their proposed algorithm is designed to mitigate this vulnerability in the context of FL, and why it might still perform well despite the challenges posed by missing classes?

All of our experiments have missing classes at clients which means that each client contains only a subset of the total classes. This is the nature of federated learning with highly heterogeneous distributions following standard practice of using dirichlet distribution with $\alpha=0.1$. Our proposed method FedCOF requires sharing of class means and class counts from each client only for those classes which are present in the respective clients. For missing classes at each client, we do not send any mean or class count. At the server side, we use means and counts for a particular class $c$ only from those clients which contain class $c$. Thus, our method is not affected by the missing classes phenomenon and is independent of how many classes are there in each client.

Dear Authors,

Thank you for your detailed response to the comments. However, it seems that the concern I raised below has not been fully addressed. I would appreciate further clarification on this matter.

The reason I raised a question regarding the problem setting is that I felt the rationale for introducing a pretrained model into Federated Learning was not clear.

As I mentioned earlier, my question is whether using a pretrained model is truly better than random initialization in arbitrary scenarios (arbitrary domain gap, arbitrary heterogeneity).

I believe this is important because, in actual Federated Learning, client datasets are not publicly available. Therefore, in such arbitrary situations, using a pretrained model should not be worse than using a randomly initialized model.

From what I understand, the study demonstrates superior performance in situations with domain gaps compared to other algorithms, but I don’t believe it provides sufficient justification for using pretrained models comparing to randomly initialized model in such arbitrary situations.

Dear Authors,

Thank you for your detailed response to the comments. First of all, I have increased the score to 3, as you have addressed some of my concerns. Based on the points you mentioned, I kindly ask you to finalize the draft, as I need to evaluate the version that has been reflected so far, rather than future versions.

I do have one question regarding the methodology. In this study, the main tables were all based on experiments using pure class frequencies. Since I believe this is not the first Federated Learning study to utilize pure class frequencies, I would appreciate further clarification on this matter. Specifically, in the text, the justification for using feature prototypes is explained as: "Sharing only class means provides a higher level of data privacy compared to sharing raw data, as prototypes represent the mean of feature representations. It is not easy to reconstruct exact images from prototypes with feature inversion attacks, as shown by (Luo et al., 2021)." Given this, is there a comparable justification for using pure class frequencies in the context of Federated Learning?

We thank all reviewers for their valuable feedback and for actively engaging in the discussions. We believe that we have addressed all the concerns of reviewers Wumb, z1Dd, 6242 and thank them for their positive ratings of our work. We are very thankful to reviewers z1Dd, 6242 for acknowledging our efforts in the rebuttal phase.

We believe we have thoroughly addressed all concerns raised by reviewer jBiQ -- including related work, preliminaries, privacy discussions, missing classes, random initialization, and novelty clarification -- and have updated our paper as a result. Unfortunately the reviewer still recommmends a 'reject' rating, referring to 'lingering doubts' on the use of pre-trained models and incremental novelty concerns. To summarize our position on these remaining issues:

1 - Problem justification (why pretrained models should be applied to Federated Learning): We first of all underscore that the focus of our contribution is on training-free federated learning, which is a scenario in which the benefits of starting from pre-trained models is evident and firmly established by recent prior works on training-free federated learning [a, b]. Nonetheless -- and contrary to reviewer claims -- our main paper contains ample discussion and references on "justification for introducing pretrained models" in L012-013, L044-047, L100-106.

The reviewer's other concern is on including the random initialization results in the main table of the paper. We have already conducted experiments comparing pretrained versus random initializations in Table 5 (Appendix H) and also in the last response after the pdf update deadline. Since this is not the main focus of our work and the performance using random initialization is very bad even after several rounds of training (as also established by FedNCM [a]), we have therefore decided that we will keep these results in the Appendix in any future version.

The reviewer also would like us to investigate whether pre-trained models are useful in "arbitrary scenarios (arbitrary domain gap, arbitrary heterogeneity)" but does not provide any specific references. From the literature we know that pre-trained features can be effective even with large domain gaps: pretrained ImageNet features have been applied to several fine-grained datasets [c], medical datasets [d], remote sensing [e], to name just a few. Our experimental evaluation already considers datasets with significant domain gaps like ImageNet-R and heterogeneity -- as we have pointed out in our rebuttal.

2 - Novelty Concerns: While other reviewers appreciated the novelty of our work and despite our clarification of novelty twice in the discussion phase, the reviewer did not engage with our clarifications, and has at no point during the review and rebuttal process engaged with the technical content and main contributions of our work.

[a] Legate et al., Guiding the last layer in federated learning with pre-trained models. In Advances in Neural Information Processing Systems, 2023.

[b] Eros Fani, Raffaello Camoriano, Barbara Caputo, and Marco Ciccone. Accelerating heterogeneous federated learning with closed-form classifiers. Proceedings of the International Conference on Machine Learning, 2024.

[c] Kornblith, Simon, et al. "Do better imagenet models transfer better?." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[d] Dack, Ethan, et al. "An empirical analysis for zero-shot multi-label classification on covid-19 ct scans and uncurated reports." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[e] Corley, Isaac, et al. "Revisiting pre-trained remote sensing model benchmarks: resizing and normalization matters." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.