Covariances for Free: Exploiting Mean Distributions for Federated Learning with Pre-trained Models

We thank the reviewer for appreciating that the paper is well structured and clearly written, that claims are well supported by extensive experimental results and analyses, that the covariance estimator is mathematically well justified and conceptually solid, and that the experimental settings are well designed and valid. Below we reply to the specific points raised by the reviewer.

While the performance improvement is promising, there is no clear justification for removing between-class scatter in FL.

The reviewer is right, we made the choice to remove between-class scatter primarily based on empirical results in the centralized setup because the Ridge Regression solution is equivalent in the centralized and federated learning setups (as discussed also in Fed3R). While we propose this based on empirical results, we provide more justification and analysis on this below.

Intuitively, to represent the feature distribution of each class we do not really need to consider the relationships between different classes which represent the distribution of the overall dataset since our goal is to estimate the class-specific classifier weights using these covariances. Based on this, we propose to remove the between-class scatter and initialize the classifier weights using only the respective within-class scatter matrix.

Let us denote the between- and within-class scatter matrices as $G_{\text{btw}}$ and $G_{\text{with}}$, respectively. We analyze the spectral properties of these scatter matrices, with a focus on their conditioning. By defining the condition number for a matrix $G$ as $k(G) = \lambda_{\text{max}}(G) / \lambda_{\text{min}}^{+}(G)$, where $\lambda_\text{max}$ is the largest eigenvalue and $\lambda_\text{min}^{+}(G)$ is the smallest non-zero eigenvalue, we observe empirically for SqueezeNet in the centralized setting that:

• $G_{\text{btw}}$ is severly ill-conditioned with condition numbers $k(G_{\text{btw}})$: $2.97\times10^7$, $2.46\times 10^7$,$2.2 \times 10^7$, $1.27\times10^7$ on CUB, CARS, ImageNet-R, CIFAR-100, respectively.
• $G_{\text{with}}$ is much better conditioned with condition numbers $k(G_{\text{with}})$: $4.5\times10^3$, $2.48\times10^4$, $8.19 \times 10^2$, $6.3 \times 10^3$ on CUB, CARS, ImageNet-R, CIFAR-100.

Including $G_{\text{btw}}$, can cause numerical instability because it is poorly conditioned. This leads the classifier to overfit to directions with very small eigenvalues, which may capture noise or dataset-specific artifacts, resulting in poor generalization on unseen samples.

We also analyze in the centralized setting how the different scatter matrices affect overfitting of the model. We fix the parameter $\lambda=0.01$, which yields optimal performance for both classifiers.

We observe that, while both methods achieve similar high training accuracies, Ridge Regression consistently underperforms on the test set. This suggests that incorporating $G_{\text{btw}}$ introduces overfitting, as the classifier learns directions that do not transfer well to unseen samples.

Finally, we analyze the impact of removing the $G_{\text{btw}}$ in a FL setup using SqueezeNet below:

In the FL setup, we again clearly see the negative effect of incorporating between-class scatter statistics. We will add all the analysis and discussion in the revised version of the paper.

The reliability of estimator heavily relies on the number of clients.

We acknowledge this in the Limitations section. The quality of our estimator depends on number of clients, as shown in Fig. 5 where using multiple class means per client helps in fewer client settings. In Appendix L (Fig. 8) we explain this in more details.

It appears that the number of hyperparmaters ($\gamma$ and $\lambda$) is reduced to one, as the covariance shrinkage can be absorbed into $\lambda I_d$ in equation (11).

The reviewer is correct, the two hyperparameters have similar purposes and can be absorbed into one. While we used $\lambda$ to maintain the same formulation as the ridge regression classifier, our method does not require this hyperparameter. Varying the $\lambda$ parameter, we do not observe any change in the performance of FedCOF. Thus, the $\lambda$ hyperparameter can be removed since the shrinkage hyperparameter already serves the same purpose.

We thank the reviewer for acknowledging the soundness of our work, the correctness of our proofs, and that the evaluation criteria is appropriate for FedL - accuracy, communication cost, FL rounds, and client participation. We also appreciate the recognition of our diverse experimental setup across multiple datasets, which reflect real-world scenarios and enhances the relevance of our work. Finally, we thank the reviewer for highlighting the quality of our estimator and the improved performance and communication efficiency over existing training-free methods. Below we reply to specific points raised by the reviewer.

The covariance estimator is unbiased under the iid assumption. Proposition 2 and its proof in Appendix C provide a mathematical foundation, showing that the expectation of the estimator equals the population covariance. However, the evidence weakens when the iid assumption is relaxed (Appendix E), where bias is acknowledged but not quantified beyond theoretical derivation, leaving practical implications partially unsupported. Proofs look correct, but the reliance on iid assumptions limits their generalizability, as acknowledged in Appendix E’s bias analysis for non-iid settings. Can you quantify the bias’s practical impact in non-iid scenarios beyond DomainNet?

In our paper we theoretically analyze the bias of our covariance estimator with non-iid client features (see Appendix E) and perform empirical evaluation on the DomainNet feature shift setting (see Table 5 in Appendix F) to study the impact of the bias.

Based on the Reviewer's suggestion to evaluate the methods in a feature shift setting beyond DomainNet, we perform experiments on the Office-Caltech10 dataset using MobileNetv2 as done in [X]. This dataset contains real-world images obtained from different cameras or environments and has 4 domains (clients) covering 10 classes. Our results are given in the following table:

In this feature-shift benchmark, we see that all the training-free methods perform similarly. Despite the theoretical bias in our estimator, FedCOF performs similar to Fed3R. We argue that this outcome is due to the generalization performance of pre-trained models. While non-iid feature shift settings have been studied in some papers not using pre-trained models, using this setting with pre-trained models works a bit differently. When using a pre-trained model, the generalization capabilities of the pre-trained model can help in moving the distribution of class features across clients towards an iid feature distribution even if the class distribution across clients is non-iid at the image level.

We observe a dip in performance for FedCOF compared to Fed3R in our experiments on DomainNet (see Table 5 in Appendix F) indicating the effect of the bias in our estimator on that particular benchmark. We believe that more comprehensive analysis of feature-shift settings when using pre-trained models requires more extensive benchmarks and could be an interesting direction to explore in future work. We will add the experiments on Office-caltech10 and this discussion in the revised version.

[X] Li et al., Fedbn: Federated learning on non-iid features via local batch normalization. In ICLR, 2021.

Justification of using only within class covariance terms are supported only by numerical results (ablative study in Table 2), and this methods underperforms Fed3R in the presence of feature shift. More extensive analysis and justification with theoretical evidence is required for the proposed method.

We discuss this in response to reviewer FJXY. We kindly ask the reviewer to refer to that discussion. We will incorporate this discussion and motivation of excluding between-class covariances in the paper.

---

Lacks convergence analysis for fine-tuning, weakening claims of improved convergence.

Our claims of improved convergence using the proposed FedCOF initialization is based on our empirical results (see Fig. 3) across multiple datasets. We propose how to better initialize the classifier before performing federated optimization methods like FedAvg and FedAdam which have already established the theoretical guarantees of convergence in their respective works. We discuss this in detail in the Appendix M.

---

How gamma and lambda are chosen for each benchmark?

Fed3R proposed to use $\lambda$ for numerical stability and we use the same value as Fed3R ($\lambda=0.01$) in all our experiments.

Regarding selection of shrinkage hyper-parameter $\gamma$, we do not optimize it for each benchmark and use $\gamma=1$ for all datasets when using SqueezeNet and ViT-B/16 networks. When using MobileNetv2, we use $\gamma=0.1$ due to its very high feature dimensionality (d=1280). We selected hyperparameters on one dataset (ImageNet-R) and used that value for all others, which works well. We discuss the impact of $\gamma$ in Appendix L (see Table 9).

We thank the reviewer for appreciating the clarity and readability of our paper, as well as the good experimental results that validate our mathematical derivations. Below we reply to each of the points raised by the reviewer.

The work of Luo et. al. [1] was mentioned briefly in the Introduction as a previous work that used the same techniques: class means and covariances from all clients for classifier calibration. However, the author did not include this work (namely, CCVR) as a baseline. Which, in turn, weakens the novelty of the paper.

We thank the reviewer for the suggestion to include CCVR in our comparison. Since CCVR was originally proposed for calibrating classifiers after training, we adapt it to our setting and use it as an initialization method. CCVR trains a linear classifier on features sampled from aggregated class distributions from clients. We show that the proposed FedCOF outperforms CCVR in most settings despite having significantly lower communication cost (in MB):

Note that CCVR does not require federated training since it trains a linear global classifier unlike closed form, training-free approaches.

We also show that FedCOF initialization is better for further finetuning using a pre-trained SqueezeNet that we finetune with FedAdam for 100 rounds after different initialization methods:

We will add the CCVR baseline in the main experiment tables of our paper in the revised version.

---

The paper is clearly written and easy to follow. The demonstration figure, however, is not. I would recommend replacing this figure with a more comprehensive one in case this work is accepted for publication.

We thank the reviewer for the suggestion and will improve the figure with a more comprehensive one in the revised version of the paper.

We thank the reviewer for acknowledging that our claims are justified both theoretically and through numerical experiments, for recognizing the correctness of our theoretical results, the soundness of our experimental design and analyses, the comprehensiveness of our evaluation, and for appreciating that we address the interesting and timely topic of training-free methods, providing an interesting contribution. Below, we respond to each point raised by the reviewer.

Most of the theoretical analyses in this paper are rather standard and straightforward. The theoretical contributions of this paper are marginal.

While our analysis builds on established mathematical tools, it provides non-trivial theoretical insights crucial for the effectiveness of our method and relevant to general-purpose Federated Learning.

Specifically, we rigorously derive an unbiased estimator of class covariances using only first-order statistics (Prop. 2), enabling the estimation of second order statistics without sharing them. This approach avoids higher communication costs and mitigates privacy concerns. We believe this novel theoretical result is impactful for training-free method initialization, as demonstrated by our results. Moreover, it can be used in broader federated learning contexts where second-order statistics are needed but costly to transfer since communication costs can be bottleneck for real-world FL systems.

Additionally, in Prop. 3 we perform novel derivation of the Ridge Regression solution that, when combined with our estimator, avoids the need to share large matrices, thereby significantly reducing communication costs. We emphasize that this derivation is not straightforward, as also acknowledged by Reviewer diuQ, and, to the best of our knowledge, we are the first to provide it.

Although the authors provided theoretical bias analysis and conducted empirical studies, it remains unclear theoretically how non-i.i.d. data could affect the proposed FedCOF.

We acknowledge that our covariance estimator for learning-free FL relies on standard i.i.d assumption across clients -- a common assumption in most of FL literature -- and we explicitly state this limitation in our paper. In Appendix E, we provide a detailed derivation of the bias introduced when this assumption is violated: $\text{Bias}(\hat{\Sigma}) = \mathbb{E}[\hat{\Sigma}] - \Sigma = \frac{1}{K-1} \sum_{k=1}^K (\Sigma_k - \Sigma) + \frac{1}{K-1}\left(\sum_{k=1}^K n_k (\mu_k -\mu )(\mu_k -\mu)^\top \right).$

This derivation quantifies the bias that arises when the distribution of a class within a client differs from the global distribution of the same class. In Appendix F we empirically demonstrate the impact of this bias in feature-shift setting using DomainNet.

While we have focused our theoretical analysis on quantifying the bias, our extensive empirical evaluation on large-scale real-world non i.i.d benchmark -- such as iNaturalist-120k, having 1203 classes across 9275 clients, 120k training images of natural species collected around the world -- further demonstrates that our method is robust (see Table 3 and Fig. 4). We will consider exploring further theoretical implications of non-i.i.d data in future work.

The comparisons between the proposed FedCOF and FedAvg may not be fair, since one is based on pre-trained models and the other is based on training from scratch.

We would like to clarify that all FedAvg and FedAdam experiments in the main paper (See Figs. 3-4 and Table 4) use pretrained backbones and are not trained from scratch, thus ensuring fair comparison with FedCOF. We have more experiments in the Appendix (see Table 8) in which we compare FL methods with and without pre-trained backbone initialization.

While learning-free FL methods are interesting particularly with pre-trained models, it appears that their use cases are rather limited (e.g., linear classification problems). Could the authors illustrate more relevant use cases for the proposed learning-free FL method?

We would like to highlight that learning-free FL methods use cases are not limited to linear classification problems. Their purpose is to exploit the feature distribution given by a pre-trained model to initialize a classifier. Training-free classifier initialization can be followed by full federated finetuning to solve non-linear classification tasks (see Fig. 3). We demonstrate that this approach (FedCOF+FedAdam, Fig. 3) can improve performance at a much lower costs compared to federated full finetuning, with pre-trained backbone, and a classifier randomly initialized (FedAvg, FedAdam, Fig. 3).

Finally, training-free methods could be used in other tasks like object detection or semantic segmentation. As an example, object detection networks like Faster-RCNN use a classification head and prototype-based closed form approaches could be adapted to those network heads.

We thank the reviewer for acknowledging the soundness of our propositions, the positive improvements in performance and communication cost over other training-free and training-based solutions, and the quality of our experimental design. We appreciate that the reviewer acknowledged that Proposition 3 is a clever rewrite of the ridge regression solution, which neatly avoids the need to share large matrices. Below we address the specific points raised by the reviewer.

Federated full-finetuning was not considered due to the expensive cost.

We would like to clarify that our paper considers federated full-finetuning for comparison with FedCOF (See Fig. 3 and Table 4). We consider FedAvg and FedAdam as the federated full-finetuning baselines starting from a pretrained model initialization. All methods denoted as FedAvg and FedAdam in the main paper perform federated training from pretrained model initialization in which only the classifier is randomly initialized without using training-free initialization methods. It is important to note that these models are not trained from scratch. We have some experiments in Appendix K (see Table 8) in which we compare federated training methods with and without pretrained backbone initialization.

I strongly believe this paper should compare against parameter efficient federated fine-tuning approaches. The paper is missing a body of literature on federated parameter efficient fine-tuning, which should be a more appropriate competitor than classification head fine-tuning.

We thank the reviewer for suggesting recent works on parameter-efficient federated fine-tuning. We will discuss these papers in the revised version. For comparison with these methods, we consider the recent work Probabilistic Federated Prompt-Tuning (PFPT) from NeurIPS 2024, as suggested by the reviewer. We compare PFPT, FedAvg-PT, and FedProx-PT (prompt-tuning variants of FedAvg and FedProx) with training-free initialization approaches, and also perform PFPT with training-free classifier initialization.

We use a pre-trained ViT-B32 on CIFAR-100 and TinyImageNet with a Dirichlet distribution ($\alpha=0.1$) following PFPT. We use the same training hyperparameters as PFPT. The results are summarized in the following table:

Fed3R and the proposed FedCOF are competitive with the prompt-based training methods, without any training. Training-free methods require lower communication budget with respect to prompt-tuning methods. Finally, these results show that further finetuning of prompts after training-free initialization with Fed3R and FedCOF achieves state-of-the-art results on these benchmarks.

Ridge regression may not be the best choice for complex task. For example, looking at Fig. 3 of this paper, we can see that the accuracy continues to increase after the training-free stage of FedCOF/Fed3R/FedNCM, which should not be the case if ridge regression is reasonably good & we can compute its closed form solution.

In Fig 3. all methods perform federated full-finetuning starting from a pre-trained backbone with the classifier initialized randomly (FedAvg, FedAdam) or with a training-free method. This improves the quality of feature representations and thus the classification performance improves. Instead if we consider Fig. 4, where we do training-based federated linear-probing (keeping the backbone fixed), we see that the performance improves only marginally after the Fed3R and FedCOF suggesting that they are pretty good classifier. While linear probing marginally improves performance, it requires training (linear-probing) for several rounds and incurs larger communication and computational costs.

It would be interesting to see if FedAvg would eventually outperform both the training free performance and the training free + finetuning performance. Regardless, I still see the merit of this method as an inexpensive way to initialize the classifier, which could save 100 or more communication rounds in practice.

We compare FedAvg trained for 400 rounds with FedCOF+FedAdam trained for 100 rounds and observe that FedAvg which converges after 400 training rounds still does not outperform FedCOF+FedAdam:

In the revised version of the manuscript, we will include the plot of FedAvg for 400 rounds.