One-shot Federated Learning with Training-Free Client

• Observations on the backbone drift and head drift don’t seem to provide new insights. Training for a large number of epochs on local clients can lead to overfitting, and local models might diverge significantly from each other in a non-iid setting, as already demonstrated in [1-2].

  • It would have been better to show the similarity in feature representations across different layers.
  • It would be more convincing to choose froze if the proposed method FedTC performs worse with few local epochs (e.g., 10) for updating the backbone or subsequent layers of the backbone (not the $M_1$ case in the paper; include the prototype learning as well).

• Unclear experimental details and an unfair comparison.

  • In the ablation studies, it would be better to show how the proposed method perform without the prototypes part (simply aggregation).
  • Also, this paper chooses pre-trained weights from CLIP and pre-trained weights from ImageNet for different scale datasets. It would provide better clarity if the authors could show how the pre-trained weights would affect the performance of the proposed method, FedTC.
  • All methods for comparison are employed with the adapter, but for example, the DENSE method uploads local models to the server; in this case, the adapter might not be necessary and even could result in a performance drop.
  • Clarification is needed on whether the competing methods use identical pre-trained weights. Moreover, the DENSE method is a data-free methodology; how is this method implemented for comparison?

• Currently, experiments only consider local models that are homogeneous, while it might have different model architectures across clients in the real world [3]. It might be beneficial if the proposed method could further consider different local models with different pre-trained weights and apply an adapter to handle prototypes in different dimensions.

[1] Lin, Tao, et al. "Ensemble distillation for robust model fusion in federated learning." Advances in Neural Information Processing Systems 33 (2020): 2351-2363.

[2] Heinbaugh, Clare Elizabeth, Emilio Luz-Ricca, and Huajie Shao. "Data-Free One-Shot Federated Learning Under Very High Statistical Heterogeneity." The Eleventh International Conference on Learning Representations. 2022.

[3] Zhang, Jie, et al. "Dense: Data-free one-shot federated learning." Advances in Neural Information Processing Systems 35 (2022): 21414-21428.

• Lack of novelty.

  • The main idea of this paper shares a huge overlap with CCVR [1], but the author even didn't mention it. In CCVR, the authors retrain the classifier on the server using the class prototypes of clients' local data after federated learning. Ignoring the small tricks in the proposed method (e.g., filtering out feature outliers and adding an adapter), the only difference between CCVR and the proposed FedTC is that FedTC uses a pre-trained model to replace the federated training of the global model. Thus, I reckon the authors may have limited technical novelty contributions.
  • Actually, the most interesting insight of this paper is that fixing pre-trained backbones and adapting the classifiers are enough to boost the generalization in federated learning. The classifiers are also biased if federated learning, so the authors propose to learn the unbiased classifier via prototypes. It makes me wonder: how about initializing the prototypes as the classifiers? Since the prototypes are already expressive enough to distinguish the features into the classes' centroids. Therefore, the proposed method needs further justification.

• Lack of fair comparison.

  • Following my last point, the authors should include the most straightforward approaches into the experiment comparison to justify the method design. Also, FedAvg with only classifier training is also necessary to be a baseline, i.e., FedAvg-v in Figure 1.
  • I think it is a bit unfair for the baselines to train the backbones, and as a result, the baselines are extremely low. I think maybe a more fair comparison is to also fix the backbone and only train the classifier for the baselines (e.g., for DENSE).

• Discussion on current literature is inadequate.

  • Besides CCVR, I think the authors may miss some important related works that, at least, should be cited and discussed.
  • For the role of pre-trained models in federated learning: [2] [3].
  • For class prototypes and classifier biases in federated learning: [4] [5].

• Lack of necessary experiments.

  • Some experiments are needed. For example, experiments with more numbers of clients (also with partial participation) are needed to verify the scalability of the proposed method. It is clear that in the current version, the number of clients is only 10, which is on a small scale.

[1] Luo M, Chen F, Hu D, et al. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data[J]. Advances in Neural Information Processing Systems, 2021, 34: 5972-5984.

[2] Chen H Y, Tu C H, Li Z, et al. On the importance and applicability of pre-training for federated learning[C]//The Eleventh International Conference on Learning Representations. 2023.

[3] Nguyen J, Wang J, Malik K, et al. Where to begin? on the impact of pre-training and initialization in federated learning[C]//The Eleventh International Conference on Learning Representations. 2023.

[4] Li Z, Shang X, He R, et al. No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023.

[5] Dai Y, Chen Z, Li J, et al. Tackling data heterogeneity in federated learning with class prototypes[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(6): 7314-7322.

1. FedTC's inability to train the backbone and only making minor adjustments to pre-trained backbones on specific datasets significantly limit its applications.
2. The motivation in this paper does not emphasize the issue of client computation but focuses on addressing client drift. This may not align well with FedTC's main advantage, which is training-free clients, rather than generalization performance.
3. Most existing federated learning methods should perform reasonably well on the selected small datasets, especially when everyone shares the same pre-trained backbone. However, as observed in Table 1, most methods lack classification capability. These results may not hold much meaning. This paper should compare these non-one-shot methods after multiple iterations and expand the comparison to more one-shot methods.

Experiments are not well designed.

1. I think the authors should first compare FedTC against recent one-shot federated learning methods on standard IID settings. FedTC doesn't need to beat those methods, but I'd expect FedTC show on par results compared with those methods. I did not see that in the paper.

2. Table 1 which is the main results of the paper, shows FedTC largely outperforms other methods. But to my understanding all of the compared methods (maybe most, I did not check carefully) are not designed to handle non IID cases. Is it expected to see those methods fail? Then what is the main point of Table 1?

3. Since the main selling point is FedTC works better on non IID datasets, I'd like to see FedTC compared against traditional non IID methods (by traditional, I mean training in the standard way rather than federated learning, which presumably should work better than federated learning)

4. Since one shot federated learning is largely unsolved, maybe it makes more sense to improve that rather than to solve much more challenging one shot, non IID, federated learning. This is just a comment.

Lack of novelty: This paper uses a straightforward approach for computing batch-wise class prototypes using a pretrained backbone. The proposed methodology is not well explained and presented, particularly the training process of the classification head using prototypes. Additionally, the lack of elaboration and clarity in this aspect is a drawback. It would be interesting to see a more challenging experimental setup involving domain shift, such as training the pretrained CLIP model on a medical dataset or training a model that is pretrained on synthetic instances on real-world images.