Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning

We would like to thank the reviewer for the time in reviewing. We appreciate that you consider our presentation is clear, and our method is effective and robust. Please see our detailed feedback for your concerns below.

W1: Potential solutions and complexities of implementation.

Ans for W1: Thank you for your valuable comments. Indeed, FAFI may have limitations in handling domain shifts/multi-task scenarios. FAFI assumes that invariant feature representations should remain consistent or similar, enabling them to mitigate model inconsistencies. However, in domain shift/multi-task scenarios, feature representations often exhibit significant discrepancies across different domains or tasks, particularly in the isolated training paradigm.

A potential solution is to cluster and ensemble feature representations across various domains or tasks, allowing unbiased representations to encapsulate multi-domain/task knowledge.

Regarding implementation in real-world FL environments, as detailed in Algorithm.1(Appendix C), FAFI requires only two modifications compared to existing OFL methods, making it easily adaptable for both clients and the server:

1. On the client side, we train the feature extractor in a contrastive manner and use the prototypes instead of the classifier.
2. On the server side, fusion is performed at the feature level rather than the parameter or prediction level.

We will highlight these in our revised version.

W2: Comparisons with other methods

Ans for W2: Thanks for your important comments. We have conducted new experiments comparing 3 methods, i.e., FedCompress, FedKD, and FedACG, which reduce the communication cost by gradient compression, gradient fractorization, and transferring gradient momentum. These new results on CIFAR-10 are provided below and added to our revision. Results show that FAFI achieves better efficiency compared with all baselines.

W3: Privacy issue

Ans for W3: Thanks for your very valuable comments. We agree with the statement there is no free lunch for the privacy-utility trade-off[1], for that the tenet of FL is to seek balance among efficiency, effectiveness, and privacy. FAFI, as well as most OFL methods, all aim to ensure efficiency for affordable deployment while striving to find the effectiveness boundaries as much as possible and maintain privacy below an acceptable level. As to FAFI, the possible leakage is the prototypes, which have been proven in previous works to compromise no privacy. For stricter privacy requirements, one potential solution is implementing differential privacy or using noised samples with the cost of some effectiveness degradation.

W4 & Q1: Comparisons with other methods

Ans for W4 & Q1: Thanks for your valuable comments. We present the difference between our proposed FAFI with feature-enhanced and prototype-based methods as follows:

For feature-enhanced works, we note that FAFI is one-shot and selective enhances the feature at the inference stage for high performance.

For prototype-based methods, FAFI aligns global prototypes by aggregating learnable local prototypes in one round.

Mul. is short for multi-round. We will add these discussions in our revision.

W5: More in-depth efficiency analysis

Ans for W5: Thanks for your valuable comments. We have conducted the memory and computation cost analysis of FAFI. Due to limited space, please see the response to Reviewer Chua W1 & Q1.

W6 & W7 & Q2: Writing issues

Ans: Thanks for your careful reading.

1. The $\Delta_{intra} = \vert L(x, y) - L(x', y) \vert$ is the performance discrepencies between any two samples $x, x'$ with the same label $y$. For any two samples $x, x'$, we can find a function $A$ that satisfies $x' = A(x)$. $A$ can abstract the feature variation among samples.
2. The 'performance' in L69 represents the performance of the local model, we observe that better local performance is always accompanied by large parameter discrepancies(Fig.2-b). The performance in L258 denotes the performance of the global model.

Sorry for these misunderstandings. We will revise these writing issues and format errors and provide more explanations for clarity.

[1] No free lunch theorem for security and utility in federated learning. ACM TIST, 2022.

We sincerely thank the reviewer for taking the time to review our work. We greatly appreciate you find our method is efficient and data-free. Please find our detailed responses to your concerns below.

W1 & Q1: Ablation study on $L_{ssl}$ and $L_{proto}$.

Ans for W1 & Q1: Thanks for your important comments. As suggested, we have conducted new ablation studies on $L_{ssl}$ and $L_{proto}$, and the results are shown below.

We note that the performance of FAFI is significantly improved by using $L_{ssl}$ and $L_{proto}$ for enhanced feature representation and discriminative prototypes, and the combination of $L_{ssl}$ and $L_{proto}$ achieves the best. Additional details will be added in Appendix F in our revision.

W2 & Q2: Impact of the feature fusion strategy.

Ans for W2 & Q2: Thanks for your important comments. As suggested, we have conducted new evaluations on the impact of the feature fusion strategy with $Dir(0.1)$, and the results are shown below.

We note that $\mathcal{N}(0,1)$ facilitates the best performance in the tested cases, as it effectively captures feature information compared to other strategies. Additional details and discussions about this hyperparameter will be included in Appendix F in the revision.

We sincerely thank the reviewer for taking the time to review our work. We greatly appreciate you find the paper is well-motivated and well-demonstrated. Please find our detailed responses to your concerns below.

W1 & Q1: Details in IFFI.

Ans for W1 & Q1: Thanks for your important comments. Sorry for missing the explanations about IFFI. We use the Gaussian distribution $\mathcal{N}(0, 1)$ in Equ.8. For clarity, we will add an ablation study on the hyperparameters. More details can be seen in response to Reviewer BvHZ W2 & Q2 and will be added in Appendix E in the revision.

W2 & Q2: More settings to verify the effectiveness of FAFI.

Ans for W2 & Q2: Thanks for your important comments. We have conducted new evaluations on CIFAR-10 with data heterogeneity $Dir(0.05)$ and $Dir(0.1)$, and the results are shown below.

$C$ is the communication rounds. We note that FAFI can achieve competitive performance even in extreme heterogeneous scenarios (i.e., $Dir(0.05)$), requiring only 44.7 MB of communication cost. Additional details will be included in Appendix F in our revised version.

C1 & C2: Typos and open source code

Ans for C1 & C2: Thanks for your comments. We will fix the typos in Appendix A, and we promise we will open the source code.

We sincerely thank the reviewer for taking the time to review our work. We greatly appreciate your recognition of the proposed method as both technically sound and high-performing. Please find our detailed responses to your concerns below.

W1 & Q1: Applicability to resource-constrained scenarios.

Ans for W1 & Q1: Thanks for your important comments. We note that our proposed FAFI can support lightweight models and is applicable to resource-constrained scenarios. We have complemented statistics on computational and memory cost with MobileNet and ResNet-18 on CIFAR-10.

We use the GPU memory occupied during local training as the metric for memory cost, while computation cost is measured by the time taken per epoch on GPU(RTX 4090) and CPU(Intel Core i7-11700K). Here, $b$ represents the batch size. We believe that FAFI can achieve competitive performance even in resource-constrained scenarios, requiring only 11 seconds on a GPU with less than 2GB of memory or approximately 3 minutes on a CPU. Additional details will be included in our revised version.

W2: Assumption on the number of classes, which may not hold in dynamic or open-set FL scenarios.

Ans for W2: Thanks for your comments. FAFI assumes that the number of classes is known in advance. However, in dynamic or open-set FL scenarios, the number of classes may change over time. We note that FAFI can be easily extended to these scenarios by extracting the invariant features of the new classes and learning a new discriminative prototype for them. We will include this discussion in our revision.

W3: Some notions are not clear.

Ans for W3: Thanks for your comments. Sorry for the unclear notations. For clarity, 1) the $\Delta_{intra} = \vert L(x, y) - L(x', y) \vert$ is the performance discrepancies between any two samples $x, x'$ with the same label $y$; 2) $A$ refers to the function that can abstract the feature variation among samples. We will clarify these notations in the revision.

W4 & Q1: Model heterogeneity scenarios.

Ans for W4 & Q1: Thanks for your comments. We note that FAFI can support heterogeneous models. We have conducted new evaluations on heterogeneous models with five different architectures (LeNet, ResNet-18, VGG, MobileNet, ResNet-50) on CIFAR-10 with data heterogeneity $Dir(0.1)$ as follows and added more details in Appendix F. We only report part of them here due to the limited space.