1. Broader Applicability and Merit of FedGMKD

Thank you for your insightful feedback regarding the scope of our experiments and the potential broader applicability of FedGMKD. We appreciate the opportunity to clarify and expand upon our work by exploring its performance on datasets beyond the computer vision domain.

a. Choice of IMBD Dataset

To address the reviewer's concerns, we conducted additional experiments using the IMBD (Internet Movie Database) dataset, a well-known resource in natural language processing for sentiment analysis. This text-based dataset, comprising a vast number of movie reviews, allowed us to test FedGMKD's adaptability across different data modalities. IMBD's popularity in NLP research makes it an ideal benchmark for evaluating federated learning algorithms, particularly as sentiment analysis challenges models to generalize effectively across diverse client data distributions.

b. Experimental Setup

In alignment with the paper's experimental setup, we utilized 10 clients and conducted the experiment over 50 epochs. Due to the transition from computer vision to NLP, the model used was changed from ResNet-18 to BERT, which is well-suited for text-based tasks like sentiment analysis. BERT's capability to capture contextual word embeddings made it an appropriate choice for handling the IMBD dataset. The main metrics for evaluation were local accuracy, global accuracy, and average computation time per client.

c. Results and Analysis

The results demonstrate that FedGMKD consistently outperforms other federated learning algorithms in both local and global accuracy metrics on the IMBD dataset. This suggests that the core mechanisms of Cluster Knowledge Fusion (CKF) and Differential Aggregation Technique (DAT) are effective not only for image-based datasets but also for text-based datasets like IMBD. The high local accuracy of 85.11 and global accuracy of 51.58 highlight the framework’s robustness and adaptability to different data types, further validating its broader applicability beyond just vision datasets.

2. Response to typo of line 129

We apologize for the typo on line 129. We have carefully reviewed the manuscript and have fixed all typos and mistakes, including the missing space in "jin." The final version of the manuscript reflects these corrections.

3. Response to Assumptions

Thank for your highlighting, the final version of my paper will include assumptions in main paper.

4. Response to Comparison with FjORD

Thank you for highlighting the importance of comparing FedGMKD with architectures like FjORD, which address model heterogeneity. Here, we clarify why FjORD was not initially chosen for comparison and provide an analysis of the experimental results by using FjORD and FedGMKD.

a. Reason for Not Initially Choosing FjORD

Both FjORD and FedGMKD aim to address distinct challenges in federated learning. FjORD primarily focuses on model heterogeneity by employing dropout techniques that dynamically adjust model capacity, allowing models to efficiently adapt to diverse client computational resources. This makes FjORD effective in handling varying hardware environments. Conversely, FedGMKD targets data heterogeneity through CKF and DAT, aiming to enhance model performance by aligning client data distributions and improving aggregation efficiency. These differing focuses led us to initially compare FedGMKD with frameworks that similarly prioritize data heterogeneity, rather than FjORD’s emphasis on model heterogeneity.

b. Experimental Comparison and Analysis

In the absence of official code for the FjORD method, we combined various reproduced code implementations to create an experimental version for comparison. This allowed us to assess FjORD's performance under the same conditions used for FedGMKD. The results are summarized below:

The comparison results indicate that while FjORD effectively manages model heterogeneity through dropout techniques, FedGMKD provides superior performance in scenarios where data heterogeneity is the primary concern. The ability of FedGMKD to adapt to diverse client data distributions and enhance aggregation strategies makes it a more effective solution in the contexts tested. These results underscore FedGMKD's capability to address federated learning challenges beyond model heterogeneity, reinforcing its broader applicability and effectiveness.

Thank you for your insightful feedback. We appreciate the opportunity to address the concern regarding the exploration of a wider range of hyperparameters to demonstrate FedGMKD's robustness& sensitivity and discuss the justification of using GMM in FedGMKD.

1. Hyperparameter Exploration in FedGMKD

a. Exploration of Hyperparameters \gamma\ and \lambda\

The provided ablation study systematically varies the hyperparameters 𝛾 and λ within the FedGMKD framework using the CIFAR-10 dataset with 10 clients over 50 epochs. Below is a summary of the results:

b. Robustness of FedGMKD

1. Consistency Across Hyperparameter Values

• High Local and Global Accuracies: FedGMKD maintains high accuracy across different \gamma\ and \lambda\ values, indicating robustness.
• Optimal Performance Stability: With \gamma\ = 0.06 and \lambda\ = 0.06, FedGMKD achieves its peak performance, highlighting adaptability.

2. Comparative Performance

• Superior Performance: FedGMKD outperforms FedAvg, FedProto, and FPL, underscoring its robustness and efficiency.

c. Sensitivity of FedGMKD

Adaptive Tuning Insights

• Parameter Optimization: While FedGMKD operates effectively across various settings, fine-tuning \gamma\ and \lambda\ is essential to leverage its full potential and adapt to specific datasets.

2. The Justification of Using Gaussian Mixture Models (GMM) in FedGMKD

Below, we provide a detailed justification for selecting GMM over other clustering methods, supported by relevant literature and content.

a. Handling Data Heterogeneity

• Robustness in Diverse Environments: GMM effectively handles heterogeneous data distributions, crucial for federated learning. The study "Personalized Federated Learning under Mixture of Distributions" highlights GMM's robustness in diverse client data environments.
• Managing Non-IID Data: GMM efficiently manages non-IID data distributions, enhancing learning efficiency in federated contexts, as shown in "An Efficient Framework for Clustered Federated Learning."

b. Flexibility and Adaptability

• Effective Utilization of Task Similarities: GMM leverages unknown task similarities, offering theoretical guarantees for convergence, as discussed in "Robust Unsupervised Multi-task and Transfer Learning on Gaussian Mixture Models." This adaptability is advantageous for FedGMKD, improving model generalization.
• Transfer Learning Capabilities: GMM enhances clustering performance through transfer learning, as seen in "A General Transfer Learning-based Gaussian Mixture Model for Clustering," enabling FedGMKD to handle limited data per client.

c. Privacy and Outlier Management

• Data Privacy and Heterogeneity: GMM addresses data privacy and heterogeneity, demonstrated in "Federated Learning for Misbehavior Detection with Variational Autoencoders and Gaussian Mixture Models," making it suitable for federated learning applications.
• Robustness Against Outliers: GMM's probabilistic nature allows it to handle outliers effectively, as evidenced in "Robust Unsupervised Multi-task and Transfer Learning on Gaussian Mixture Models," making it an ideal choice for FedGMKD.

references

• Smith, V., Chiang, C.-K., Sanjabi, M., & Talwalkar, A. (2017). "Personalized Federated Learning under Mixture of Distributions." arXiv:1705.10467
• Zhang, Y., & Yang, Q. (2017). "Robust Unsupervised Multi-task and Transfer Learning on Gaussian Mixture Models." arXiv:1711.05995
• Zhao, Y., Liu, P., Cheng, J., Chen, M., & Chen, L. (2020). "Federated Learning for Misbehavior Detection with Variational Autoencoders and Gaussian Mixture Models." IEEE Transactions on Intelligent Transportation Systems. DOI:10.1109/TITS.2020.3006572
• Sattler, F., Wiedemann, S., Müller, K.-R., & Samek, W. (2020). "An Efficient Framework for Clustered Federated Learning." arXiv:2004.03337
• Luo, P., Ding, X., & Zhao, X. (2019). "A General Transfer Learning-based Gaussian Mixture Model for Clustering." Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI-19). DOI:10.24963/ijcai.2019/243

Thank you for your feedback. We appreciate the opportunity to discuss the weakness and questions.

1. Motivation and Insights

a. Motivation and Insight of FedGMKD

FedGMKD tackles key challenges in personalized federated learning, particularly the non-IID client data that can lead to suboptimal model performance. Unlike traditional KD-based PFL methods that require public datasets, raising privacy issues and struggling with data heterogeneity, FedGMKD introduces Cluster Knowledge Fusion (CKF) and Differential Aggregation Technique (DAT). CKF uses Gaussian Mixture Models to create prototype features and soft predictions without public datasets, maintaining privacy and handling data heterogeneity. DAT improves server aggregation by prioritizing client data quality, allowing high-quality data to have more influence on the global model, thereby enhancing both local and global accuracy.

b. Tracking and Handling Stragglers

Although stragglers are not explicitly focused on, DAT mitigates their impact by weighting client contributions based on data quality, ensuring reliable aggregation.

2. Response to Lack of Necessary Citations in the Convergence Analysis Part

Our convergence analysis relies on standard theoretical principles widely accepted in federated learning, which do not typically require citations. For example, the FedProto presents similar analyses without additional citations. Our analysis aligns with these practices, focusing on mathematical rigor consistent with academic norms.

3. Response to Comment on Computation and Communication Overhead

Table 1 displays the average times for FedProto and FedGMKD across three datasets. FedProto reduces computational demands and communication overhead by using prototype sharing and only transmitting prototypes. In contrast, FedGMKD optimizes computations through CKF and DAT while sharing prototypes, predictions, and quality assessments to efficiently transmit necessary data. When compared, FedGMKD is more computationally efficient due to its optimized operations and minimizes communication overhead by selectively exchanging essential data.

4. Calculating Discrepancy with Non-IID Data

a. Purpose of Discrepancy Calculation

The discrepancy is calculated to determine the weights for global aggregation for each class. This process ensures that the contribution of each client to the global model is proportional to the quality and relevance of its data.

b. Handling Missing Classes

If a client lacks data for certain classes, it will not provide corresponding prototype features and soft predictions. These equations are used by clients to calculate prototype features and soft predictions only for the classes they have.

Here, $\hat{h}\_i^{j}$ and $\\hat{q}\_i^{j}$ represent the prototype feature and the soft prediction vector for class $j$ at client $i$, synthesized from the cluster knowledge, where $R$ is the number of clusters. So if clients miss some classes, they will not calculate $\hat{h}\_i^{j}$ and $\\hat{q}\_i^{j}$.

c. KL Divergence for Missing Classes

This equation is used to compute the KL divergence for the classes present in the client data. If a client lacks a class, the KL divergence for that class is effectively zero.

Then, the weights for a client's contribution to each class are calculated, excluding those for which the client lacks data. Therefore, clients without data do not contribute to the global aggregation for those classes, while clients with data for the missing classes have more weight, ensuring that the aggregation is dominated by contributions from clients with relevant data, improving the robustness and accuracy of the global model.

5. Detailed Federated Learning (FL) Setting and client participation

a. FL setting

The paper has provided the details of the federated learning (FL) setting:

• Datasets (Lines 222-236): SVHN, CIFAR-10, CIFAR-100.
• Non-IID Partitioning (Lines 449-454): Dirichlet distribution
• Model Architecture (Lines 238-243): ResNet18
• Baselines (Lines 245-251): Includes FedAvg, FedProx, Moon, FedGen, FedMD, FedProto, and FPL.
• Implementation (Lines 253-259): PyTorch on NVIDIA A100 GPUs, using Adam optimizer, with specific hyperparameters.

b. Client Participation

Setting the client participation rate to 1 is common in federated learning to provide a controlled benchmark for evaluating algorithm performance. This approach allows researchers to assess the theoretical limits and maximum potential of federated learning algorithms under ideal conditions without the complexities introduced by client dropout or partial participation. By using full participation as a baseline, studies can more accurately compare the efficiency and accuracy of different algorithms when all clients contribute to the training process.

In practice, this setting is widely adopted across various federated learning studies:

• Tan et al. (2022). "FedProto: Federated Prototype Learning Across Heterogeneous Clients." arXiv:2105.00243.

• Mendieta et al. (2022). "Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning." In CVPR.

• Karimireddy et al. (2020). "SCAFFOLD: Stochastic Controlled Averaging for Federated Learning." In ICML, PMLR, 119:5132-5143.

• Huang et al. (2023). "Rethinking Federated Learning with Domain Shift: A Prototype View." In CVPR.

• McMahan et al. (2017). "Communication-Efficient Learning of Deep Networks from Decentralized Data." In AISTATS.

• Chen et al. (2023). "The Best of Both Worlds: Accurate Global and Personalized Models Through Federated Learning with Data-Free Hyper-Knowledge Distillation." In ICLR. arXiv:2301.08968.

Thanks to the extra experiments. I know that must spend lots of time. In these experimental results, we can see that Local Accuracy (PR=0.1) and Global Accuracy (PR=0.1) in SVHN dataset is about under 10.0. However, a random initialized model can get 10.0 ACC in a 10-class problem. So does this mean that all of these methods have lost their effectiveness? At the same time, there are lots of methods that gain higher ACC at this setting (100 clients, PR=0.1 with $\alpha=0.5$ SVHN and CIFAR100). At the same time, they also have no public dataset. In summary, I am still concerned about the practicability and effectiveness of these methods compared to other methods. I will keep my score.

Thank you for your insightful feedback. We appreciate the opportunity to discuss FedGMKD with more complex neural network architectures, its adaptability, and its real-world impacts and applications.

1. FedGMKD with More Complex Neural Network Architectures

To address your suggestion, we conducted additional experiments using the ResNet-50 model on the CIFAR-10 dataset. Below is a summary of the results compared with our original experiments using ResNet-18:

While ResNet-50 theoretically offers higher representation power, practical challenges like increased model complexity, potential overfitting, and greater communication requirements limit its performance in federated settings. FedGMKD with ResNet-50 still outperforms other schemes but does not surpass the results with ResNet-18. These findings highlight the importance of balancing model complexity with the realities of federated learning environments. In future work, we plan to explore advanced aggregation techniques and personalized strategies to mitigate these challenges and fully harness the capabilities of complex models like ResNet-50.

2. Adaptability of FedGMKD

FedGMKD is designed with a modular architecture that enhances adaptability to diverse data distributions and experimental settings through its use of Gaussian Mixture Models for clustering and flexible parameter customization.

a. Modular Design of FedGMKD

• Feature Extraction and Clustering: FedGMKD utilizes GMM for clustering, which can be adapted to various datasets by adjusting the number of Gaussian components according to the dataset's complexity (Section 3.2).
• Differential Aggregation Technique (DAT): DAT weights client contributions based on data quality, making aggregation adaptable to different distributions and class imbalances (Section 3.3).

b. Handling Diverse Data Distributions

• Customizable Hyperparameters: Parameters such as clustering centres, learning rates, and temperature for knowledge distillation are adjustable for different datasets. Our experiments (Section 3.7) showed flexibility for SVHN, CIFAR-10, and CIFAR-100.
• Adaptive Clustering: GMM models data as a mixture of Gaussian distributions, adapting to varying distributions and complexities by adjusting Gaussian components and clustering settings.

c. Detailed Experimentation with Different Datasets

• Experimental Settings: The option.py file in FedGMKD's code allows customization of parameters like the number of clients, classes, learning rates, and epochs.

3. Real-World Impacts and Applications of FedGMKD

FedGMKD enables impactful applications across healthcare, finance, and smart cities by enhancing data integration, privacy, and communication efficiency.

a. Healthcare

• Improved Local and Global Performance: FedGMKD can significantly improve diagnostic models by integrating diverse medical data from multiple institutions. This results in more accurate and comprehensive predictions, enhancing both local models tailored to specific patient populations and a robust global model.
• No Need for Public Data: Hospitals can train collaborative models without the need for publicly available medical datasets, ensuring compliance with privacy regulations and safeguarding patient data.
• High Communication Efficiency: The efficient communication protocol ensures that even hospitals with limited bandwidth can participate in federated learning, making advanced diagnostic tools accessible across a wide range of medical facilities.

b. Finance

• Enhanced Fraud Detection: FedGMKD improves the performance of fraud detection models by aggregating diverse transaction data from various financial institutions. This collective intelligence helps in identifying complex fraud patterns more accurately.
• Privacy Preservation: Financial institutions can collaborate without exposing sensitive customer information, maintaining data confidentiality and complying with stringent financial regulations.
• Efficient Aggregation: The differential aggregation technique ensures efficient use of communication bandwidth, which is crucial for real-time fraud detection and quick response to fraudulent activities.

c. Smart Cities

• Optimized Resource Management: FedGMKD can enhance traffic management and energy consumption models by integrating data from different city departments, leading to more efficient use of city resources and improved quality of life for residents.
• Scalable Solution: The high communication efficiency allows the deployment of smart city solutions across multiple, distributed sensors and devices without overloading the network, making it scalable for large urban environments.
• Handling Heterogeneous Data: FedGMKD effectively integrates diverse data sources for comprehensive urban planning.