DFL$^2$G: Dynamic Agnostic Federated Learning with Learngene

Q1. Assume a one-shot dataset in the client.

Thank you for your attention to this detailed question. In our method, the data uploaded to the server for the clustering process consists of one-shot vectors obtained through truncated singular value decomposition of a common dataset. However, the client-specific common dataset remains local and is used exclusively for local model training.

Q2. Lack of Dynamic Cluster Management.

Thank you for pointing this out! Firstly, on the server side, clustering is performed based on the low-dimensional left singular vectors uploaded by the client, with the number of clusters can be randomly set based on the data distribution. Subsequently, the unknown clients select the closest clusters to join based on the existing cluster distribution. Therefore, the number of clusters is typically determined initially based on the distribution of the existing clients.

Q3. Insufficient Privacy Guarantees.

Thank you for pointing this out! We must emphasize that the vectors uploaded to the server for clustering are used in a one-shot manner and have been shown to be effective in protecting data privacy [1]. Additionally, the iDLG method we employ reconstructs data at pixel-level accuracy based on model gradient information, which is commonly used to evaluate the privacy guarantees of federated learning methods [2-4]. Moreover, during the validation phase, the server recovers the original data solely through the transmitted learngene rather than the uploaded vectors.

Q4. Presentation

1) Citation format；2) Algorithm location.

Thank you for such valuable and detailed comments. We have revised the citation format throughout the text and moved the algorithm to the main body of the paper.

3）Describtion of $d$.

Thank you for your suggestion. Here, $d$ is the number of left singular vectors selected after performing truncated singular value decomposition on the local data. Specifically, we define the decomposition as $\mathcal{X} _ {i,d} = \mathbf{U} _ {i,d} \mathbf{\Sigma} _ {i,d} \mathbf{V} _ {i,d}^T$, where $\mathbf{U} _ {i,d} = [\mathbf{u} _ 1, \mathbf{u} _ 2, \ldots, \mathbf{u} _ d]\in \mathbb{R}^{m \times d}$ (with $d \ll$ rank($\mathcal{X} _ i$) and $m$ denotes the number of samples for client $i$) represents the top $d$ most significant left singular vectors, capturing the essential features of the underlying data distribution. We follow the [1] and select $d = 5$ to mitigate the risk of data leakage. Additionally, to facilitate linear algebraic computations, we transform the matrix $\mathbf{U} _ {i,d}$ into a vector $\mathbf{u} _ {i,d} \in \mathbb{R}^{md \times 1}$.

Q5. Description of the cluster updating.

Thank you for your suggestions. Clusters are established based on the data distribution when the existing clients begin training. When a new client joins the federated learning system, it is assigned to an existing cluster based on similarity and updates the cluster's mean vector (line 20 in Algorithm 1).

References:
[1] Vahidian S, Morafah M, Wang W, et al. Efficient distribution similarity identification in clustered federated learning via principal angles between client data subspaces[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(8): 10043-10052.

[2] Wu Y, Kang Y, Luo J, et al. Fedcg: Leverage conditional gan for protecting privacy and maintaining competitive performance in federated learning[J]. arXiv preprint arXiv:2111.08211, 2021.

[3]Scheliga D, Mäder P, Seeland M. Combining Variational Modeling with Partial Gradient Perturbation to Prevent Deep Gradient Leakage[J]. arXiv preprint arXiv:2208.04767, 2022.

[4] Ma Y, Yao Y, Xu X. PPIDSG: A Privacy-Preserving Image Distribution Sharing Scheme with GAN in Federated Learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(13): 14272-14280.

Q1. Readability

1） The description of text and notations.

Thank you for pointing this out. In the revision, we will simplify the notations of local datasets. We specifically designed $\sum_{l=1}^{L} \xi_{k,i}^{(l)}=1$ to normalize the scores across all layers, ensuring that scores are measured on a unified scale. This allows for clearer comparisons of the relative contributions of each layer to the overall model updates, thereby facilitating the identification of learngene within the model.

2） The description of objective function.

Thank you very much for your constructive comments. We will give a more thorough and clearer description of the loss function: $\mathcal{L} _ {all} = \mathcal{L} _ {cls} + \lambda _ 1\mathcal{L} _ {gen} + \lambda _ 2\mathcal{L} _ {elg}$. The corresponding hyperparameter ablation study is shown in Appendix A.3.

Q2. Problems in the SVD decomposition and formulation.

Thank you for your valuable feedback! The client applies truncated SVD decomposition to their private data, selecting the top $d$ most significant left singular vectors. These vectors effectively capture the essential characteristics of the underlying data distribution while minimizing privacy leakage [1]. For simplicity in linear algebra computations, the matrix $\boldsymbol{U} _ {i,d} = [\boldsymbol{u} _ 1, \boldsymbol{u} _ 2, \ldots, \boldsymbol{u} _ d]\in \mathbb{R}^{m \times d}$ (with $d \ll rank(\mathcal{X} _ i)$) is further reshaped into a vector form $\boldsymbol{u} _ {i,d} \in \mathbb{R}^{md \times 1}$.

Q3. Privacy analysis.

Thanks you for pointing this out! Firstly, the vectors uploaded to the server are used solely for one-shot clustering, and it has been demonstrated in [1] that this approach effectively safeguards data privacy. Second, the iDLG method employed achieves pixel-level accurate data reconstruction based on model gradient information, which is commonly used for privacy verification in federated learning. During the validation phase, we recover the original data solely using the gradient information of the model initialized with the learngene, without relying on these vectors. The privacy verification results of different FL methods, as shown in Figure 5, are based on the same experimental setup including the clustering pre-processing between clients.

Q4. The setting of local epochs.

Thank you sincerely for your valuable suggestions. Setting local epochs = 10 is a common practice in FL methods [2-4]. Furthermore, using a smaller number of local epochs may help mitigate overfitting to the client data, which could lead to a slight improvement in the performance of our method. We plan to test this in future work.

Q5. Reason of new metric.

The reason for introducing the new metric is that, in real-world scenarios, many edge devices face communication constraints. To validate our proposed method's ability to reduce communication costs while maintaining model performance, we also considered a fair comparison with other model pruning methods. The use of this metric in Table 1 is specifically to assess model performance, while other modules are evaluated using different metrics. For example, privacy protection is validated using PSNR.

Q6. The bias and variance of the experimental results.

Thank you for pointing this out. Figure 4 presents the experimental results for SVHN with $s$ = 4, while Table 3 reports the average results over the final 10 epochs, which explains the observed differences. The reason for the high variance lies in the poor performance during the initial epochs. This is because the initialization of the agnostic client models combines learngenes with random parameters, requiring an adaptive process to the data. However, the subsequent performance demonstrates a consistent upward trend.

The limited improvement observed in the baseline method indicates that the inherited model possesses strong generalization capabilities, providing good performance from the initial stages. However, this also restricts the ability to learn personalized knowledge for new client, resulting in local models with straight-line performance.

Q7. Table captions and citation format.

We will modify these questions in the revised manuscript.

References:
[1] Vahidian S, Morafah M, Wang W, et al. Efficient distribution similarity identification in clustered federated learning via principal angles between client data subspaces[C]. AAAI. 2023.

[2] Wu Y, Kang Y, Luo J, et al. Fedcg: Leverage conditional gan for protecting privacy and maintaining competitive performance in federated learning[J]. arXiv preprint arXiv:2111.08211, 2021.

[3] Yi L, Wang G, Liu X, et al. FedGH: Heterogeneous federated learning with generalized global header[C]. ACM MM. 2023.

[4] Wang J, Yang X, Cui S, et al. Towards personalized federated learning via heterogeneous model reassembly[J]. NeurIPS, 2024.

Q1. Lack of convergence proof and theoretical support.

Thank you for your suggestions. First, we must emphasize that the proposed dynamic agnostic federated learning addresses a practically significant application problem, aiming to overcome the limitations of traditional federated learning in dynamic and agnostic scenarios. Second, we introduced the innovative concept of "Learngene," which provides a highly practical solution. Finally, to validate the effectiveness of the proposed method, we conducted a series of experiments to evaluate the model's performance in terms of training, communication efficiency, and privacy preservation.

Q2. The experimental results are limited.

We really appreciate your constructive feedback! Due to time constraints, we added another non-IID scenario ($\beta$ = 0.5, 0.1) using the Dirichlet distribution on the benchmark dataset CIFAR10, as shown below:

The table below presents a performance comparison of model training after initialization for agnostic clients:

Q3. There is no comparison with the baselines having similar objectives. Thank you very much for your constructive comments. While it might appear that our approach shares similar objectives with FedProto[1] and FedTGP[2], there are fundamental distinctions in scope and purpose. FedProto and FedTGP aim to address joint optimization in distributed networks, leveraging class prototypes to minimize communication costs. In contrast, our method is not limited to low-cost communication but is designed to ensure effective initialization for models on agnostic clients.

In the dynamic agnostic FL scenario, agnostic clients and previously trained clients share no overlapping classes ( $\mathcal{C} _ {\text{agnostic}} \cap \mathcal{C} _ {\text{known}} = \emptyset$). Methods like FedProto and FedTGP rely on server-learned global class prototypes, which is unsuitable for our scenario. Using old class prototypes to initialize distinct new class prototypes on agnostic clients is infeasible and unjustifiable due to the lack of overlap between their class distributions. We will discuss prototype-based communication approach in the revised manuscript.

Q4. Questions about the cef measure.

Thank you for your insightful question. In FL, discussions often focus on overcoming communication constraints and improving model performance during collaborative learning. Our proposed "cef" measure is specifically designed for scenarios where communication resources are limited, but devices need to learn from the knowledge of others to improve their own performance. Since there is a trade-off the communication costs and model performance, the "cef" measure was introduced to allow fair comparisons with methods that require the transmission of entire models.

References:
[1] Tan Y, Long G, Liu L, et al. Fedproto: Federated prototype learning across heterogeneous clients[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(8): 8432-8440. [2] Zhang J, Liu Y, Hua Y, et al. Fedtgp: Trainable global prototypes with adaptive-margin-enhanced contrastive learning for data and model heterogeneity in federated learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(15): 16768-16776.

Q1. Ambiguous Notation for Agnostic Clients.

Thank you for your suggestion. We have provided a clear explanation of this in the revised manuscript, specifically in lines 125-130.

Q2. Scalability Concerns Due to Server-Side Storage Overhead.

First, the bottlenecks of federated learning are typically centered around communication overhead [1, 2] and the limited storage capacity of edge devices [3]. Thus, the goal of our proposed method is to reduce costs by leveraging communication based on learngenes. Second, the storage overhead at the server side is generally caused by high-frequency data generated by time-series sensors [4]. In contrast, in our approach, the uploaded vectors used for clustering are one-shot, and the number of cluster models can be customized.

Q3. Insufficient Explanation of the FIM Computation.

Thanks. We have described the FIM computation in more detail in lines 208-213 of the manuscript. We approximate the diagonal of the Fisher information matrix for each parameter indexed by $j$ in the model $\tilde{\theta} _ {i}$ (refers to $\tilde{\theta} _ {k,i}$), expressed as $F _ {i,j}=\mathbb{E}\left[\left(\frac{\partial \log \mathcal{h}( \tilde{\theta} _ i \mid \mathcal{D} _ i)}{\partial \tilde{\theta} _ {i,j}}\right)^{2}\right]$. Here, the likelihood function $\mathcal{h}( \tilde{\theta} _ i \mid \mathcal{D} _ i)$ represents the fitness of the model parameters given the data $\mathcal{D}_i$. In our implementation, the log-likelihood is indirectly computed using the log_softmax output, which corresponds to the log probability of the correct class label. The Fisher information diagonal is then obtained by computing the gradient of the log-likelihood with respect to the model parameters, aligning with the concept of Fisher information.

Q4. Complexity of the Learngene Concept

Thank you for pointing this out. Learngene is an innovative paradigm inspired by biological genetic evolution, designed to distill inheritable knowledge from ancestral models in an open-world setting, creating learngene that enable efficient adaptation to new tasks. We deeply consider the "Collaborating & Condensing & Initializing" mechanism in dynamic FL based on the perspective of "Accumulating & Condensing & Inheriting" in Learngene to enhance model interpretability. Specifically, this involves leveraging "collaborative learning of local models, condensation of generalized knowledge, and initialization of agnostic client models" to achieve low-cost communication and adapt to dynamic agnostic scenario.

Q5. Unclear Combined Loss Function.

Thank you very much for your constructive comments. We set different hyperparameters $\lambda_1$ and $\lambda_2$ for different loss functions, and performed ablation studies in Appendix A.3.

Q6. Ambiguities in Experimental Figures and Tables.

Thanks. We acknowledge that the performance during the first 10 rounds is lower than that of other methods. The reason for this is that the learngenes used to initialize the agnostic client models are information fragments from the model, while the other components are initialized randomly. As a result, the model requires an adaptation process to adjust to the new task. However, in the subsequent rounds, the performance consistently outperforms that of other methods, demonstrating the model's generalization capability and providing a solid foundation for further training. Table 4 and Table 5: We have added detailed descriptions of the datasets and specific statistical measures to Tables 4 and 5. While the original manuscript used the same hyperparameter settings, we also conducted ablation studies on the Elastic and Learngene components, as shown in the second row of Table 4.

Q7. Absence of Theoretical Convergence Guarantees.

Thanks. First, we must emphasize that the proposed dynamic agnostic federated learning addresses a practically significant application problem, aiming to overcome the limitations of traditional federated learning in dynamic and agnostic scenarios. Second, we introduced the innovative concept of "Learngene," which provides a highly practical solution. Finally, to validate the effectiveness of the proposed method, we conducted a series of experiments to evaluate the model's performance in terms of training, communication efficiency, and privacy preservation.

References:
[1] Luping W., Wei W., Bo L. I. CMFL: Mitigating communication overhead for federated learning. ICDCS, 2019.
[2] Malaviya S., Shukla M., Lodha S. Reducing communication overhead in federated learning for pre-trained language models using parameter-efficient finetuning. PMLR, 2023.
[3] Dai Y., Xu D., Maharjan S., et al. Joint load balancing and offloading in vehicular edge computing and networks. IEEE Internet of Things Journal, 2018.
[4] Zhang T., He C., Ma T., et al. Federated learning for internet of things. ACM Conference on Embedded Networked Sensor Systems, 2021.

Thank you for this.My main concerns centered around three areas:

1. Presentation and writing of the paper ie: systematic development of the concept learn gene
2. Issues in the ablation part of the paper ie: Combined Loss function
3. Lack of theoretical convergence guarantees.

The authors have addressed 2) for me. 1) can be addressed in a separate version. 3) is not attended at the moment by the authors.

Therefore, I am raising the score slightly.

Thanks