FedL2G: Learning to Guide Local Training in Heterogeneous Federated Learning

We appreciate the reviewer's recognition of the value, theoretical analysis, and sufficient experiments of our work. Below, we reply to the identified weaknesses and questions. Lines xxx and Section x correspond to specific lines and sections in our paper.

Response to weakness 1: Figure 1 is unclear.

• We appreciate the reviewer's feedback regarding Figure 1. In the revised manuscript, we have updated Figure 1 and its corresponding main text to make the challenges addressed in this work more intuitive and easier to understand. Key elements have been clarified and emphasized.
  The revised text and figure are highlighted in blue for the reviewer's convenience.

Response to weakness 2: Figure 2 is complicated.

• We acknowledge the reviewer's concerns about the complexity of Figure 2. In the revised version, we have simplified Figure 2 by focusing on the key contributions (steps 3 and 4 of our FedL2G framework) and removing unnecessary lines. The figure caption has been expanded to include detailed explanations, ensuring readers can easily follow the processes depicted.

Response to weakness 3: the paper does not validate the phenomenon illustrated by Figure 1.

• As discussed in Section 4.5 of the original manuscript, the phenomenon of objective mismatch is validated through experiments. Specifically, Figure 3 demonstrates how the original local loss (cross-entropy loss) fluctuates during federated training in methods like FedProto and FedDistill due to the objective mismatch problem, resulting in inferior model performance.

Response to weakness 4: uploading local class prototypes or features of local data samples may leak local data privacy.

• Clarification: Our FedL2G method does not upload raw features or local class prototypes. Instead, it uploads gradients of guiding vectors, as shown in Line 12 of Algorithm 1, which are initialized randomly and iteratively refined through client feedback. These gradients are not directly related to sensitive local data or class-specific statistical information.
• Privacy Analysis: In Section 4.6 FedL2G Protects Feature Information, we provide a detailed discussion on this topic. Figure 4 demonstrates that guiding vectors differ significantly from class prototypes, ensuring privacy.
• Enhanced Privacy: To further enhance privacy, we incorporated Gaussian noise into the gradients of guiding vectors following the approach in FedPCL [1]. FedL2G retains strong performance while improving privacy protection (see Table 1).

Table 1: Test accuracy on Cifar100 in the Dirichlet setting using HtFE$_8$ with a noise scale of s and perturbation coefficient p.

[1] Tan Y, Long G, Ma J, et al. Federated learning from pre-trained models: A contrastive learning approach. NeurIPS, 2022.

Response to weakness 5: misuse of symbol n.

• In the revised manuscript, we have replaced "n" with "D" to clearly denote the total data size, distinguishing it from the number of clients N .

Response to weakness 6: There is a lack of qualitative or quantitative analysis to prove that the proposed method effectively reduces the cross-entropy loss on users' local data.

• In Section 4.5 FedL2G Prioritizes the Original Task, we present both qualitative and quantitative analyses. Results show that FedL2G-l and FedL2G-f consistently reduce the original local cross-entropy loss, addressing the objective mismatch problem and enhancing local model performance.

We appreciate the reviewer's recognition of the well-motivated objective mismatch issue and sufficient experiments of our work. Below, we reply to the identified weaknesses. Lines xxx and Section x correspond to specific lines and sections in our paper.

Response to weakness 1: extend theoretical analysis on a newly proposed optimizer besides SGD.

• Our theoretical analysis focuses on the fundamental and widely-used SGD optimizer, aligning with most FL research. This approach ensures our analysis is general and adaptable, making it extensible to other specific optimizers. While the suggested combination of Nesterov-accelerated SGD (outer) and AdamW (inner) is effective for distributed low-communication training, it is tailored for a specialized use case rather than standard FL for distributed training of a large model. We believe maintaining the focus on SGD strengthens the generalizability of our contributions, but we are open to exploring other optimizer settings in future work.

Response to weakness 2: missing insights for theorems.

• Key Insights: The primary purpose of our theorems is to establish the non-convex convergence of FedL2G at a convergence rate of O(1/T), as highlighted in Section Abstract, Section Introduction, Line 316, and Line 917. This result demonstrates the theoretical soundness and efficiency of FedL2G in heterogeneous FL settings.

Response to weakness 3: client participation ratio.

• Adherence to Existing Standards: In our current experiments, we evaluate FedL2G using a client participation ratio (ρ) of 0.5, which is a widely accepted standard in FL research [1,2].
• Additional Results for Lower Participation Ratios: As suggested by the reviewer, "the participation ratio is lower with more clients," we conducted additional experiments with ρ=0.1 (10% client participation) for 100 clients. For comparison, our manuscript includes experiments with ρ=0.5 for 50 clients and ρ=1 for 20 clients. As shown in Table 1, FedL2G continues to outperform all baselines, demonstrating its robustness and scalability to lower participation ratios.

Table 1: Test accuracy on Cifar100 in the Dirichlet setting using HtFE$_8$ with ρ=0.1.

[1] Zhang J, Liu Y, Hua Y, et al. Fedtgp: Trainable global prototypes with adaptive-margin-enhanced contrastive learning for data and model heterogeneity in FL. AAAI, 2024.

[2] Li Q, Diao Y, Chen Q, et al. Federated learning on non-iid data silos: An experimental study.ICDE, 2022.

We appreciate the reviewer's recognition of the clarity, value, efficiency of meta-learned guiding vectors, and sufficient emprical and theoretical analysis of our work. Below, we reply to the identified weaknesses and questions. Lines xxx and Section x correspond to specific lines and sections in our paper.

Response to weakness 1 and question 1: Feature Extraction Consistency.

• Effectiveness of Averaging: Averaging, as introduced in FedProto [1], is a widely accepted and effective practice in FL for aggregating and sharing global information under both data and model heterogeneity. In our FedL2G framework, updating local models using global guiding vectors plays a crucial role in aligning local models and promoting consistency in their feature extraction. Without the global guiding vectors, local models lack this critical alignment, resulting in significantly poorer performance, as demonstrated in Table 1.
• Consistency Improvement During the FL Process: As discussed above, aligning local models with global guiding vectors gradually enhances feature extraction consistency across heterogeneous models. This improvement is evidenced by the increased cross-client cosine similarity shown in Table 2.

Table 1: Test accuracy on two datasets in the Dirichlet setting using HtFE$_8$.

Table 2: Cross-client cosine similarity (mean) of extracted features for the same class in FedL2G-f on Cifar10 under the Dirichlet setting with HtFE$_8$.

[1] Tan Y, Long G, Liu L, et al. Fedproto: Federated prototype learning across heterogeneous clients. AAAI, 2022.

Response to weakness 2 and question 2: Ablation Study of Warm-Up Period.

• Computational Efficiency: The warm-up phase, which includes steps 1, 3, 4, 5, 6, and 7, is computationally lightweight and closely mirrors the main FL process, with the exception that step 2 (local model updates) is skipped (see Lines 233-240). This design ensures that the warm-up phase requires minimal additional effort.
• Scalability: As shown in Lines 221-225, we only require participating clients to join instead of all clients in the warm-up phase.
• Ablation Study Results: As shown in Table 3, FedL2G maintains competitive performance even with no warm-up (T'=0). The introduction of the warm-up phase does not impact the overall convergence speed. However, a short warm-up phase enhances guiding vector initialization, improving subsequent rounds' performance. Notably, FedL2G-f benefits more from a warm-up phase due to the higher learning capacity of the high-dimensional feature space. Overly large T' negatively impacts both variants due to overfitting on untrained client models.

Table 3: Ablation study of the number of warm-up rounds (T') on Cifar100 in the default Dirichlet setting using HtFE$_8$ with different T'. Results (a, b) represent (accuracy, the total number of converged rounds including the warm-up round). The total converged round represents convergence speed.

We appreciate the reviewer's recognition of the clarity, value, and rigorous convergence analysis of our work. Below, we reply to the identified weaknesses. Lines xxx and Section x correspond to specific lines and sections in our paper.

Response to weakness 1: Comparison to FedPCL and FPL.

FedPCL and FPL differ from our FedL2G in both scenario and objective, and their direct application is inapplicable in our model heterogeneity setting. However, we have modified these methods to enable a fair comparison and included results in Table 1.

• Scenario: Both FedPCL and FPL focus solely on data heterogeneity with homogeneous client model architectures. Specifically,
  • FedPCL requires pretrained and frozen backbones shared among clients and only trains homogeneous projection layers (two fully-connected layers) locally.
  • FPL is tailored for domain-shift settings and aggregates homogeneous client models into a global model during each communication round.
  • In contrast, FedL2G is designed for practical scenarios with both data and model heterogeneity, enabling collaboration among diverse architectures such as shallow CNNs, GoogleNet, MobileNet, ResNets, and ViT.
• Objective:
  • Both FedPCL and FPL focus on adjusting local features according to the global/cluster prototypes (aggregated/existing feature information extracted by client models), where local features can be misled by poor prototypes in early FL rounds.
  • FedL2G instead learns guiding information (independent of extracted features) to meta-learn guidance that minimizes local loss during each FL round. This prioritization of local objectives differentiates FedL2G from FPL's prototype-based approach.
• Comparison: We modified FedPCL and FPL for use in our scenario:
  • FedPCL*: Trains the entire local model from scratch.
  • FPL*: Omits global model aggregation due to model heterogeneity. Table 1 shows their relatively poor performance, reflecting their limitations in model heterogeneity settings and the amplification of poor prototypes by their contrastive learning strategies.

Table 1: Test accuracy on four datasets in the Dirichlet setting using HtFE$_8$.

Response to weakness 2: Using small quiz set and warm-up period is complex.

• Quiz set: The quiz set is not an additional dataset but a small portion held out from the original training data (see Lines 65-67), ensuring fairness and no extra data advantage over other baselines. As shown in Table 2, only 2 to 5 samples are sufficient for our FedL2G to achieve strong performance. This implementation is straightforward and supported by tools like higher.
• Warm-up period: The warm-up phase (containing step 1, 3, 4, 5, 6, 7) is computationally lightweight and identical to the main FL process, except for skipping step 2 local model updates (see Lines 233-240). It requires minimal additional effort. Additionally, our FedL2G-l achieves an accuracy of 41.7 with (T'=0) (i.e., no warm-up), and FedL2G-f achieves 41.6 with (T'=1).

Table 2: Test accuracy on Cifar100 in the Dirichlet setting using HtFE$_8$ with different quiz set size (qss).

Response to weakness 3: Adding more recent relevant FL methodologies.

• At submission, the most recent baseline identified was FedGH (2023), which we compared extensively (Section 2.1 & 4). These comparisons already demonstrate FedL2G's strong performance in handling data and model heterogeneity.
• We have now incorporated FedPCL and FPL comparisons (as detailed above). Additionally, we include results for FedMRL [1], a newly identified baseline (NeurIPS 2024). Our FedL2G maintains its superiority in Table 1, even when FedMRL utilizes both the local model and an auxiliary global model for inference.

[1] Yi L, Yu H, Ren C, et al. Federated Model Heterogeneous Matryoshka Representation Learning. NeurIPS, 2024.

We sincerely appreciate your time and effort in reviewing our paper and recognizing its contributions. However, we would like to clarify two misunderstandings in your comments:

1. Collaborative Setting:
   Our theoretical analysis is indeed conducted within a collaborative setting, as detailed in Appendix C. Specifically, we follow the framework established in FedProto [1], proving convergence with a focus on individual client losses that include a collaborative term, $\mathcal{G}$, as shown in Equations (C.13) and (C.14). In the subsequent proofs, $\mathcal{G}$ is further expressed using the collaborative notation $\mathbb{E}_{i \sim [N]} \pi_i$. For simplicity, we omit the client ID $i$ in Equations (C.15–C.20), as mentioned in our manuscript. Therefore, our theoretical guarantees are firmly rooted in a collaborative setting.

2. Agents vs. Federated Learning:
   Our work is centered on the federated learning domain, not on multi-agent systems. The focus is on client-server interactions typical of federated learning frameworks, rather than agent-based paradigms.

We hope this clarification addresses your concerns and helps in further understanding our contributions. Thank you again for your thoughtful review.

Reference:
[1] Tan, Yue, et al. "FedProto: Federated Prototype Learning Across Heterogeneous Clients." AAAI, 2022.