Multi-order Orchestrated Curriculum Distillation for Model-Heterogeneous Federated Graph Learning

We thank the reviewer for your thoughtful reviews and are happy to hear that you find the work valuable. We address your concerns below and hope these clarifications will help you re-evaluate and update the score:

Cons 1: The paper lacks of a theoretical analysis of the proposed method.

Thank you for this valuable suggestion. We have offered an example line chart in our anonymous GitHub repository demonstrating convergence on the Cora dataset under the setting of $\alpha$ = 0.5, and we report the detailed loss trajectory during training as follows:

For theoretical analysis, with regard to WDAD, we add a regulation term to WDAD loss ($\eta \cdot q_{ij}\log q_{ij}$) following the maximum-entropy principle. Taking advantage of this regulation, it can be solved extremely quickly with the Sinkhorn algorithm. Convergence of this algorithm has been proved in previous works[1]. Therefore, our analysis majorly focuses on ACDM and PCNS.

With regard to ACDM, for the adversarial dynamics in Eq. 10 with fixed learning rates $\eta_{model}$ = 0.01, $\eta_{temp}$ = 0.001, we establish:

Theorem 1. Under $\beta$-smooth loss ($||\nabla^2L(\theta)|| \leq \beta$) and bounded gradient variance $\mathbb{E}[||\nabla L - \mathbb{E}[\nabla L]||^2] \leq \sigma^2$, after T iterations:

$\min_{1≤t≤T} \mathbb{E}[||\nabla L(θ_{model}^t)||^2] \leq \frac{2[L(θ^0)-L^*]}{η_{model}T} + η_{model} \beta \sigma^2 + C \eta_{temp} \beta \sigma$,

where $C = O(1)$ captures the timescale separation. Additionally, the gradient norm ratio satisfies empirically:

$\frac{\mathbb{E}[||\nabla L(\theta_{temp})||]} {\mathbb{E}[||\nabla L(\theta_{model})||]} \leq 0.1, \forall t \geq t_0$.

With regard to PCNS, for the pacing function $f(t) = \min(1, \lambda + (1-\lambda)t/T)$ with $\lambda$ = 0.5, $T$ = 50:

Lemma 1. With $\beta$-smooth loss and $\epsilon_{pseudo}$-bounded pseudo-labels ($P(\hat y_{pseudo} \ne y) \leq \epsilon_{pseudo}$):

$|L_t - L_{full}| \leq \beta D_{max}(1 - f(t))^2 + \epsilon_{pseudo}$,

where:

• $\beta \in [10^{-4},10^{-3}]$ measures the maximum ratio of gradient norms between consecutive GNN layers: $\beta = \max_{v \in V} \frac{||\nabla h_v^{l+1}||}{||∇h_v^{l}||}$,

• $D_{\max} = \max_{v \in \mathcal{V}} D(v_i)$ represents the maximum node difficulty based on the difficulty measurer,

• $\epsilon_{pseudo} \leq 0.1$ assumes that the worst-case validation accuracy across clients is at least 90%, which is plausible for well-pretrained local models: $\epsilon_{pseudo} = 1 - \min_k acc_{val}(w_{pretrain}^k)$.

We will include a more detailed discussion of this in the camera-ready version.

Cons 2, Question 1 and Question 2: Key implementation details are missing.

For FL methods, all baselines except FedAvg support model heterogeneity, and they work the same way as they do in computer vision tasks. But for FGL methods, to the best of our knowledge, we are the first to explore model heterogenity in FGL settings. Consequently, existing FGL methods (and also FedAvg) are originally designed for homogeneous models. To adapt them for our heterogeneous setting, we remove direct parameter sharing between architecturally different models and maintain all other original components and hyperparameters. For evaluation, we distill a global model from local private models and assess its accuracy.

Cons 3: The authors do not report statistical measures.

We strongly agree with this suggestion. We report standard deviations on five datasets to assess robustness of our framework. The results (mean ± standard deviation) are shown below:

The small standard deviations demonstrate consistent performance of TRUST across different datasets.

Cons 4: The figures are not sufficiently self-explanatory.

We appreciate this valuable feedback. We add excessive details in order to provide comprehensive visual explanations of our method and its components through color-coding different modules, which has been recognized by some other reviewers like hwGA ("The problem illustration and framework overview are clear and intuitive, and the methodology is also explained with clarity, making the work easy to follow"). In the revised version, we will streamline all figures to focus on the most essential elements and maintain a balance between completeness and readability.

Cons 5: Lack performance evaluation at more extreme levels of data heterogeneity.

We conducted extensive experiments under severe data heterogeneity ($\alpha$=0.1) to evaluate robustness of our method. Results are shown below:

These results demonstrate the superior performance of TRUST even under extreme data heterogeneity, outperforming baselines on 5 of 6 evaluation metrics, with only the local accuracy on the Citeseer dataset showing slightly lower performance.

Cons 6: The paper does not include a runtime or memory complexity analysis.

We also strongly agree with this suggestion. In our experiments, it takes about 15 minutes to run 200 epochs on small-sized datasets like Cora, and for mid-sized datasets like Pubmed and large-sized datasets like ogbn-arxiv, this expands to 1 hour and 8 hours respectively, which is about 25% higher compared to FedType. As for memory usage, in actual experiments, we found that the memory usage of TRUST is similar to that of other baselines, demonstrating that our architectural enhancements do not incur significant memory overhead.

Question 3: How does TRUST scale with the number of clients? What is the maximum number of clients tested in the experimental evaluation?

In the paper, we keep the client number at 10 throughout the main experiment. However, we supplement some experiments on the model performance under varying client number settings. The results are shown below:

Notably, in federated learning settings, data is typically partitioned among clients using Dirichlet distribution. When the number of clients becomes excessively large (25-50), each client receives only a small data portion, leading to highly fragmented subgraphs where local models cannot adequately capture neighborhood information. This can be reflected in the table where performance suffer a great drop in local accuracy when client number is large. However, within a reasonable range, our method proves robustness when client number changes.

All in all, thank you again for your recognition of our work and your valuable suggestions. All suggested improvements will be carefully incorporated into the revised version and please look forward to TRUST being better presented to academic peers.

[1] Cuturi, Marco. "Sinkhorn distances: Lightspeed computation of optimal transport." Advances in neural information processing systems 26 (2013).

Dear Reviewer BwgV,

We sincerely appreciate your continued engagement with our work and the opportunity to further clarify our methodological choices and contributions. Below we address your remaining concerns and hope these clarifications will help improve your assessment of our work:

Choice of baselines:

Thank you for your concern, and we clarify our baseline selection strategy below. For FL baselines, we consider three different approaches as our baselines: (1) Conventional federated learning: FedAvg and a variation FedNova; (2) Knowledge distillation frameworks: MOON and FedType; (3) Prototype learning method: FedProto. For FGL baselines, we select two state-of-the-art FGL works: AdaFGL and FedGTA. While these FGL methods were originally designed for homogeneous settings, we think it is plausible since we are the first to explore model heterogeneity in FGL to the best of our knowledge, and this adaptation can also be seen in some established practices in model-heterogeneous FL works like FedSAK[1]. To ensure fairness, we maintained all original components except direct parameter sharing and preserved hyperparameter configurations. For example, for FedGTA, we remove the uploading of smoothness $H$ which acts as aggregation weights of client model parameters, and we maintain the computation of soft labels.

However, we understand the reviewer's concern about fair comparison between homogeneous and heterogeneous baselines. Therefore, we also conduct homogeneous model experiments (all private models adopt the same GCN architecture) on cora to compare fairly with these homogeneous baselines and demonstrate consistent superiority of TRUST:

As shown above, besides outperforming other baselines in heterogenous settings, TRUST also achieves the highest global accuracy and maintains competitive local accuracy (only marginally lower than the top-performing AdaFGL) in homogeneous settings.

Significance of results:

The experimental results presented in our work demonstrate substantial and meaningful improvements that advance the field of federated graph learning. First, across five different datasets we tested and varying levels of data heterogeneity, TRUST consistently outperforms state-of-the-art baselines. These performance gains are achieved while maintaining practical efficiency, with the additional computational time limited to no more than 25% and memory overhead about the same compared to baseline methods like FedType.

Furthermore, our work represents the first systematic study and formal characterization of model-heterogeneous federated graph learning, identifying and addressing three fundamental challenges in knowledge distillation-based approaches: the need for dynamic task difficulty, adaptive distillation signaling, and cross-class relational transfer. We believe this work will represent a significant step forward in establishing a general model heterogenous framework that is not limited to Euclidean data and making federated learning more flexible and applicable to real-world scenarios with heterogeneous systems and requirements.

We hope these clarifications demonstrate the rigor and significance of our work, and we are deeply grateful for the opportunity to have our work's strengths reconsidered by you.

Best regards,

Authors

[1] Lu, Yuxiang, Shengcao Cao, and Yu-Xiong Wang. "Swiss army knife: Synergizing biases in knowledge from vision foundation models for multi-task learning." arXiv preprint arXiv:2410.14633 (2024).

Dear Reviewer BwgV,

Thank you again for your evaluation of our work and valuable feedback you have provided. At present, all your concerns are responded in the rebuttal. However, as the rebuttal process is already halfway through, we kindly request your feedback to confirm that our response effectively addresses your concerns. If there are any remaining issues, we would like to respectfully convey our willingness to address them to ensure the quality of our work. We sincerely hope that you find our response convincing and kindly consider revisiting your rating.

Best regrads,

Authors

We sincerely appreciate your constructive feedback and positive assessment of our work. Below we provide detailed responses to your valuable comments:

Weakness 1: Need more recent state-of-the-art FGL methods.

Adding more state-of-the-art FGL baselines is indeed a good suggestion. We have conducted experiments on five datasets to confirm the superiority over other FGL methods. The results are shown below:

From the results, we can see that TRUST achieved the best performance on 90% of the evaluation metrics, only falling short on the local accuracy of the Photo dataset, where FedTAD achieved the best performance. The results confirms TRUST's effectiveness against current state-of-the-art methods.

Weakness 2: The authors should conduct computation time and memory usage study.

We strongly agree with this suggestion. In our experiments, it takes about 15 minutes to run 200 epochs on small-sized datasets like Cora, and for mid-sized datasets like Pubmed and Large-sized datasets like ogbn-arxiv, this expands to 1 hour and 8 hours respectively, which is about 25% higher compared to FedType. As for memory usage, in actual experiments, we found that the memory usage of TRUST is similar to that of other baselines, demonstrating that our architectural enhancements do not incur significant memory overhead.

Thank you for your thorough review and encouraging feedback. We are grateful for your positive assessment about importance of our work. We address your questions below:

Weakness 1 & Question 1: Need evaluation metrics for larger datasets.

Thank you for the feedback. We conducted experiments on larger datasets like twitch-gamer and ogbn-arxiv, and we report our experimental results below:

As shown, TRUST consistently outperformed baselines across all datasets, which demonstrates that our method maintains its advantages even at larger scales. We will include these results in the experiment section in the revised version.

Weakness 2 & Question 3: Need clearer implementation details.

We thank the reviewer for this important suggestion. For experimental setup, there are 10 clients in our experiments, where each client maintains a private model with architecture randomly selected from GCN, GAT, or GraphSAGE to simulate real-world model heterogeneity. To facilitate collaboration, each client is equipped with an additional small proxy model that serves as a communication bridge. This proxy model uses a standardized GCN architecture with 3 layers to ensure compatibility across all clients. On the server side, we implement a global model that also adopts a GCN backbone while maintaining consistent configurations with client models for fair comparison.

For baseline implementations, all FL baselines we choose except FedAvg support model heterogeneity, and they work the same way as they do in computer vision tasks. But for FGL methods, to the best of our knowledge, we are the first to explore model heterogenity in FGL settings. So all FGL baselines chosen only support model homogeneity. To include them (and also FedAvg) in heterogenous settings, we remove direct parameter exchange between architecturally different models and maintain all other original components and hyperparameters.

We will add more implementation details in the revised version.

Weakness 3: Limited discussion of computational overhead for WDAD.

We strongly agree with this suggestion. In our experiments, it takes about 15 minutes to run 200 epochs on small-sized datasets like Cora, and for mid-sized datasets like Pubmed and Large-sized datasets like ogbn-arxiv, this expands to 1 hour and 8 hours respectively, which is about 25% higher compared to FedType.

Question 2: Need more parameter sensitivity analysis.

We strongly agree with you that more critical hyperparameters should be included in the sensitivity analysis. We conducted experiments evaluate the impact of key hyperparameters $\tau_{min}$ and $\tau_{max}$, which control the bounds of our adaptive temperature mechanism. The results are shown below:

As shown in the results, when $\tau_{min}$ is too low, the probability distribution of the teacher model will be close to the original one-hot form, which prevents effective knowledge transfer. However, within a reasonable range of $\tau_{min}$ and $\tau_{max}$, the model demonstrates consistent performance. We will incorporate these sensitivity analyses into the revised version to further demonstrate the robustness of our approach.

We sincerely appreciate your thoughtful review and valuable feedback on our work. We address your concerns below and hope these clarifications will help you re-evaluate and update the score:

Weakness 1 and Question 1: Need more evaluation and implementation details of FGL baselines.

We thank the reviewer for this constructive suggestion. To thoroughly evaluate TRUST's performance, we have conducted comprehensive experiments comparing against other state-of-the-art FGL methods across five benchmark datasets:

From the results, we can see that TRUST achieved the best performance on 90% of the evaluation metrics, only falling short on the local accuracy of the Photo dataset, where FedTAD achieved the best performance. The results confirms TRUST's effectiveness against current state-of-the-art methods.

With regard to implementation details, all FL baselines we choose except FedAvg support model heterogeneity, and they work the same way as they do in computer vision tasks. But for FGL methods, to the best of our knowledge, we are the first to explore model heterogenity in FGL settings. So all FGL baselines chosen only support model homogeneity. To include them (and also FedAvg) in heterogenous settings, we remove direct parameter exchange between architecturally different models and maintain all other original components and hyperparameters.

Weakness 2: Limited discussions on the computational complexity.

We strongly agree with this suggestion. In our experiments, it takes about 15 minutes to run 200 epochs on small-sized datasets like Cora, and for mid-sized datasets like Pubmed and Large-sized datasets like ogbn-arxiv, this expands to 1 hour and 8 hours respectively, which is about 25% higher compared to FedType.

Weakness 3: Inconsistent terminology throughtout the paper.

Thanks for this valuable suggestion. We will carefully review and standardize all terminology in the revised version.

Question 2: Details about computational aspects of WDAD?

Following the maximum-entropy principle, we add a regulation term to WDAD loss ($\eta \cdot q_{ij}\log q_{ij}$). Taking advantage of this regulation, it can be solved extremely quickly with the Sinkhorn algorithm. This algorithm exhibits linear convergence and is inherently parallelizable and can be fully vectorized, making it particularly suitable for modern GPU architectures.

Thank you for your positive review and insightful questions. We are grateful for your recognition of our work and we address each of your comments below:

Weakness 1: Need more hyperparameter sensitivity analysis.

We appreciate this valuable feedback. Regarding hyperparameter $\alpha$, which balances the weight between two difficulty measurers $D_{1}$ and $D_{2}$, our analysis reveals:

As shown above, $\alpha$=0.75 and $\alpha$=1 provide optimal performance for local and global accuracy respectively. Values that are too small ($\alpha$=0.5) compromise robustness when pseudo-labels are incorrectly predicted, while larger values ($\alpha$>1.0) underestimate neighborhood distribution impact on difficulty, which results in introducing nodes with vague class representations too early. Importantly, the model maintains stable performance across a reasonable range of $\alpha$ values.

Weakness 2: Limited discussion of synergistic interactions of components.

We thank the reviewer for this insightful observation. We agree that analyzing the interplay between components is crucial to understanding our design. For example, we conducted experiments demonstrating the complementary relationship between PCNS and ACDM:

As shown in the table, the combined approach achieves the best performance by leveraging their complementary strengths: PCNS progressively introduces challenging nodes based on neighborhood distributions and prototype misalignment, while ACDM dynamically adjusts the distillation temperature to match the current difficulty level. This synergistic interaction enables more effective knowledge transfer of complex topological information.

Question 1: Implementation details of baselines?

For FL methods, all baselines except FedAvg support model heterogeneity, and they work the same way as they do in computer vision tasks. But for FGL methods, to the best of our knowledge, we are the first to explore model heterogenity in FGL settings. Consequently, existing FGL methods (and also FedAvg) are originally designed for homogeneous models. To adapt them for our heterogeneous setting, we remove direct parameter sharing between architecturally different models. For evaluation, we distill a global model from local private models and assess its accuracy.

Question 2: Reasons for competition of temperature with model parameters?

We are grateful that you asked this question, because we have the opportunity to explain to you once again an important feature of TRUST. When the proxy model learns well (low L), $\theta_{temp}$ increases to raise $\tau$, softening the possibility distributions and making the task harder to prevent underutilization of model capacity. When the model struggles (high L), $\theta_{temp}$ decreases to lower $\tau$, sharpening the task focus on high-confidence knowledge. In this way, $\tau$ self-adjusts to client capabilities without manual tuning.

Formatting Concerns

We will correct the relevant typo in the revised version.