DKDR: Dynamic Knowledge Distillation for Reliability in Federated Learning

Response to Reviewer YmgD

Dear Reviewer YmgD,

Thank you very much for your thoughtful and thorough review. We are encouraged by your recognition of our innovative approach to federated knowledge distillation. We appreciate your careful examination of our work and hope our detailed responses below will address your concerns.

---

Weaknesses

W1: Computational Overhead Discussion.

A1: Thank you for this important practical concern. We want to clarify the computational distribution and provide quantitative analysis:

Computational Distribution:

• Client-side overhead:
  • Only involves simple arithmetic for dynamic weight calculation: $\frac{\gamma\_s}{\gamma\_s+\gamma\_l}$ and $\frac{\gamma\_l}{\gamma\_s+\gamma\_l}$
  • No SVD or clustering operations on client devices
• Server-side operations:
  • SVD and FINCH clustering perform once per communication round
  • Typical federated learning servers have sufficient computational resources

Further Analysis:
We conduct experiments in OfficeCaltech to further analyze it, as shown in the below table.

Table. Comparison of computational and communication costs in OfficeCaltech dataset.

We also conduct experiments on varying client numbers, as shown in the following table. The computational and communication overheads are within an acceptable range, and the performance improvements are worthwhile.

Table. The relationship of client Number $K$ and computation cost in OfficeCaltech dataset.

W2: Discussion on Domain Overlap Scenarios.

A2: Thank you for highlighting this important assumption. We acknowledge that significant domain overlap poses challenges to our knowledge decoupling approach. However, we have conduct analysis showing that DKDR exhibits graceful degradation rather than catastrophic failure in such scenarios:

Robustness Under Domain Overlap:
When domain boundaries become unclear due to significant overlap, our method handles this situation through multiple mechanisms:

1. Hierarchical Clustering Resilience: FINCH clustering naturally accommodates overlapping structures by creating hierarchical representations, allowing it to identify meaningful groupings even when domains are not perfectly distinct.
2. Fallback to Dynamic Distillation: Even when knowledge decoupling produces suboptimal domain experts due to overlap, the Dynamic Knowledge Distillation (DKD) component continues to provide substantial benefits. The dynamic weighting mechanism adapts to knowledge discrepancies regardless of domain expert quality.
3. Graceful Performance Degradation: In worst-case scenarios where clustering completely fails to separate domains, DKDR essentially reduces to an enhanced federated distillation method that still outperforms standard approaches due to the DKD component.

Empirical Validation:
Our ablation studies (Sec.4.4, Page 9) demonstrate that even when the KDP module is disabled (✗ KDP), the DKD module alone (✓ DKD) still provides consistent improvements across all datasets, confirming the method's resilience to domain overlap issues.

W3: Abalation Study of Local Epoch.

A3: Excellent point! We conduct additional experiments analyzing the sensitivity of DKDR to local training epochs:

Table. Ablation study of Local Epochs T on Digits dataset. DKDR achieves consistent better performance compared to three baselines (FedAVG, FedProto, Scaffold) in 4 of 5 settings.

---

Questions

Q1: Why minimizing forward KLD results in significant distillation bias?

A1: Excellent question that goes to the heart of our motivation! The significant distillation bias from forward KLD arises from its mean-seeking behavior in multi-domain federated scenarios. Let us elaborate with both theoretical and empirical perspectives:

Theoretical Foundation:
Forward KLD aims to minimize: $KL[Z^{t-1}||Z^t\_{k}] = \sum\_j z^{t-1}\_j \log\frac{z^{t-1}\_j}{z^t\_{j,k}}$ From the gradient analysis (Eq. 8a): $\frac{\partial L\_{fkd}}{\partial z^t\_{j,k}} = z^t\_{j,k} - z^{t-1}\_j$

This shows that larger $z^{t-1}\_j$ values receive proportionally higher weights in the loss function. When the global model produces multi-peaked distributions (common in heterogeneous FL), the mean-seeking behaviour of forward KLD forces the local model $z^t\_{j,k}$ to cover all peaks of the global distribution $z^{t-1}\_j$, including low-density regions between peaks.

Concrete Example:
Consider a scenario where:

• Client A specializes in cats (peak at class 3)
• Client B specializes in dogs (peak at class 7)
• Global model after aggregation shows peaks at both classes 3 and 7

When Client A uses forward KLD to distill from this global model, forward KLD forces Client A to assign probability to both cats and dogs. This creates unreasonable high probabilities in low-density regions (the valley between peaks), which results in bias.

Empirical Evidence:
Our experiments (Figure 2) demonstrate this phenomenon. Forward KLD shows higher knowledge discrepancies in AKCs (lower-probability regions) while reverse KLD shows higher discrepancies in DKCs (dominant regions), which validates that forward KLD struggles with multi-peaked distributions.

Thank you for your comprehensive and thoughtful response to my concerns. I appreciate the effort you’ve put into addressing each point with detailed explanations, quantitative analyses, and additional experimental results.

Your clarification on computational overhead effectively addresses my initial worries about practical feasibility. The quantitative evidence showing that computational and communication costs remain within acceptable ranges, even with varying client numbers, is particularly convincing.

I also find your analysis of domain overlap scenarios compelling. The explanation of how hierarchical clustering and dynamic distillation mechanisms enable graceful degradation, combined with ablation results demonstrating the resilience of the DKD module, helps alleviate concerns about the method’s robustness in real-world heterogeneous settings.

Finally, your theoretical and empirical explanation of why forward KLD introduces distillation bias in multi-domain scenarios clarifies a key motivation for your work, enhancing the depth of your contribution.

Based on these thorough responses and the additional evidence provided, I am pleased to increase my score. I encourage you to incorporate these clarifications and experimental details into the final version of the paper to strengthen its overall impact.

Response to Reviewer rkk9

Dear Reviewer rkk9,

Thank you very much for your positive and constructive review. Your acknowledgment of our technical soundness and high impact are deeply appreciated. We hope our responses below will further strengthen your confidence in our work.

---

Weaknesses

W1: Computational Overhead of DKDR.

A1: Sorry for the misunderstanding. We want to clarify that the computational overhead occurs primarily on the server side, not on resource-constrained clients. We will fix it in the final version. Here is the detailed breakdown:

Client-side Operations: Clients only perform standard local training and knowledge distillation using the dynamic weighting mechanism. The dynamic weight calculation ($γ\_s/(γ\_s+γ\_l)$ and $γ\_l/(γ\_s+γ\_l)$) involves simple arithmetic operations with negligible overhead.

Server-side Operations: SVD decomposition and FINCH clustering are performed once per communication round on the server, which typically has sufficient computational resources in federated learning scenarios.

Quantitative Analysis: We conduct overhead analysis on the OfficeCaltech dataset, as shown in the following table.

Table. Comparison of Time and Memory Usage in OfficeCaltech dataset.

W2 & Q1: Data Heterogeneity Ablation.

A2: Excellent point! We conduct additional experiments under extreme heterogeneity settings. We use Dirichlet parameter $ζ=0.05$ (much smaller than our original $ζ∈{0.1,0.3,0.5}$) to create extremely non-IID scenarios in Cifar-10:

Table. Ablation study of ζ in Cifar-10. The smaller ζ is, the more imbalanced the local distribution is.

DKDR shows consistent improvements under extreme heterogeneity, demonstrating that our dynamic distillation mechanism is effective when dealing with severe data heterogeneity. This validates the robustness of our approach.

W3 & Q2: Client Scalability Discussion.

A3: Thank you for this crucial scalability question. We conduct extensive scalability experiments on the OfficeCaltech dataset with varying client numbers:

Table. Ablation study of client Number $K$ on OfficeCaltech dataset. DKDR achieves consistent better performance compared to three baselines in three settings.

For large-scale deployment, we can perform knowledge decoupling every few communication rounds rather than every round, reducing server-side overhead.

Response to Reviewer bzDB

Dear Reviewer bzDB,

Thank you very much for taking the time and effort to review our paper. Your positive feedback on the contribution of our approach, the clarity of our writing, and the comprehensiveness of our experiments is very encouraging. We hope that our responses below will address your concerns and update the score.

---

Weaknesses & Questions

W a): Novelty Discussion of DKDR.

A1: Thank you for the feedback. We would like to emphasize that the core contribution of DKDR lies in being the first to conduct systematic observational analysis of distillation behavior in federated learning. We discover that using different distillation methods leads to distinct knowledge discrepancies between local and global models in federated learning. Specifically, when using forward KLD, local models better fit the Dominant Knowledge Components (DKCs) of the global model, while when using reverse KLD, they better fit the Ancillary Knowledge Components (AKCs). Based on this significant finding, we propose a dynamic weighting mechanism that effectively addresses the bias introduced by different distillation methods, thereby solving the unreliability problem of distillation pathways. Our systematic analysis of federated distillation and the consequent revelation of the essential behaviors of different distillation methods constitute the core innovation. It fundamentally shifts how researchers should approach federated KD by providing a theoretical framework for understanding KLD behaviors. We will emphasize this key point in the final version.

W b).1 & Q a): Convergence Theory.

A2: Thank you for the advice. We extend analysis in Sec.3.2 (Page 4) to prove DKD convergence under non-convex federated settings. Under L-smooth losses with bounded gradients, our main result shows: $\frac{1}{T}\sum\_{t=0}^{T-1}\mathbb{E}|\nabla f(w\_t)|^2 \leq \frac{4(f(w\_0) - f^*)}{\eta T E} + 6L'\eta(E-1)G^2 + \frac{\sigma^2}{K} + 8(L')^2 E\eta\left(G^2 + \frac{\Lambda(\gamma)}{K}\right) + 2\kappa^2 \eta^2 E G^2,$ where $\Lambda(\gamma) = \mathbb{E}\_k\left[|\nabla f\_k(w) - \nabla f(w)|^2\right] \leq \Lambda\_0 - \beta \frac{\gamma\_s \gamma\_l}{(\gamma\_s + \gamma\_l)^2}\mathbb{E}\left[|\nabla L\_{skd} - \nabla L\_{rkd}|^2\right],$ Key Innovation: The gradient diversity term satisfies:
$\Lambda(\gamma) \leq \Lambda\_0 - \beta \frac{\gamma\_s \gamma\_l}{(\gamma\_s + \gamma\_l)^2}\mathbb{E}\left[|\nabla L\_{skd} - \nabla L\_{rkd}|^2\right]$ This proves the dynamic weighting mechanism reduces effective variance compared to standard federated averaging and static KLD approaches ($\Lambda\_0$), leading to faster convergence. The reduction occurs because forward KLD prioritizes DKCs while reverse KLD focuses on AKCs, and their dynamic combination minimizes gradient conflicts. It validates our empirical observations (Table. 6, Page 9) that DKD outperforms static KLD approaches by adapting to knowledge discrepancies.

_Due to space limitation, the complete proof with detailed lemmas and technical derivations will be provided in the appendix in the final version.

W b).2: Dynamic Weight Mechanism (DKD) Discussion.

A3: Let $L\_{dynamic} = α(t)L\_{forward} + (1-α(t))L\_{reverse}$ where $α(t) = γ\_l/(γ\_s+γ\_l)$. As knowledge discrepancies decrease, $α(t)$ adapts to minimize dominant bias sources, thus outperforming fixed weights. We conduct experiments with some fixed weight strategies to confirm it, in which DKD outperforms all fixed weight strategies, as shown in the following table.

Table. Ablation study of the proposed weight strategy DKD on Cifar-100.

W c) & Q b): SOTA Method Comparison (Later 2024).

A4: We have incorporated two SOTA methods (FedTGP (AAAI 2024), FedAA (AAAI 2025)) for comparison with our method. The results are shown in the table below. Our method consistently outperforms these SOTA methods across four datasets. We will include these two baselines in the final version.

Table. Comparison with the SOTA methods in four datasets.

W d) & Q c): Discussion on Large-scale Client Scenarios.

A5: We conduct additional ablation experiments on client numbers N, as shown in the table below. Our method consistently outperforms baselines, demonstrating the robustness of DKDR in large-scale client scenarios. We will include this ablation study in the final version with detailed discussion.

Table. Ablation study of client Number $N$ on OfficeCaltech dataset. DKDR achieves consistent better performance compared to three baselines in three settings.

Q d): Dataset Supplementation.

A6: To further demonstrate the superiority and robustness of DKDR, we introduce the OfficeCaltech dataset for additional comparison with baseline methods, as shown in the table below. The results show that our method consistently outperforms all the baselines.

Table. Comparison with the SOTA methods in OfficeCaltech dataset.

W e) & Q e): Overhead Comparison with SOTA Methods.

A7: We conduct experiments on computational and communication costs, comparing our method with baselines. We calculate the average communication and computational cost per communication epoch for each method, as shown in the table below. The computational and communication costs are acceptable and the resulting performance improvements are worthwhile.

Table. Comparison of computational and communication costs in OfficeCaltech dataset.

Q f): Clustering Method Discussion.

A8: We provide theoretical justification for choosing FINCH over K-means and DBSCAN based on their fundamental formulations and hyperparameter requirements:

K-means requires pre-specification of cluster number $k$ and aims to minimize:
$J=∑\_{i=1}^k∑\_{x∈Ci}∣∣x−μi∣∣^2$ where $\mu\_i$ is the centroid of cluster $C\_i$. The hyperparameter $k$ must be determined a priori, which is challenging in federated settings where the number of domains is unknown.

DBSCAN requires two hyperparameters:

• $ε$ (epsilon): neighborhood radius
• $MinPts$: minimum number of points to form a dense regio A point p is a core point if: $|N\_ε(p)| \geq MinPts$, where $N\_ε(p) = {q \in D : dist(p,q) \leq ε}$ Its performance is highly sensitive to these parameters, and optimal values vary significantly across different data distributions.

FINCH is a parameter-free clustering algorithm. Given the integer indices of the first neighbor of each data point, FINCH directly defines an adjacency link matrix. The elements of the adjacency matrix are defined as follows:

$a\_{ij} = \begin{cases} 1 & \text{if } j = \kappa\_1^i \text{ or } \kappa\_1^j = i \text{ or } \kappa\_1^i = \kappa\_1^j \\\ 0 & \text{otherwise} \end{cases},​$

where $\kappa\_1^i$ symbolizes the first neighbor of point $i$. For a dataset with $N$ points, this produces an $N \times N$ sparse adjacency matrix:

$A = \begin{bmatrix} a\_{11} & a\_{12} & \cdots & a\_{1N} \\\ a\_{21} & a\_{22} & \cdots & a\_{2N} \\\ \vdots & \vdots & \ddots & \vdots \\\ a\_{N1} & a\_{N2} & \cdots & a\_{NN} \end{bmatrix}.$

The clustering equation directly delivers clusters as connected components of the graph $G = (V,E)$ where $V = {1,2,...,N}$ and $E = {(i,j) : A(i,j) = 1}$, without requiring graph partitioning or distance thresholds.

In general, FINCH is more suitable for our federated domain expert identification task because:

• Parameter-Free Operation: Unlike K-means (requires pre-specifying k clusters) and DBSCAN (requires ε and MinPts hyperparameters), FINCH automatically determines optimal clustering without any hyperparameter tuning, making it robust across diverse federated scenarios.

• Adaptive to Unknown Domain Numbers: In federated settings, the number of domains is typically unknown and varies across deployments. FINCH naturally adapts to this uncertainty, while K-means fails without knowing k and DBSCAN requires extensive parameter search.

• Empirical Validation: Our experiments in Table.1 (Page 7) also validate FINCH is more suitable: 50.57% avg accuracy vs. K-means (46.48%) and DBSCAN (43.62%) on Office31, confirming its effectiveness for domain expert identification.

Response to Reviewer CP1N

Dear Reviewer CP1N,

Thank you very much for your detailed and constructive review. We are encouraged by your positive feedback on our motivation, technical contributions, and experimental validation. Your recognition of our novel analysis of forward/reverse KLD in federated learning and the practical innovations of our framework is greatly appreciated. We hope our responses below will address your concerns and strengthen your confidence in our work.

---

Weaknesses

W1: Notation Table of DKDR.

A1: We appreciate this feedback on improving readability. You are absolutely right that a notation summary would enhance accessibility. We will add a comprehensive notation table in the final version that clearly defines all mathematical symbols used throughout the paper, including:

*Table: Notation table for DKDR.

W2: Hyperparameter Details for Clustering Methods.

A2: For the ablation study in Table 1, we used the following hyperparameters:

• K-means: optimal k=3 (corresponding to the 3 domains in Office31)
• DBSCAN: appropriate $ε=0.05$ and $MinPts=1$
• FINCH: No hyperparameters required (parameter-free)

---

Questions

Q1: How SVD-filtered improves domain expert identification?

A1: Excellent question! The key insight is that raw client parameters contain both shared knowledge and domain-specific signals mixed together. By clustering raw parameters, we would group clients based on their overall model similarity, which is dominated by shared knowledge rather than domain-specific characteristics.

Our approach specifically isolates domain-specific signals through two steps:

1. Shared knowledge removal: $v^t\_k = w^t\_k - w^{t-1}$ removes the shared knowledge
2. SVD filtering: Retains the top-r singular values to extract the most significant domain-specific patterns while filtering noise

This design ensures that clustering focuses purely on domain-specific knowledge differences, leading to more accurate domain expert identification (see details in Sec.4.3, Page 9).

Q2: How μ=0.5 was determined and whether it generalizes across all datasets?

A2: The choice of μ=0.5 is both theoretically motivated and empirically validated:

Theoretical Justification: μ represents the threshold for separating Dominant Knowledge Components (DKCs) from Ancillary Knowledge Components (AKCs). When μ=0.5, we capture exactly half of the cumulative probability mass in the DKCs, providing a natural balance between the two components. This creates an optimal trade-off where:

• μ < 0.5: DKD degenerates toward reverse KLD (mode-seeking bias)
• μ > 0.5: DKD degenerates toward forward KLD (mean-seeking bias)

Empirical Validation: Our sensitivity analysis (Figure 5) shows that μ=0.5 consistently achieves optimal performance across both Cifar-100 and Office31 datasets. The stability across different dataset types (single-domain vs. multi-domain) demonstrates its generalizability.

Q3: Conceptual Difference with Prototype-based Methods.

A3: While both approaches leverage domain-specific knowledge, they differ fundamentally in methodology and granularity:

FedProto: Uses class prototypes to align feature representations across clients. It focuses on feature-level alignment by matching class centroids, assuming similar classes should have similar representations across domains.

DKDR: Creates specialized teacher models for different domains at the knowledge level leveraging domain-specific knowledge.

Our experimental results show DKDR (50.57%) outperforms FedProto (39.98%) on Office31, demonstrating the effectiveness of our knowledge-centric approach.

Q4: Ablation Studie of Backbones.

A4: The choice of backbone is based on the complexity of the datasets. Single-domain tasks, such as Cifar-10 and Cifar-100, are relatively simpler, so we use ResNet-10 to reduce computational cost while maintaining performance. Multi-domain tasks, such as Office31 and Office Home, are more complex due to domain shifts, requiring the more powerful ResNet-18 to capture intricate patterns effectively. This is detailed in Sec.4.1 (Page 8). We also conduct experiments with ResNet-18 on Cifar-100 to validate consistency:

Table. Ablation study of backbone on Cifar-100 dataset ($\zeta=0.5$). DKDR achieves consistent better performance compared to three baselines (FedAVG, FedProto, Scaffold).

The consistent improvement demonstrates that the effectiveness of DKDR is backbone-agnostic. We will include this ablation in the final version.

---

Limitations

L1: Computational Overhead Discussion.

A1: Sorry for the misunderstanding. We want to clarify that while SVD and clustering add computational cost, this occurs only once per communication round on the server side, not on resource-constrained clients. The server-side computation is typically acceptable in federated learning scenarios. Additionally, our analysis shows the overhead is reasonable compared to the significant performance gains achieved. We conduct experiments in OfficeCaltech to further analyze it, as shown in below table.

Table. Comparison of Time and Memory Usage in OfficeCaltech dataset.

L2: Discussion on Domain Overlap Scenarios.

A2: You are correct that the knowledge decoupling module relies on the assumption that domains are sufficiently distinct for clustering to accurately identify domain experts. When domains have significant overlap, this assumption may not hold, potentially affecting the quality of domain expert identification. When domain boundaries are unclear, our method doesn't fail catastrophically. Instead, it exhibits graceful degradation:

• Scenario 1: If clustering fails to separate domains correctly, some clusters may contain clients from multiple domains,
• Scenario 2: The dynamic distillation (DKD) component continues to provide benefits even when domain experts are suboptimal,

Result: DKDR reduces to a form closer to standard federated distillation but still benefits from the DKD. It can still perform well in such situation (see details in Sec.4.4, Page 9).

Response to Reviewer Wn6p

Dear Reviewer Wn6p,

We sincerely thank you for your comprehensive review and positive feedback on our DKDR framework. We appreciate your recognition of the theoretical analysis of forward and reverse KL divergence. We aim to address your concerns in our detailed responses below, hoping to provide clarity and demonstrate the effectiveness of our proposed approach.

---

Weaknesses & Questions

W1 & Q1: Training Overhead and Latency of DKDR.

A1: Thank you for your advice! We conduct comprehensive experiments evaluating both computational and communication costs using the OfficeCaltech dataset. Compared with baselines, the computational and communication expenses of DKDR are acceptable and the performance gains are worthwhile.

Table. Comparison of Time and Memory Usage in OfficeCaltech dataset.

We also conduct scalability experiments with varying client numbers (K ∈ {20, 60, 100})， as shown in following table. For K=100 clients, server overhead remains under 3 seconds per round.

Table. The relationship of client Number $K$ and computation cost in OfficeCaltech dataset.

W2 & Q2: Non-image Dataset Supplementation.

A2: Thank you for this valuable suggestion! We acknowledge that our current evaluation focuses primarily on image classification tasks, which limits the generalizability of our findings. To address this concern, we conduct additional experiments on federated recommendation task to demonstrate the effectiveness of DKDR across different modalities. We evaluate DKDR on federated recommendation using the Adressa dataset. Adressa is publicly released by Adresseavisen, a local newspaper company in Norway. we use the 6-th day click to build training dataset and construct historical clicks from the first 5 days samples. We randomly sample 20% clicks from the last day clicks for validation and the rest clicks for testing. The historical clicks of validation and testing dataset are constructed from the first 6 days samples. Following many previous news recommendation works ([1,2,3]), we use AUC, MRR, nDCG@5 and nDCG@10 as evaluation metrics. AUC measures a model's ability to distinguish positive and negative classes. MRR averages the reciprocal rank of the first relevant item in a ranked list. nDCG@5 and nDCG@10 evaluate ranking quality for the top 5 and 10 items, respectively, weighting relevant items higher when they appear earlier, normalized for comparison. The results are shown in the following table. Our analysis reveals that core principles in DKDR (pathway reliability and teacher reliability) are indeed modality-agnostic. The multi-peaked distribution problem occurs in both vision and recommendation domains when aggregating heterogeneous client models, validating our theoretical foundation. We will include comprehensive cross-modal evaluation in our final version to demonstrate the broad applicability of DKDR beyond computer vision tasks.

Table. Comparison with the SOTA in the non-image dataset Adressa.

[1] Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news recommendation with long- and short-term user representations. In ACL, pages 336–345.

[2] Tao Qi, Fangzhao Wu, Chuhan Wu, Yongfeng Huang, and Xing Xie. 2020. Privacy-preserving news recommendation model learning. In EMNLP 2020, pages 1423–1432.

[3] Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020a. User modeling with click preference and reading satisfaction for news recommendation. In IJCAI, pages 3023–3029.