Rethinking the Temperature for Federated Heterogeneous Distillation

We thank reviewer TEBy for the constructive comments: "…enhance accuracy under model/data heterogeneity", "…reduces communication costs compared to proxy-based methods", "… represents a novel synthesis of ideas from robust FL, KD, and malicious detection … ", "… address a variety of practical deployment scenarios". To thoroughly address your concerns, we will answer the questions one by one:

Questions:

Q1: Potential Interaction

The interaction of category temperature tuning (Eq.4) with the multi-level distillation (Eq.3) can have a positive impact on mitigating model heterogeneity. We facilitate knowledge transfer by applying category scaling to global logits, which simultaneously improves multi-level distillation. The ablation experiments in Tab.4 show that the combination of the two significantly improves performance, demonstrating that they are complementary in dealing with heterogeneity.

Q2: Differences with Existing Dynamic Temperature Methods

Please refer to Reviewer JGAK's Q1 for the answer to this question.

Weaknesses:

W1: Extension to Transformer

Logits-based knowledge distillation is also applicable to the Transformer architecture, and similar work [1] has validated the effectiveness of hierarchical distillation on Transformers. Since we deliver logits information and eliminate architecture specificity through branching exits, our approach is more general and can be widely adapted to different architectures.

W2: More Detailed Clarity

The exact model distribution for heterogeneous clients: the four models AlexNet, ShuffleNetV2, ResNet18, and GoogleNet are randomly and evenly assigned to clients. In distillation learning, logits serve as high-level guidance. Our elastic temperature calculation uses next-stage logits (vs. current-stage) to avoid local stage overfitting and enhance stability. Besides, the experiments on Cifar-10 can verify this: | Cifar-10 | next-stage | current-stage | | --- | --- | --- | | accuracy | 72.76% | 71.54% |

Responses to Questions about Theoretical Analysis

Theorem 4.1: Theorem 4.1 does not require specific assumptions and it is valid in the framework of both multilevel elastic temperature and category-aware global temperature scaling. According to the information entropy formula, the information entropy of logits is only affected by the softmax temperature and the model output and is only related to the temperature coefficient when the model output cannot be changed. In Eq.3, we set the initial temperature ξ and limit its variation by a scaling factor γ and a log function to avoid the boundary case. In addition, in lines 237-238 of the code file client.py, we set the temperature range to 0.5 ≤ τ ≤ 5.

Corollary 4.2: We refer to the assumptions of [4] to simplify the theoretical proof process. In addition, Corollary 4.2 is proposed to provide theoretical support for category temperature setting. Regardless of the initial state of the model, we dynamically adjust the category temperatures of the global logits based on the local learning of different categories. Since temperature scaling is done at the softmax level, it is applied evenly to all classes in the same round, thus ensuring that the scaling effect remains consistent. In addition, we focus mainly on model heterogeneity and non-iid data distributions in FL, whereas changes in the category distribution are not our concern.

Theorem 4.3: Although Theorem 4.3 assumes category equilibrium for analytical simplicity, temperature scaling remains effective in heterogeneous settings, enhancing gradient updates for difficult categories. Tab.1(a) and Fig.1 validate this in non-IID scenarios (e.g., α = 0.1, 0.5, pat), showing significant performance gains. Fig.2 further demonstrates that increasing distillation layers from 1 to 4 improves accuracy, plateauing at 5, confirming the validity of our multi-level mechanism.

Supplementary experiments Experiments on model heterogeneity in DeL

These decentralized baselines except DeSA[3] basically do not consider model heterogeneity, and it is unfair to forcefully change them to baselines for heterogeneous models. Due to time constraints, we supplemented our experiments with model heterogeneity on only one dataset: |Flower102|DeSA|ReT-FHD| |---|---|---| |accuracy|37.17%|40.82%|

References:

[1] One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation

[2] Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation

[3] Overcoming data and model heterogeneities in decentralized federated learning via synthetic anchors

Due to space constraints, I apologize for not being able to answer some of the questions, but if there are still questions, we can respond to them in the second discussion. Finally, thank you for your time and effort in reviewing our work.

We thank reviewer e9EW for the constructive comments: "…dynamically adjust distillation intensities across different model layers...", "…validate the effectiveness of ReT-FHD...". To thoroughly address your concerns, we will answer the questions one by one:

Questions:

Q1: Connection between elastic temperature scaling and the security components, and the rationale for incorporating security

Our motivation is to explore whether logits can be applied to complex heterogeneous FL scenarios. To this end, we adopt Z-Score to normalize the knowledge distillation process and further introduce multi-level distillation, elastic temperature scaling, and secure validation mechanisms for optimizing cross-architecture knowledge distillation and enhancing model robustness, respectively. In addition, the verification mechanism of the model-based blockchain FL approach is difficult to apply to the logits sharing scenario. Therefore, we design a Z-Score-based verification mechanism, which is customized for the logits sharing FL method, which ensures efficient logits-based communication and effectively defends against malicious attacks.

Q2: Lack of annotation in equations

• Eq.3 describes the multistage elastic temperature mechanism, where each stage of the model is assigned an independent temperature adjustment. The temperature of each stage of the model is adjusted according to the logarithmic difference of the next stage. The temperature is bounded by $\xi$ to maintain stability and prevent degradation. The scaling factor $\gamma$ controls sensitivity. $\Delta Z_l$ represents the difference between current and next stage logits, guiding the temperature adjustment. $\Delta Z_{\max}$ normalizes the temperature across stages for consistency.

• Eq.4 describes the category-aware temperature scaling mechanism, which dynamically adjusts each category’s temperature based on its cumulative loss relative to the average loss. The weighting factor $\beta$ controls the sensitivity of temperature updates. $\mathcal{L}_{\mathrm{CE}}(f(\tilde{x}_c))$ represents the cumulative loss for category c, and the average loss across all categories serves as a global reference. The maximum absolute difference standardizes the temperature update across categories, ensuring consistency.

• Eq.5 defines the optimization objective of ReT-FHD. Here, $|\mathcal{D}|$ and $|\mathcal{D_i}|$ denote the total and local data sizes, respectively. $\mathcal{L}_{\mathrm{CE}}$ is the cross-entropy loss. The model applies KL divergence for knowledge distillation at each layer. $\sigma^t\left(\cdot\right)$ and $\sigma^s(\cdot)$ represent the softmax functions of the global and local logits, respectively. $\mathbf{z_t}$ denotes the global logits, $\bar{z_t}$ local logits' mean, and $\operatorname{Var}\left(\mathbf{z}_t\right)$ local logits' variance. $G(\Delta Z)$ and $\tilde{\tau}_c$ represent the temperature coefficients for model stages and categories, calculated using Eq.3 and 4, respectively. Unlike global logits, local logits are unaffected by category temperature adjustments.

Experimental Designs Or Analyses:

Our validation mechanism, designed for logits-based scenarios, standardizes malicious logits detection and is mainly applicable to logits sharing, rather than model parameter-sharing FL frameworks. In security experiments, we demonstrate its effectiveness compared to other FL methods. On CIFAR-10, each validation (Eq.7) requires ~100-200 FLOPs, while a ResNet18 forward pass takes ~1.8 GFLOPs, making the blockchain overhead negligible. Moreover, since the blockchain mechanism only verifies logits without altering the FL communication process, ReT-FHD maintains the same communication efficiency as standard FL, as confirmed in Tab.5.

Essential References Not Discussed:

Thanks for pointing out these references. Due to time constraints, we add additional experiments on Cifar10 to more fully assess the effectiveness of ReT-FHD in dealing with model heterogeneity and safety. While FPL[1] mitigates domain shift, it may introduce information redundancy in our setting, whereas SDEA[2] is designed for homogeneous models and relies on high-quality public datasets. |Hetero. & No attack|RethinkFL|ReT-FHD| |---|---|---| |accuracy | 68.21% | 72.76% | |Attack & Homo.|SDEA|ReT-FHD| |accuracy|50.89%|51.75%|

Other Strengths And Weaknesses:

If you have any doubts about specific equations, please refer to our response to Q2. As for Fig.1, the y-axis represents accuracy, and the x-axis indicates the number of communication rounds.

References:

[1] Rethinking federated learning with domain shift: A prototype view

[2] Self-driven entropy aggregation for byzantine-robust heterogeneous federated learning

Thank you for dedicating your time and effort to reviewing our work. Please let me know if I have addressed your concerns and if not, we welcome any further questions you might have.

We thank reviewer JGAK for the constructive comments: "…reducing communication costs and enhancing security …", "… provides clear experiments …". To thoroughly address your concerns, we will answer the questions one by one:

Questions:

Q1: Differences from existing dynamic temperature knowledge distillation methods

Existing dynamic temperature adjustment strategies—BGNN[1] (network-learned temperature for GNN nodes), CTKD[2] (course-learning-based scheduling), and DTKD[3] (sharpness approximation for temperature optimization)—are designed for centralized learning with homogeneous architectures and IID data, adjusting temperature across training rounds (e.g., early vs. late epochs). Our approach 1) employs class-specific calibration for global logits to mitigate non-IID label skew and 2) assigns independent temperature coefficients to distinct model stages (e.g., shallow vs. deep layers) within a single training round to address model heterogeneity.

Q2: Which one contributes more to solving model heterogeneity in FL

As detailed in Line 32 and Line 78, the multi-level distillation is foundational to our heterogeneous distillation framework. As in the Ablation Study (Tab.4)(removing multi-level settings reduces performance by 3.08%), we validate the effectiveness in heterogeneous federated learning scenarios. Meanwhile, dynamic temperature( in Theorem 4.1 and 4.3) allows for more knowledge about global logits and improves performance by 2.82% (Tab.4). We therefore respectfully assert that these components form an indivisible contribution core.

Weaknesses:

W1: Not much discussion of dynamic temperature work

Our motivation is to explore whether logits distillation can be adapted to federated heterogeneous learning. According to the answer for Q1, temperatures are adjusted between training rounds. This coarse-grained adaptation relies on global training progress and fails to localize dynamic changes within a single round. In FL, due to clients' non-IID data, models across clients may exhibit significantly divergent optimization states within a single round. However, cross-round temperature updates[1][2][3] require synchronization in the next round, leading to slowed convergence and even suboptimal solutions caused by delayed adaptation.

W2: The novelty is not so sufficient

Our work is not a "simple" combination of multi-level distillation and dynamic temperature, as detailed in Q1 and W1. We clarify our contributions as follows:

1. Incorporating dynamic temperature distillation into heterogeneous FL, enhancing knowledge transfer across diverse models;
2. Designing a Z-score-based malicious node verification mechanism, effectively detecting and filtering abnormal logits to improve model robustness;
3. Providing a theoretical justification for ReT-FHD’s effectiveness from the perspective of information entropy and gradient updates, offering solid theoretical support for our method;
4. Extensive experiments on centralized FL (Tab.1 and Tab.3), decentralized FL (Tab.2), and blockchain FL (Tab.6, Tab.7, and Fig.3) validate our robustness and flexibility.

We respectfully request reconsideration of this core contribution.

W3: Blockchain deployment is not suitable as a main contribution

We do not clarify that blockchain deployment is our core contribution. In our contribution 3, we clarify that "our framework employs Z-score verification to validate logit distributions against dynamic boundaries, enabling automated reward/punishment protocols that deter malicious behaviors."We address its unique challenge in logits-sharing scenarios:

• Motivation: Traditional blockchain FL [4][5][6] validates via model parameters, but this fails for logits-based distillation.

• Contribution: Our Z-score validation (Eq.7) ensures compatibility without parameter exposure, achieving: 20.11% post-attack accuracy (Tab.7 vs. baseline 11.43%)

This mechanism directly enables our main contribution—Z-Score Guard: (Eq.7) for Standardized Finding of Malicious Logits.

References:

[1]Boosting Graph Neural Networks via Adaptive Knowledge Distillation

[2]Curriculum Temperature for Knowledge Distillation

[3]Dynamic Temperature Knowledge Distillation

[4]Robust blockchained federated learning with model validation and proof-of-stake inspired consensus

[5]Blockdfl: A blockchain-based fully decentralized peer-to-peer federated learning framework

[6]Bit-fl: Blockchain-enabled incentivized and secure federated learning framework

Please let me know if I have addressed your concerns and if not, we welcome any further questions you might have. Finally, thank you for your time and effort in reviewing our work.