Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning

Thank you for your valuable contributions to improving this paper. In response to your suggestions, please find our detailed replies below.

1. FedBN Baseline

FedBN is designed for personalised FL without obtaining a unified global model, as it keeps BN parameters client-specific, while focusing on the general FL scenarios, which train a global model for all the local clients. Therefore, we did not compare with FedBN in our experiments. However, we will include more detailed discussion of the techniques used in FedBN in our final version.

2. FedFN Terminology

We thank the reviewer for this valuable suggestion. We confirm that FedFN is implemented consistently with FN as described in our paper. To improve the accuracy of description, we will explicitly replace FN with FedFN in the revised version to avoid any potential ambiguity.

3. Hyperparameter Search

The use of inconsistent local update epochs would compromise the fairness of comparisons, as it primarily impact on the number of local updates. To mitigate this, we fixed the epoch value to 1 in our work. Similarly, the local batch size affects the communication frequency, which is why we haven't made adjustments for local update epochs. As you rightly observed, different algorithms exhibit distinct optimal learning rates across datasets (please refer to the anonymous link: https://anonymous.4open.science/r/ICML_2025-F29D/grid_search.png). We ensure all the methods could stably converge within the searched range. The empirical results indicate that larger learning rates prevent trainability under our small batch size constraints.

4. Combining with Classifier-Freezing Methods

In Table C, we compare the performance of the suggested three classifier-freezing approaches, integrated with our HBN, following the original experimental setup of MobileNet on CIFAR-100 (sharding partition strategies with 10 shards per client) [1]. In our implementation, we solely replace the standard BN layers in MobileNet with HBN. HBN achieves encouraging performance gains. HBN enhances activation normalisation during training by adaptively blending local and global statistics, thereby mitigating bias introduced by relying exclusively on local statistics. This hybrid approach ensures more stable and representative feature distributions across heterogeneous data partitions, which is particularly critical in classifier-freezing methods.

Table C: A comparison of classifier-freezing methods with BN and HBN.

[1] S. Kim et al., "FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning," TMLR, 2025.

We appreciate your thoughtful suggestions. Please find our response below.

1. FedTAN Baseline

FedTAN employs real-time communication to synchronise the use of shared global statistics. However, to obtain these global statistics, FedTAN requires three rounds of communication per BN layer during the forward propagation. This requirement fails to accommodate a fundamental FL constraint: communication rounds restrictions. While the theoretical discussion of the impact of BN in FL provides valuable insights, we have intentionally excluded FedTAN from our comparative baselines because of these limitations. We acknowledge its conceptual contributions while noting its impractical communication overhead in real-world FL applications. Our method adds only one round overhead compared to the original BN, making it more suitable for practical scenarios.

2. Analysis of Results

FixBN and FBN employ shared global statistics for normalisation at different stages. When these global statistics are reliable, they benefit local training. Conversely, when the global statistics are unreliable, they exhibit detrimental effects. Our experimental scenario in Table 1 is deliberately challenging——featuring strong heterogeneity, numerous clients, and restricted batchsize. In such conditions, FixBN and FBN fail to judge whether the local statistics, which are derived from diverse local models, can directly be aggregated, resulting in unreliable global statistics. In addition, FN is a local normalisation method that normalises features using the statistics of a single sample. In non-iid scenarios, FN, like GN and LN, lacks awareness of the global structure, leading to performance degradation in our setup.

3. Weight Standardisation Methods

We further analysed the weight standardisation solution, Fedwon, which achieves promising results. Fedwon modifies the convolution layers directly to adjust the distribution, without using the standard normalisation layer. Therefore, the starting point of Fedwon is different from ours. What is exciting is that when we combine HBN with Fedwon, as shown in Table B, Fedwon+HBN yields impressive performance improvements, demonstrating the complementary power of our design towards Fedwon. In our work, we want to emphasise the supportive role of BN's global statistical information in FL. However, inappropriate implementations of BN in FL fail to fully activate its potential. Our key contribution is resolving the BN's dilemma in federated learning. Through analysis and empirical validation, HBN proves both simple and effective.

Due to the lack of available open-source code for [1], we are unable to replicate it within a limited time. We will provide a comprehensive analysis of weight standardisation methods in the final version.

Table B: Compatibility with weight standardisation methods.

[1] Siomos, Vasilis, et al. "Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration." arXiv, 2024.

4. Why do we need hybrid batch normalisation?

If we rely solely on local statistics, this reduces to the standard BN. The standard BN degrades performance in non-iid scenarios, as the local statistics are diverse among different clients. Consider a toy example: two clusters following Gaussian distributions, simulating data from two clients in FL. Normalising each cluster separately using local statistics would cause their distributions to overlap (please refer to the link: https://anonymous.4open.science/r/ICML_2025-F29D/adaptive_normalisation.png). In contrast, global normalisation preserves the global structure of two clusters. However, real-time global normalisation is impractical, as discussed above about FedTAN. Using only historical global statistics ,whose timeliness is constrained by intermittent communication, will yield a suboptimal result. But historical global statistics still retain valuable global structural information. Inspired by that, our proposed HBN adaptively combines historical global statistics with current local statistics, achieving more effective normalisation.

5. Amelioration of Figure 2

We replace Figure 2 with an intuitive toy example (please refer to the link: https://anonymous.4open.science/r/ICML_2025-F29D/adaptive_normalisation.png), where hybrid normalisation can standardise the size of two clusters while maintaining the global structure.

6. Technical Details

EMA is also used in FBN, excluding other methods.

Thank you for your constructive comments and suggestions. Please find our response below.

1. Client Number Experiment

We conducted comparative experiments on CIFAR-10 ($\beta$ = 0.6) across varying client numbers (10 clients sampled per round). As shown in Table A, HBN consistently outperforms baselines at all scales (from K=100 to K=1000).

Table A: Experiments on CIFAR-10 (β = 0.6) across varying client numbers.

2. Hyperparameter Sensitivity

We apologise for any confusion caused. We clarify that $\alpha$, $\beta$, and $\gamma$ are learnable parameters (not hyperparameters). $\alpha$ is an adaptive hybrid distribution factor, while $\beta$ and $\gamma$ are learnable affine transformations. In this work, they are initialised as $\alpha=0$, $\beta=0$, and $\gamma=1$ respectively.

3. Algorithm Typo

Thank you for this observation. What we want to express is that updating statistical parameters does not require backpropagation to calculate gradients, which can save computational costs. For clarity, we will modify the algorithm annotation to 'without backpropagation' and 'using backpropagation' respectively in the final version.

4. Details of Model Architecture and Symbol Explanations

Regarding the model architecture of Simple-CNN, we provide its details in Appendix B.1. $\phi$ is the Dirichlet distribution factor that controls the degree of label heterogeneity [1] (visualisation is provided in Appendix B.2), while $B$ is the batch size.

[1] Hsu, Tzu-Ming Harry, Hang Qi, and Matthew Brown. "Measuring the effects of non-identical data distribution for federated visual classification." arXiv preprint arXiv:1909.06335 (2019).