FedWon: Triumphing Multi-domain Federated Learning Without Normalization

Thanks for your valuable comments. We hope the new version can adequately address your concerns.

---

Q1: Lack of Technical Novelty: The paper's technical innovation is limited to reparameterizing the convolution layers using the Scaled Weight Standardization technique. While this approach may have some benefits, it lacks novelty as the Scaled Weight Standardization technique has been proposed and utilized in previous studies.

A1: We would like to highlight that we are the first work that considers FL without normalization. Normalizations have been an important topic in FL to handle statistical heterogeneity. Although Scaled Weight Standardization has also been used in previous research, we believe our extensive empirical studies provide useful insights and lessons for the community on the intriguing properties of FL without normalization.

---

Q2: Insufficient Theoretical Analysis: The paper lacks a theoretical analysis of the effectiveness of the proposed Scaled Weight convolution (WSConv) layer. The authors should provide a more rigorous theoretical foundation to explain why the WSConv layer is suitable for addressing the challenges in federated multi-domain learning.

A2: We are also keen to provide more theoretical guarantees of our proposed FedWon. However, as the first work that considers FL without normalization, we have demonstrated multiple important properties of this direction through extensive experiments. We will provide more theoretical guarantees in the extension of the paper.

---

Q3: Limited Comparative Analysis: The paper does not sufficiently compare with other federated multi-domain methods, such as PartialFed and FMTDA. What are the advantages and characteristics of FedWon compared with these methods?

A3: Thank you for pointing out these two works. We have included them in the discussion in the paper. The table below compares our performance with PartialFed on the Office-Caltech-10 dataset. FedWon generally achieves better performance than PartialFed. Besides, PartialFed may be also not scalable to cross-device FL like FedBN as the clients are stateful and load the loading strategy from the previous round. In contrast, our proposed FedWon is versatile for both cross-silo FL and cross-device FL. The setting of FMTDA is different, FMTDA assumes source domain data in the server and aims to adapt to target domains in clients. Our proposed method, as well as FedBN and PartialFed, assumes there is no data in the server. Thus, it is not fair to compare the methods directly.

---

Q4: Limited Applicability to Convolutional Neural Network (CNN) Models: The proposed method is limited to CNN-based deep learning models, which restricts its applicability to a specific class of models, such as recurrent neural networks (RNNs), graph neural networks (GNNs) or transformer.

A4: Thank you for raising this interesting perspective. As the first work considering FL without normalization, we primarily focus on CNN models and discover many interesting properties of via extensive experiments, such as versatility to both cross-silo and cross-device FL and applicability to batch sizes as small as 1. We will further explore FL without normalization for the other networks mentioned by the reviewer, such as RNN, GNN, or transformers, in the extension of the paper.

Thanks for your time reviewing our paper and the thoughtful comments. Following your suggestions, we have run additional experiments and added the new results to the revised paper. We hope the new version can adequately address your concerns. We are very happy to run more experiments if you have further concerns.

---

Q1: The proposed method lacks innovation; it essentially directly applies the weight standardization and gradient clipping from the NF-Net series [1, 2] to the federated learning setting. It does not offer targeted improvements to address the unique challenges of the federated learning setting. [1] Brock, Andrew, Soham De, and Samuel L. Smith. "Characterizing signal propagation to close the performance gap in unnormalized ResNets." International Conference on Learning Representations. 2020. [2] Brock, Andy, et al. "High-performance large-scale image recognition without normalization." International Conference on Machine Learning. PMLR, 2021.

A1: We would like to clarify that the proposed method of FL without normalization is mainly motivated by the unique challenges of the multi-domain FL setting. As shown in Figure 1b, the BN statistics between clients and between a client and the server differ significantly, which hinders the performance of multi-domain FL. We are the first work that considers FL without normalization. We believe that this new perspective offers important insights for the community via extensive empirical studies and provides useful lessons for the community on the intriguing properties of FL without normalization.

---

Q2: The experiments for cross-device FL in the paper are not sufficient for the proposed method's effectiveness. The cross-device FL experiments only include a single dataset and 100 clients.

A2: Thank you for the comment. We would like to clarify that we actually provide experiments of cross-device FL on both Digits-Five and CIFAR-10 datasets. To further validate its effectiveness in cross-device FL, we further conduct the experiments of a total of 1000 clients, with a selection of only 0.1 clients per round, in the table below. FedWon also generally outperforms FedAvg under this setting.

---

Q3: In the upper image in Figure 1b, it seems that the green points representing the server are entirely invisible, while in the lower image, it appears that there is no information at all.

A3: Thank you for raising the concern. We have revised Figure 1b to show the variation of BN statistics between two clients to make it clearer. The full comparison of BN statistics between clients and between the client and the server is provided in Figure 12 in the Supplementary.

---

Q4: The presentation of feature maps in Figure 5 doesn't seem very informative. What information can we extract from the feature maps, and how are they related to the model's performance? Additionally, it appears that there is not much difference between the feature maps of FedAvg and FedBN.

A4: Feature maps are commonly used to offer interpretability. Clear and discernible features normally indicate that the model is learning meaningful representations from the data. FedWon has much better feature maps focusing on the object of interest compared to the other methods. As for FedBN and FedAvg, the feature maps between clients differ significantly, indicating large discrepancies in models learned in different domains. The feature maps of FedBN are generally slightly clearer than FedAvg, especially on the Client 1 local models.

Thanks for your valuable comments. We hope our response below can adequately address your concerns.

---

Q1: The novelty is quite limited, where the proposed method is to use an existing reparametrization trick (Brock et al. (2021a)) in the FL setting.

A1: We would like to highlight that we are the first work that considers FL without normalization. Normalizations have been an important topic in FL to handle statistical heterogeneity. Our new perspective offers important insights for the community. We believe our extensive empirical studies provide useful lessons for the community on the intriguing properties of FL without normalization.

---

Q2: Why the proposed method is only applied to the convolution layer? From Eq. 1, it seems that it can be applied to any kinds of layers represented by a weight matrix W.

A2: The equation outlines a general methodology applicable to various layers represented by weight matrices. We initially focus on convolution layers as they are commonly followed by a BN layer (e.g., ResNet and model architectures in Table 5 and 6 in the Supplementary). It has the potential to extend to other layers such as fully connected layers in scenarios where it comes with BNs.

---

Q3: In Eq. 1, how is γ chosen? In particular, how robust is the training result to the choice of γ?

A3: Thank you for raising this question. Eq. 1 is the standard BN formulation, so we suppose the reviewer is interested in our method's γ in Eq. 3. The value of γ is associated with the activation function used. For ReLUs, $γ = \sqrt{\frac{2}{1-(1-\pi)}}$.

---

Q4: Since the proposed method does not depend of batch statistics, I'm curious why FedWon B=1 better than B=2 (e.g., Figure 4)?

A4: Thank you for this interesting question. The results shown in the manuscript use the same learning rate of 0.01 for both batch sizes B = 1 and B = 2 of FedWon. To further analyze this result, we further tune the learning rate of B = 2 to 0.02 and they achieve similar performance, as shown in the table below. We have updated the paper to reflect these results.

---

Q5: Also, in figure 4, I'm curious to see how would FedAvg (B=1) perform.

A5: We also intend to report results for FedAvg with B = 1. However, methods such as FedAvg contains BN layers and cannot run with B = 1. This is because the statistics computed in BN (such as mean and variance) rely on batch-wise computations, and with a batch size of 1, there's no variance among samples, making normalization difficult.

---

Q6: It is claimed in this paper that FedBN can't do cross-device FL. However, there is not enough evidence, as far I can tell, from the paper that supporting this argument. Can the authors elaborate more on why FedBN can't do cross-device FL?

A6: FedBN stores BN statistics locally in clients and reloads these BN statistics after receiving the global model aggregated in the server. It assumes that the clients are stateful -- each client may participate in each round of the computation, carrying these BN statistics from round to round.

However, cross-device FL described in [1] involves a vast number of clients, with only a small subset participating in FL training per round. In cross-device FL, clients are mostly stateless -- each client will likely participate only once in the whole training.

When applying FedBN to cross-device FL, the randomly selected clients have a high possibility of not carrying the BN statistics from the previous round. Thus, FedBN is largely unsuitable for cross-device FL.

[1] Kairouz, Peter, et al. "Advances and open problems in federated learning." Foundations and Trends® in Machine Learning 14.1–2 (2021): 1-210.

We thank the reviewer for the positive feedback and address the detailed comments below.

---

Q1: The paper brought the Scaled Weight Standardization (SWS) technique to handle the multi-domain FL problem. However, there is less analysis about the SWS’s impacts to the FL process, e.g., will it lead to a better convergence bound?

A1: Thank you for raising the question. We provided empirical analysis on the convergence of the FL process in Figure 4 (right), Figure 6 (right), and Figure 10 (left). As the first work that considers FL without normalization, we will provide more theoretical analysis on the convergence bound in the extension of the paper.

---

Q2: Many methods are compared in the paper, some of them have BN, some of them are suitable for cross-silo FL only, and some of them are suitable for cross-device FL, it would be clearer to have a structured summarization to help understand the scenarios where these methods are suitable for.

A2: Thank you for this constructive comment. Below is the table to summarize and compare these methods. We have also added this table in Table 7 in the Supplementary.

---

Q3: Only one dataset is evaluated for the skewed label distribution problem.

A3: Thank you for raising the concern. As the paper primarily focuses on multi-domain FL, we conduct experiments on one dataset to demonstrate the adaptability to the skewed label distribution problem. We will conduct further analysis on more datasets in the extension of the paper.

---

Q4: Sec.4 claims that the FedWon will achieve competitive results even at a batch size of 1 while [https://arxiv.org/abs/1602.05629] shows the FedAvg will degenerate to the FedSGD and be less effective. It is an interesting topic, and do you mind to report more details about the learning curves (communication round v.s accuracy) at different batch sizes or on different datasets?

A4: Thank you for raising this interesting point. We revised the manuscript and provided the learning curves in Figure 14 in the Supplementary. Different batch sizes tend to have a similar trend of convergence.

---

Q5: What is the impact of local epochs on the proposed method?

A5: We provided analysis on the impact of local epochs in Table 14 in the Supplementary. Our proposed FedWon is robust on different local epochs and consistently outperforms FedBN under different settings.