FedConv: Enhancing Convolutional Neural Networks for Handling Data Heterogeneity in Federated Learning

The self-attention free CNN is also very similar to ConvMixer and this is not a new observation. Can you highlight how FedConv is any different and how the involved components are discovered/developed? The motivation is largely lacking and it looks like these component are a result of running a large-scale hyperparameter optimization?

Number of clients, i.e., 12, are too low to make any strong claims about data heterogeneity. Why not systematically vary the number of clients from 10 to 200?

No comparison with existing approaches. It looks like this paper is a first attempt at studying data heterogeneity in federated setting.

1. This paper primarily relies on empirical insights and lacks a comprehensive theoretical analysis. It is difficult to understand the underlying principles that drive the efficacy of this method. (see Q1)

2. The main content of this paper is centered around the examination of large models like ConvNeXt and Swin ViT. The potential applicability of the proposed methods to models of varying sizes remains uncertain, which holds significant importance given the large body of FL for training models on edge devices, such as mobile phones.

3. Some of the implementation details are missing. (see Q2, Q3)

1. How the authors are motivated to explore and lead to the findings can be further clearly explained. For example, there are many model configurations for a specific CNN model, why activation function and normalization layers are selected as key components to explore can be further justified.

2. The number of parameters in CNN and ViT differs a lot, which can lead to different representation abilities of CNN and ViT. This aspect should be taken into consideration when comparing these two architectures.

Proposed principle – removing all normalization layers and normalization-free setup:

The paper conducts an experiment to show the effectiveness of removing all normalization layers only on the neural architecture view and lacks consideration for other factors.

It's important to note that the aim of normalization technique is to create faster and more stable training, while improving generalization (i.e generalization performance of the network on unseen data.)

The normalization layer is highly "model size and dataset" aware; for instance, the larger the dataset, the superior results with the normalization you will get. In your experiments, for instance, the normalization technique in ResNet is targeted to a larger dataset i.e ImageNet. So instead of simply keeping and removing normalization, you should also consider the dataset and mode size. Since FL is a complex system, some clients may run large-size models, and simple removal of normalization from a large model might easily go to over-fittings.

In short, more comprehensive factors need to be discussed.

Proposed principle – Reducing activation layers:

The effect of reducing activation is similar to adjusting the size of Deep Neural Networks ( reducing the depth of DNNs and increasing the width of the DNN).

For instance, let's consider 2 hidden layers in DNNs: layer 1 with weights W1, and layer 2 with weights W2. The calcuation with activation $\sigma$: $Y =\sigma(W_2 \sigma(W_1 X))$

if we remove the activation between layer 1 and 2, the computation between layer 1 and 2 will become: $Y = \sigma(W_2 W_1 X)$ According to the associative property of matrix multiplication, we can have $Y = \sigma((W_2 W_1) X)$, and let $W_3 =W_2 W_1$, then we have $Y = \sigma(W_3 X)$. Hence, it's similar to replacing layer 1 and layer 2 with a new layer with learnable weights $W_3$

Again, in this paper's case, the effect of removing activation layers is essentially we modify the DNNs' size (the depth and width), and the size of the DNNs is a complex hyper-parameter tuning and AutoML topic, it depends on many factors and no universal answers.

Proposed principle – Kernel Size: The paper [1] also has a similar conclusion, they use large kernel design 31*31, but not in FL settings, and achieve competitive performance with ViT. However, selecting the right kernel size raises trade-offs among model size, computational complexity, memory usage, etc. It's not always universal for the larger kernel size the better, it depends on specific questions and tasks.

The interpretability of neural architecture in FL shouldn't separate from existing research:

It's very hard to separate the interpretability of the neural architecture in FL from existing research, again, FL systems are complex and dynamic, and it's challenging to find a universal guideline for all cases. Even the authors in this paper tried to limit their domain to heterogeneous data (Non-IID) FL, the experiment settings are primarily focused on neural architecture view. and lack the ablations on different data-heterogeneous settings. I list some of the references for investigating the effect and interpretability of neural architecture components for general model optimization guidelines. [1, 2, 3, 4]

Unfair design of experiments:

Since FL challenges traditional centralized training with its heterogeneous data, simply active and removing some components of DNNs is unfair for investigating the effect in FL. Different non-IID settings need to be discussed. For instance, the normalization designed for overfittings and generalizability, and the overfitting and generalizability should also be discussed, even in an FL setup.

Reference

[1]Ding, Xiaohan, et al. "Scaling up your kernels to 31x31: Revisiting large kernel design in cnns." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[2] Ş. Öztürk, U. Özkaya, B. Akdemir and L. Seyfi, "Convolution Kernel Size Effect on Convolutional Neural Network in Histopathological Image Processing Applications," 2018 International Symposium on Fundamentals of Electrical Engineering (ISFEE), Bucharest, Romania, 2018, pp. 1-5, doi: 10.1109/ISFEE.2018.8742484.

[3] Yu, Jiahui, and Konstantinos Spiliopoulos. "Normalization effects on deep neural networks." arXiv preprint arXiv:2209.01018 (2022).

[4]Liu, Sheng, et al. "Convolutional normalization: Improving deep convolutional network robustness and training." Advances in neural information processing systems 34 (2021): 28919-28928.