Low Precision Local Training is Enough for Federated Learning

Q1: The biggest concern is regarding the novelty w.r.t. SWALP [40].

A1: We summarize the differences between our method and SWALP as follows:

1. SWALP is designed for standard sgd in centralized training, our approach focuses on FL.
2. We show both empirically and theoretically the effecitivenss of our approach in heterogeneous datasets.
3. We show that besides FedAVG, our method can be integrated with various FL algorithms.

Q2: Typos, e.g., "sever" in Line 108 and missing a space before "i.e.".

A2: Thanks. We will correct the typos and improve the writing accordingly.

Q3: Eq (5) is different from its counterpart in [40] where the power was F-1 but now W-F-1. Please explain the reason of difference.

A3: The reason for this issue is that in Section 3.1 of SWALP, the definitions of W and F are different between the "Fixed Point Quantization" section and the "Block Floating Point (BFP) Quantization" section. Our definition of W and F is the same as in the "Fixed Point Quantization" section. In our paper and "Fixed Point Quantization" section, F represents the number of bits occupied by the fractional part, while in the "Block Floating Point (BFP) Quantization" section, F stands for the number of bits occupied by the shared exponent.

Q4: Eq(7) uses E which is not clear until continuing reading to Line 161 and Algorithm 2 Lines 3-4.

A4: We provided the definition of E in line 115 of the paper.

Q5: Algorithm 2 Line 11, t' is only briefly mentioned in Line 161 without even referring to the used lines.

A5: Thank you for your suggestion, we will add further description after t'.

Q6: It would be good to estimate the time reduction with the professional hardware (real acceleration).

A6: It is difficult to conduct experiments on professional hardware in university, which is expensive and involves hardware programming. Fortunately, we note that our simulation is standard and approved by the community. The reasons are

1. Such low precision training methods can be implemented on the machine learning accelerators efficiently to achieve real saving in computational and communication cost;
2. The results would be always consistent with those obtained from simulation.

For details, please refer to [r9] and [r10].

[r9] Kuzmin A, et al. Fp8 Quantization: The Power of the Exponent. NeurIPS 2022.

[r10] Nagel M, et al. Overcoming Oscillations in Quantization-Aware Training. ICML 2022.

Dear Reviewer 1d4C,

We sincerely appreciate your support and valuable comments on improving our paper. We believe the primary similarity between SWALP and our method is their high-level inspiration from Kolmogorov’s law, which asserts that the sample average can almost surely converge to the expected value despite the presence of noise. We also find it highly appropriate to develop efficient low-precision local training methods for federated learning, given that the clients in many federated learning applications are resource-constrained. Our proposed method is both simple and effective, presenting a new framework/paradigm for accelerating federated learning. We hope it will inspire researchers to create more efficient federated learning algorithms based on low-precision local training in the future. We believe this contribution holds comparable significance to our proposed method itself.

As the deadline for the author-reviewer discussion phase approaches, could you please let us know if any further discussion or clarification is needed? Thank you again for your valuable time; your support is greatly appreciated.

Sincerely, Authors of Submission 11555

Q1: Contribution in terms of test accuracy.

A1: It is a misunderstanding. We do not aim to improve test accuracy, but rather to demonstrate that low precision local training is sufficient for FL and can be used to reduce training and communication cost. Our method, which performs low precision local training, achieves comparable (or even higher) accuracy with full-precision methods. The improvements on the training efficiency and communication cost can be guaranteed as our low precision operators are standard.

Q2: Compare with FedPAQ, Quantized-SGD and dynamic-model-averaging.

A2: Thanks. In our paper, we gave the results with strong baselines HeteroFL [9] and SplitMix [14]. Below is the result on CIFAR10 ($\alpha=0.01$) of FedPAQ. "()" denotes the percentage of models on the clients. Following [r6] we use the number of weights, activation, and gradients of local training to approximate training cost. It shows that we can reduce communication and computational cost while maintain high accuracy.

We gave the results of another SOTA CoCoFL in A2 of Reviewer 5cLQ. We have not yet found the code of Quantized-SGD and dynamic-model-averaging, due to the time limitation, we postpone their comparison to revision.

Q3: Typos.

A3: Thanks. We will improve the writing accordingly.

Q4: This result does not clearly show low precision local training improves performance, since it is combined with moving average.

A4: Low-precision training typically tends to decrease instead of improve the accuracy. We introduce it to reduce communication and computational costs, while moving average is adopted to compensate the above accuracy degradation. This point was confirmed in Table 1 and Figure 2. That is that with moving average, the accuracy of the low-precision model can be comparable to full-precision model, and sometimes even better due to its effect on reducing over-fitting risk.

Q5: Unclear use-case.

A5: Reducing training and communication cost is an important topic in FL since the clients in FL are always resource-constrained edge devices with low-bandwidth communication network. We address this challenge by using low precision local training, which can be implemented efficiently on machine learning accelerators. We note that using accelerators with low precision operators is an new paradigm to accelerate AI models. For details, please refer to the surveys [r7] and [r8].

Q6: Discuss FL with quantization literature.

A6: Thanks. We gave comparison results with SOTA CoCoFL and FedPAQ with quantization in the table in A2 of Reviewer vykk and the table in A2 of reviewer 5cLQ. We will include them and discuss in details in the revision if accepted.

Q7: How quantization interacts with FedBN.

A7: Thanks. The results ($\alpha = 0.01$ ) given below show that our method works well with FedBN. More results will be included in the revision.

Q8: Benefits stem from quantization and moving average.

A8: We would like to clarify two things:

1. Quantization is used to reduce the computational and communication cost and typically it tends to produce negative instead of positive impact on accuracy and we adopt moving average to alleviate this impact.
2. In some tasks, quantization can reduce the risk of over-fitting leading to higher test accuracy. It is verified by Table 1 and Fig. 2 in our paper.

Q9: Does moving average have a large positive effect regardless of quantization?

A9: Moving average is used to reduce the error introduced by quantization. When full precision local training is adopted, its effect is relatively limited compared with that of low-precision, see Table 1 in our paper.

Q10: More limitation discussion.

A10: Thanks. For computation and communication, we follow the standard settings in FL. We gave a series of additional experimental results in this rebuttal and we will give more discussion in the revision.

[r6] Efficient personalized federated learning via sparse model-adaptation. ICML 2023.

[r7] AI and ML Accelerator Survey and Trends. IEEE HPEC, 2022.

[r8] Advances and Open Problems in Federated Learning. Foundations and Trends in Machine Learning, 2021.

Q1: What about resource heterogeneity? It would be important to have a discussion (or better some experimental results) on how the proposed solution perform in such setting. Should we use the same quantization level for all clients, or can we adjust the precision according to the resource availability?

A1: Thanks for your suggestion. We give the results in the setting of HeteroFL [9]. That is, we set three types of resource devices: 1, 1/2, and 1/4, which respectively represent the devices that can train the entire network, 1/2 network, and 1/4 network. We use (x, x, x) to represent the corresponding bits in three type of devices in training. We let $\alpha=0.16$. The experimental results are as follows:

The results above show that our low-precision training mechanism works well in resource heterogeneity applications. They also indicate that when the sum of bit widths used across the three sets of devices is fixed, devices with poorer computational resources should be allocated a smaller number of bit widths.

Q2: How does these SOTA techniques FedQNN and CoCoFL perform compared with the proposed approach?

A2: Thanks for your suggestion. Since the code of FedQNN has not been released, we give the comparison results with CoCoFL. We follow the setting of CoCoFL and conduct experiments on on Shakespeare and CIFAR10. Following [r3] we calculate the communication cost by counting the size of the model transmitted from the client to the server in each round, and the training cost by counting the weights, activation, and gradients of local training.

The results above show that our method has a significant reduction in overhead, because our local training is completely low-precision, and the model transmitted to the server is also low-precision. CoCoFL still trains the unfrozen parameters with full precision, and the transmission is also full-precision, so it will have a relatively larger overhead.

Q3: Appendix F clarifies that the fake quantization is applied in the experiments. I really appreciate learning this information, as one of my question was how the quantization is implemented in PyTorch.

A3: We adopt the widely used simulation method in the studies [40, r4, r5] of low-precision training, which is approved by the community as its result is always consistent with that on the real ML accelerators. We have submitted our code in the supplementary materials, which includes the specific implementation of fake quantization. Our approach to quantization involves implementing the quantization of weights, activation values, and gradients during the training process. We have three quantization operations: the first is to implement the quantization of activations after every model module, the second is to quantize the gradients of the parameters after the loss is generated and the gradients are backpropagated, the third is to quantize the weights after the optimizer is updated. The specific implementation process can be referred to in our code.

Q4: If performing lower precision training at clients improves or worsen these privacy and security aspects.

A4: We tested different levels of label flip attack on FedAVG and ABAVG [38] by using MNIST, and the results are as follows. ABAVG maintains a validation dataset on the server side and adjusts the weight of the client aggregation according to the client's accuracy, so it can be used for adversarial attacks. We let $\alpha=0.1$. Our experimental results show that low precision training can slightly improve the performance. The reason could be that low precision training prevents the client from over-fitting the wrong labeled samples, and thus reduces the impacts of the poisonous data. More results will be included in the revision.

[r3] Chen D, et al. Efficient personalized federated learning via sparse model-adaptation. ICML 2023.

[r4] Kuzmin, et al. The Power of the Exponent. NeurIPS 2022.

[r5] Nagel, et al. Overcoming Oscillations in Quantization-Aware Training. ICML 2022.

Q1: The integration of low precision training and high precision aggregation may add complexity to the implementation.

A1: Actually, in our method, the transformation between low and high-precision parameters is performed on the server, and the computation in the clients is standard for the clients supporting low precision operators. As we know, in FL, the server always has rich computation and memory resources and the main challenge in improving the training efficiency comes from the resource constrained clients and the communication cost. Therefore, the increased implementation complexity above would not prevent us from improving the overall training efficiency.

Q2: The performance improvements are partly dependent on the hardware capabilities, such as the availability of processors supporting low precision operations.

A2: Yes. But we would like to point out some promising progress in this area. In recent years, deep learning applications urge the development of machine learning accelerators. Lots of accelerators supporting low precision operations have been announced, which make it easy to implement the low precision training/inference algorithms including our approach efficiently in practice. For details, please refer to the survey [r1].

Q3: Only image datasets are considered. Evaluation on other types of data, like text or tabular, can strengthen the results.

A3: Thanks for your suggestion. Actually, we have conducted experiments on most of the datasets used in existing studies. Below, we give the results on the text dataset IMDB and Shakespeare. The results also confirm our conclusion on text data that low-precision local training is sufficient for Federated Learning. Following [r2], we approximate the communication cost by counting the size of the model transmitted from the client to the server in each round, and the training cost by counting the weights, activation, and gradients of local training.

Q4: No integration with differential privacy or other privacy protection mechanisms. Federated Learning itself is not private and it would be interesting to see what privacy mechanisms are suitable for low precision model updates.

A4: We think that theoretically proving what privacy mechanisms are suitable for low precision model updates is too complicated to be done in the author response period due to the variety of privacy mechanisms. Below, we give the results of FL with differential privacy and low-precision local updates. It shows that our method works well with differential privacy. The reason could be that as our quantization process is stochastic, it can be approximately viewed as a differential privacy protection mechanism and therefore they are compatible with each other.

Moreover, we agree with you that theoretically FL is not private. We give the results with standard FL in the manuscript because FL is still one of the main machine learning paradigms to address the challenges on preserving data privacy. The reason could be that as indicated in Figure 4 of [29] recovering the data with details from the received local models is highly nontrivial.

Q5: How can we determine the optimal precision level for local training without extensive hyperparameter tuning, which is expensive in FL?

A5: Our empirical results show that 8-bit precision level works well on all the tasks. In practice, one can tune the precision level during training, e.g., choose 2 sets of clients with different precision levels in some rounds and choose proper precision level based on the performance of the two aggregated models.

Q6: Have you explored mixed precision strategy? E.g. using different quantization schema for gradients, activations etc.

A6: Thanks for your suggestion. We performed local mixed precision training on CIFAR10 with $\alpha=0.01$. We use the triple (x, x, x) to represent the number of bits are used in quantizing weights, activations, and gradients, respectively. The results are as follows:

The experimental results indicate that our method also supports mixed-precision. Results with more mixed precision training methods will be included in the revision if accepted.

[r1] Reuther, et al. AI and ML Accelerator Survey and Trends. IEEE HPEC 2022.

[r2] Chen D, et al. Efficient personalized federated learning via sparse model-adaptation. ICML 2023.