FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things

The methodology can be elaborated for better clarity of the overall research step.

Our work is a benchmark paper, and we submitted this work to the benchmark track of ICLR. We reviewed several published existing federated learning benchmarks [1-5] and organized the method section by following those works. We will elaborate more on the method section to make it clearer in the revision.

[1] Ogier du Terrail, Jean, et al. "FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings." Advances in Neural Information Processing Systems 35 (2022): 5315-5334.

[2] Feng, Tiantian, et al. "FedMultimodal: A Benchmark For Multimodal Federated Learning." ACM SIGKDD 2023

[3] Chen, Daoyuan, et al. "pFL-bench: A comprehensive benchmark for personalized federated learning." Advances in Neural Information Processing Systems 35 (2022): 9344-9360.

[4] Zhang, Tuo, et al. "Fedaudio: A federated learning benchmark for audio tasks." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.

[5] Lin, Bill Yuchen et al. "FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks". In Findings of the Association for Computational Linguistics: NAACL 2022, pages 157–175, Seattle, United States. Association for Computational Linguistics.

Figure 1 is not explicitly referred to in the manuscript

Thanks for pointing this out, we will reference Figure 1 in the introduction.

The author(s) can consider elaborating the methodology of how to collect and choose the datasets. What are the metrics to select and finalize the eight datasets?

The most important metric is quality. We started with a much larger pool of existing datasets and have made significant efforts to validate those datasets. However, many of the datasets in the pool are low-quality; including those low-quality datasets would waste the time of other researchers and practitioners in the community. Thus, we only included the high-quality ones. Other metrics we used to select and finalize the datasets include types of IoT devices, sensor modalities, applications, and size of the datasets. As shown in Table 1 in the paper and noted in Section 3.1, we included datasets that cover a wide range of IoT devices, sensor modalities, and applications. Moreover, our selected datasets have very diverse sizes (small, medium, large). The rationale behind this design choice is to accommodate researchers who have different computing resources. For example, researchers who have very limited computing resources can still pick the relatively small datasets included in our benchmark to develop and evaluate their algorithms.

The author(s) can specify the definition of small, medium and large datasets.

Our categorization is based on the number of training samples included in each dataset. Table 2 lists the number of training samples included in each dataset. As shown, we have 2 small datasets: UT-HAR, VisDrone, each has less than 7k samples; 5 medium datasets: AEP, CASAS, WISDM-P, WISDM-W, Widar, each has 11k-16k samples, and one large dataset: Epic Sounds which has more than 60k samples. We realize that we did not make it clear in our submission, and we will make this point clear in our revised version.

Dear Reviewer,

As the rebuttal-discussion period ends today, we would like to know if our responses address your concerns. Feel free to let us know if you have any further questions. Thanks as always.

Dear Reviewer,

We have added our responses to the revision version.

Dear Reviewer,

Most parts of the review are very generic and more importantly, are not relevant to our work.

We are very confused and do not know how to respond.

Why not leave the sounds in raw format instead of converting to the frequency domain with particular parameters?

The common practice of state-of-the-art audio processing techniques such as audio spectrogram transformer [1], CLAP [2], and whisper [3] is to first convert the raw audio data into frequency domain which is then imported into the neural networks. We followed this common practice in our work.

[1] Gong, Yuan, Yu-An Chung, and James Glass. "AST: Audio spectrogram transformer." arXiv preprint arXiv:2104.01778 (2021).

[2] Elizalde, Benjamin, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. "Clap learning audio concepts from natural language supervision." In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.

[3] Radford, Alec, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. "Robust speech recognition via large-scale weak supervision." In International Conference on Machine Learning, pp. 28492-28518. PMLR, 2023.

What is the purpose of Section 4? To demonstrate that the benchmarks can be used to compare optimizers? Enabling comparison doesn't imply enabling comparison yielding correct ranking of optimizers.

The purpose of Section 4 is to provide benchmark results on the eight IoT datasets as baselines for the research community. If someone is interested in designing a new federated learning algorithm for IoT data, he or she can evaluate his/her algorithm on our datasets and compare the results of his/her algorithm to our benchmark results as baselines.

The authors claim this is the first IoT FL benchmark but a Google Scholar search turns up "FLBench: A Benchmark Suite for Federated Learning" by Yuan Liang, Yange Guo, Yanxia Gong, Chunjie Luo, Jianfeng Zhan, and Yunyou Huang, which includes an AIoT benchmark domain. It isn't clear whether this is competing work, or work by the authors of the submitted paper. In either case, it seems highly relevant but does not appear to be cited.

Thank you for pointing this out. There are two key differences between FLBench and FedML and our proposed FedAIoT. First, FLBench and FedML are framework papers. Although they claimed that they support IoT data, none of them provide benchmark results on real IoT datasets. In contrast, our proposed FedAIoT includes eight well-selected high-quality datasets and provides benchmark results and analysis on those eight IoT datasets. Second, compared to the frameworks proposed in FLBench and FedML, the framework proposed in FedAIoT is specifically designed for IoT: it includes several modules such as IoT-specific data preprocessing, IoT-friendly models, and IoT-factor emulator. All of these modules are not included in FLBench and FedML. We realize that we did not make it clear in our submission. We will cite both FLBench and FedML and clarify the differences in our revised version.

Does the similarity matrix used for noisy labeling depend on the particular centralized learning approach? If so, does that mean that centralized training and evaluation must be redone to enable noisy labeling whenever an algorithm changes? Or is there something fundamental about the confusion matrix, i.e., is it unlikely to change much when models change?

Correct. In theory, the confusion matrix used for noisy labeling depends on the particular centralized learning approach. However, the changes between different models are minimal.

The modifications to the curated datasets are not well justified. They are reasonable, but explicit justification or basing the approach on well justified approaches from prior work would be best.

The only curated dataset we modified is Widar. In the original Widar data, the same data from one subject was used in both training and test sets. We repartitioned the training and test datasets so that they do not have any overlap to prevent training data from leaking to the test set.

Dear Reviewer,

As the rebuttal-discussion period ends today, we would like to know if our responses address your concerns. Feel free to let us know if you have any further questions. Thanks as always.

Although the paper does well in introducing a new benchmarking framework for FL for IoT, it still largely builds upon curating from existing datasets introduced by prior works.

We want to first clarify that we took serious consideration on the choice between collecting our own datasets and selecting existing ones. Our goal is to provide a benchmark that consists of many high-quality and well-validated datasets collected across a wide range of IoT devices, sensor modalities, and applications. We find that collecting datasets by ourselves is not an efficient way to achieve this goal. In contrast, there is a plethora of publicly available datasets that were already collected by different groups in the research community. Including an existing dataset that has been validated and widely used by the community is more trustworthy than a self-collected dataset that has not been validated by the community. The trustworthiness of a dataset is particularly important for benchmarking purposes. Therefore, we decided to build our benchmark based on the existing datasets.

We want to further emphasize that our choice of selecting existing datasets is not an easy task. This is because many existing datasets are not high-quality to be included as benchmarking datasets. In fact, we made significant efforts and invested time to validate a much larger pool of existing datasets and identified the high-quality ones to include in our benchmark.

We would also like to note that, to the best of our knowledge, all of the existing FL benchmarks (e.g., Flamby [1], FedCV [2], Flute [3], FedNLP [4], FedAudio [5], FedMultimodal [6]) use existing datasets that are well-validated and widely used by the community for their specific domains (e.g., medical, computer vision, natural language, audio, multimodal, etc.) other than collecting new datasets.

[1] Ogier du Terrail, Jean, et al. "FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings." Advances in Neural Information Processing Systems 35 (2022): 5315-5334.

[2] He, Chaoyang, et al. "FedCV: a federated learning framework for diverse computer vision tasks." arXiv preprint arXiv:2111.11066 (2021).

[3] Hipolito Garcia, Mirian, et al. "Flute: A scalable, extensible framework for high-performance federated learning simulations." arXiv e-prints (2022): arXiv-2203.

[4] Lee, Jean, et al. "Fednlp: an interpretable nlp system to decode federal reserve communications." Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021.

[5] Zhang, Tuo, et al. "Fedaudio: A federated learning benchmark for audio tasks." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2023.

[6] Feng, Tiantian, et al. "FedMultimodal: A Benchmark For Multimodal Federated Learning." ACM KDD (2023).

The introduced end-to-end FL framework also seems to be a collection of standard machine learning and FL ideas such as non-IID data partitioning, normalization, etc. The novel contributions of addressing noisy labels (non-uniform addition of noise) and quantized training at the client side seem limited.

Our work is a benchmark paper, and we submitted this work to the benchmark track of ICLR. Different from works that focus on proposing new techniques, we followed works that focus on federated learning benchmarks published at top-tier venues [1-5] to frame our contributions. Therefore, we believe that a comparison of our work with other federated learning benchmark studies would be a fair approach.

Specifically, compared to other FL benchmark studies referenced previously [1-5], our work has three key novel contributions:

1. Our benchmark represents the first FL benchmark that focuses on data collected from IoT devices. We have put in a lot of work to explore, validate, and include unique IoT-specific data modalities that previous benchmarks do not include.

2. Some IoT data modalities such as wireless signals are unique. For non-IID data partitioning and normalization, we include new techniques that are designed for those unique IoT data modalities that other benchmarks do not include.

3. The proposed noisy labels (non-uniform addition of noise) and quantized training at the client side, though seemingly incremental, are really important to IoT data. To be honest, we have been striving to make meaningful and useful contributions instead of making our approach look fancy. We believe these proposed techniques can make meaningful contributions to our community. We realize that we did not make it clear in our submission, and we will make this point clear in our revised version.

Dear Reviewer,

As the rebuttal-discussion period ends today, we would like to know if our responses address your concerns. Feel free to let us know if you have any further questions. Thanks as always.