MultiIoT: Towards Large-scale Multisensory Learning for the Internet of Things
MultiIoT is the largest ML for IoT benchmark to date, bringing unique challenges in multisensory modeling, temporal interactions, and heterogeneous sensors to solve tasks of real-world practical impact.
摘要
评审与讨论
This paper proposes MultiIOT which includes over 1.15 million samples from 12 modalities and 8 tasks. This paper summarizes the recent developments and key challenges in the field. Then, the authors benchmark the different model architectures for processing multi-modal sensory signals and propose some insights.
优点
- I like the author's efforts in incorporating more modalities in understanding the scenes and human behaviors. This work is in general well-motivated and I believe this work would be interesting for future ML research from an application standpoint.
- Discussions on current situations in the field and outstanding challenges are well-written and easy to follow.
缺点
The main weaknesses of this work are the technical contribution and experimental evaluation.
- For dataset and benchmark, in Sec. 2.2 the authors claim 'We collected diverse data from IoT devices, such as Inertial Measurement Units (IMU), Thermal sensors, Global Positioning Systems (GPS), capacitance, depth, gaze, and pose.' However, from my understanding, it consists of solely existing datasets while most of them contain only several modalities.
- The experiments section contains no quantitative comparison with existing methods. There are other methods proposed for these individual tasks, and it would be difficult to evaluate the performance of the evaluated model variants without comparing them with the existing baselines.
问题
- Can the authors discuss if there is extra effort in consolidating the different datasets? e.g. how to unify the data format and make them really 'one' benchmark and convenient for the research community to benchmark their algorithms on all the tasks easily.
- What are the implementation details for each task? In Sec. B there is some brief explanation like 'Network Architecture: Distinct neural architectures optimized for each modality type, such as CNNs for images and RNNs for sequential data', but it is not enough. More experimental details are needed to understand and replicate the experiments.
Minor Issues:
- No qualitative results were provided. Authors could consider including data points and visualizations for the dataset, benchmark, and method.
- Fig. 3 is of low visual quality. Authors could design more and better charts to illustrate the model comparisons.
This paper provides an extensive benchmark, MultiIoT, for machine learning of IoT applications, that contains a large amount of data samples from 12 modalities and 8 different downstream tasks. Experiments are provided to compare the performance of machine learning models trained on different learning objectives and sensory modalities on each task. The conclusion was made that multi-modal and multi-task learning is beneficial in learning useful semantics from each modality.
优点
-
The coverage of the sensory modalities and downstream tasks in the paper is fairly comprehensive.
-
Some of the observations made in the paper are interesting and can motivate future research in the IoT domain. For example, the authors found that the interaction of different tasks can facilitate single-task performance
缺点
-
As a benchmark, the authors did not provide a new dataset with comprehensive modality and task coverage that can be used for general IoT machine-learning models. Instead, the datasets evaluated in the benchmark all come from public resources, which only contain a subset of sensory modalities. For this reason, I feel it is actually an overclaim to address that the benchmark consists of over 1.15M samples, which comes from the sum of different datasets.
-
The paper lacks a thorough comparison of how different DNN architectures, e.g., CNN, RNN, and Transformer, differ in processing the IoT sensing tasks, which in my opinion, is also an important perspective in such a benchmark.
-
As an important perspective of IoT applications, the benchmark did not mention any efficiency results or considerations.
问题
-
What is the main difference between the "adapter models" and "unimodal multi-task models"? Do they only differ in the training paradigm, where the adapter models use self-supervised pretraining, while the multi-task models simultaneously optimize for multiple downstream tasks? In my opinion, they are similar because they both utilize a shared encoder to extract the general semantics of a single sensory modality signal.
-
In section 4.2, what are the scales, w.r.t the number of parameters, of each compared model? Do you guarantee that the comparison between different models is fair, by avoiding comparing the performance between models with significantly different scales?
This paper claims to present a large multi modality benchmark for Internet-of-things (IoT). There are data present from 12 modalities and 8 tasks are defined to be solved with models trained with this data. The paper further evaluates various different types of architectures to asses how best to combine the various modalities to attain the best accuracy for the various tasks. The motivation for proposing this dataset is because of the claimed need to address various challenges with multimodal data IoT data including "High-modality multimodal learning", "Temporal interactions","Heterogeneity" and "Real-time". Overall the authors find that multi-modality multi-task networks result in the best accuracy on the tasks.
优点
Multimodal IoT seems like a potentially interesting under-explored topic.
缺点
The paper is significantly below the acceptance level of ICLR for the following reasons.
- The paper is poorly written and lacks a clear structure, premise or narrative.
- It is unclear what the claimed contribution of the work is and how it advances scientific research. It reads more like an opinion piece on the topic of multimodal IoT, rather than offering any concrete scientific insights.
- Many of the datasets in the collection of 1.115M samples presented in this work are publicly available datasets from other research projects and not ones curated by the authors.
- The experiments are poorly described and simply not reproducible.
- The experiments have no clear conclusion or insights.
问题
I would strongly recommend that the authors review existing published works in ICLR and other AI and computer vision conferences to understand how to improve their papers' presentation, experiments, contributions and style, etc. In its current form the paper is not acceptable as a scientific article.
The paper presents a benchmark for multimodality in the context of IoT, with 12 modalities and 8 tasks. While the reviewers found some merit in the dataset, the main finding of the paper that multi-modal multi-task learning provides some gains on this dataset was considered to fall short of being publishable, as there were multiple issues/questions with the framing, presentation of the experiments, and explanation of the experiments. Unfortunately, the authors did not participate in the author/reviewer discussion, and as such the issues have not been clarified. While the paper is recommended to be rejected at this time, the authors are encouraged to take the reviewers' feedback into account for a future submission to make their dataset a valuable contribution to the IoT domain.
为何不给更高分
The paper falls short on multiple fronts such as Writing, Experiments, Contributions, and is far from the acceptance bar.
为何不给更低分
The dataset seems interesting on its own right if it is fleshed out more in the future.
Reject