IoT-LLM: Enhancing Real-World IoT Task Reasoning with Large Language Models

1. Most importantly, the technical components proposed in this paper lack novelty when compared to existing studies. For example, state-of-the-art approaches for multimodal LLMs (MLLMs) incorporate visual inputs, which outperform the text-only prompts used in this work [r1, r2].

2. Additionally, the concept of “IoT data simplification” is adopted from prior research (Gruver et al., 2024; Spathis & Kawsar, 2023), and feature extraction is also widely utilized in existing studies. The use of “IoT data enrichment” and “knowledge retrieval” similarly aligns with a recent study [r1]. Given that text-only prompts are no longer the most effective approach, and many of the proposed ideas are derived from prior work, I do not see substantial technical novelty in this study.

3. As a suggestion, the term “IoT task” feels too broad and potentially misleading. This work primarily focuses on sensory data, so it may be more appropriate to refer to these as “sensory tasks.” The term “IoT tasks” suggests broader applications, such as controlling IoT devices via networking.

References: [r1] By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting (EMNLP’24) [r2] Plots Unlock Time-Series Understanding in Multimodal Models (arXiv ’24)

1. The paper should address a fundamental question: why should we use LLMs for IoT data tasks, which are typically numerical in nature? While LLMs excel at handling textual data, they are not known for expertise in processing numerical data. Given the high cost associated with LLMs, what unique advantages do they offer for IoT tasks, especially when many other solutions are already available?

2. The paper lacks a baseline comparison with traditional approaches for IoT tasks. Without evaluating the gains or advantages of LLMs over these conventional methods (e.g., in terms of accuracy, efficiency, or cost), the benefits of using LLMs for IoT remain unclear.

3. Although the paper incorporates certain adaptations, such as Chain-of-Thought (CoT) and Retrieval-Augmented Generation (RAG) to make LLMs work for IoT tasks, these methods are standard. The paper’s novelty appears limited in this regard, and it would benefit from additional details and examples to illustrate precisely how these steps are implemented.

• The experimental results are limited. The only experiments in the paper are 5 classification tasks and 1 regression task using 5 LLMs. This does not provide much variety in the experimental setting and the analysis can be expanded upon. For example:

• Does the language model consistently provide reasoning for its predictions that align with the correct reasoning?
• Why does the language model perform the worst on Heartbeat?
• How does the amount of relevant literature on the IoT task affect the results? For example, what happens if we had very little literature to retrieve for the Occupancy task?
• How much of each of the datasets has the language model memorised?
• What happens if there is class imbalance? Is there class imbalance in the datasets and if so is accuracy the correct metric?
• How is the performance impacted by the number of demonstrations?
• Why does Gemini-Pro show a decrease in performance on Heartbeat?
• How do the results compare to the classical models currently used as best practise in these settings?

• There is a lot of missing experimental information, such as:

• How many repeats were done for the regression task?
• Were any repeats done for the classification tasks? Since no STD is provided.

• There should also be a discussion about the practicality of using an for IoT tasks, where compute is likely limited and costs of API calls could be significant if the model needs to make predictions continuously at a high frequency. For example:

• What is the cost of using GPT-4 or 3.5 for the heartbeat dataset (which is sampled at 72Hz)?
• Is it possible to use an LLM to make predictions on this dataset at 72Hz using an API?
• How much compute is required to use one of the open source models the authors present at 72Hz?

1. The baseline comparison over-represents the margin of improvement by setting up a simplified baseline that doesn’t accurately reflect the improvement over the previous best framework, particularly the Penetrative AI prompt. The current baseline acts as a “strawman,” making the gains seem larger than they might be if compared against a more structured baseline. The Penetrative AI prompt, for instance, already includes a rich, context-aware structure and could likely perform even better if it incorporated the information retrieval elements proposed in this IoT LLM paper. To present a more balanced view of the framework’s improvements, the authors should consider comparing against a baseline like Penetrative AI, augmented with similar components, to better showcase the incremental gains of the new approach.

Here's the Penetrative AI prompt structure:

Objective: [Clear task statement]

Background Knowledge: [Expert domain knowledge] [How to interpret different data types]

Response Format: Reasoning: [Analysis instructions] Summary: [Summary format] [Specific output requirements]

[Optional: Processing Procedure] [Step-by-step analysis instructions]

[Optional: Reasoning Example] [Sample data] [Example reasoning] [Example outputs]

Query Data: [Actual sensor data for analysis]

This is the HarGPT Prompt structure:

Instruction:

You are an expert of IMU-based human activity analysis.

Question:

The IMU data is collected from {device name} attached to the user's {location} with a sampling rate of {freq}. The IMU data is given in the IMU coordinate frame. The three-axis accelerations and gyroscopes are given below.

Accelerations: x-axis: {...}, y-axis: {...}, z-axis: {...}

Gyroscopes: x-axis: {...}, y-axis: {...}, z-axis: {...}

The person's action belongs to one of the following categories: <category list>.

Could you please tell me what action the person was doing based on the given information and IMU readings? Please make an analysis step by step.

Response: {answer}

--

These were extracted from each respective paper.

2. The paper tends to over-claim the novelty and impact of its approach. While the modular design and tailored prompt structure are thoughtful, the idea of applying LLMs to IoT tasks isn’t entirely new—similar efforts, such as Penetrative AI and HardGPT, have covered comparable ground. Reframing this work as an incremental improvement, rather than a fundamentally new approach, would provide a more accurate context. Clarifying the unique aspects of this framework, particularly in terms of modular design for IoT tasks, would help in accurately positioning its contributions.

3. The problem formulation lacks precision, and the specific properties of IoT data that drive the modular design choices are not clearly articulated. Although the framework includes components like data enrichment and domain knowledge retrieval, the connection between these components and IoT-specific data characteristics (such as sensor noise, temporal dependencies, or data sparsity) is not fully articulated. A clearer problem formulation that explicitly links these IoT-specific data properties to each module would strengthen the motivation behind the design choices and make the framework’s modularity more compelling.

4. The authors claim that the prompt structures generalize across various IoT tasks, but the evaluation doesn’t fully substantiate this. The ablation study shows some adaptability, but it’s limited in scope and doesn’t fully demonstrate generalizability across a broader range of data types and tasks. To their credit, they do articulate this in the limitations, but expanding the evaluation to include additional tasks or providing further evidence of prompt adaptability would make their generalization claims more sound.

5. The paper’s main contribution centers on IoT-specific prompt design, yet the reasoning behind each prompt structure isn’t deeply examined. Although the prompts are functional, the paper lacks a clear explanation for why certain components are emphasized or why specific details are included. It would strengthen the work if the authors provided a more detailed analysis of each prompt element’s necessity, how it aligns with IoT task requirements, and how changes to the prompt structure might impact performance. This would clarify the contribution and serve as a useful guide for adapting the framework to other IoT scenarios. Basically, it is not clear how the properties of the data are directly linked to the prompt.

6. The ablation study is useful but limited, as it covers only three out of the five tasks, which restricts its scope in evaluating the framework’s modularity. A more complete ablation study across all tasks would provide a clearer view of each module’s contribution, potentially highlighting specific strengths and limitations in greater detail.