KITS: Inductive Spatio-Temporal Kriging with Increment Training Strategy

Summary: Unfortunately, none of my comments (and that of the other reviewers) were apparently taken into account as this submission is basically unchanged w.r.t. the previous iteration.

We would like to emphasize that we have addressed all comments from the previous conference, which we explain as follows.

• Reviewer 1 for the previous submission (weak accept - score 6):
  • W1: This comment is that the reviewer did not observe significant improvement of our KITS over an existing method INCREASE due to some misinterpretation. We provided explanations on this in the rebuttal, with which the reviewer is satisfied. No changes are necessary for this comment.
  • W2: This comment is on the robustness aspect of the paper. We provided explanations on this in the rebuttal, with which the reviewer is satisfied. No changes are necessary for this comment.
  • Q1, Q2: The comments are some questions about the details of virtual node generation (Section 3.2). We have re-written this part to make it clearer.
  • Comment on our rebuttal: "I would like to thank the authors for their detailed and clear rebuttal. I am convinced with the provided answers. I keep my rating intact." This comment shows that the reviewer is convinced with our responses to the comments in the rebuttal.
• Reviewer 2 for the previous submission (weak accept - score 6):
  • W1: This comment suggests to include some empirical evidence about "graph gap issue" and "fitting" issue. We have mentioned them in introduction and included the details in Appendix B.
  • W2: This comment suggests to include more error bars evaluation (like Table 9 in Appendix H.2). We have followed this suggestion and further included Table 10 in Appendix H.2.
  • W3, W4 (minor): These comments are about two typos in Equation 1 and 2. We have fixed the typos.
  • Comment on our rebuttal: "Thank you for your rebuttal! I updated my score as my concern raised in the review was resolved. I would recommend the discussion in the author's rebuttal be involved appropriately in the revised manuscript." This comment shows that the reviewer is satisfied with our rebuttal overall. We have updated the draft as suggested by this reviewer.
• Reviewer 3 for the previous submission (you, borderline reject - score 4):
  • W1, W2, Q1: Kriging single node at a time. We have included and compared your suggested idea in Appendix H.8, which shows that the suggested idea does not work as well as our method for the problem settting we target in this paper.
  • W3.1: Isolated nodes introduced after dropping nodes. We replied to you with some statistics to answer this question during the rebuttal of the previous conference. We believe this issue has been settled in our rebuttal, and thus there is no change for this comment.
  • W3.2: More discussions about region missing. We have enriched the corresponding part, which is near Table 6.
  • W5 (minor): More baselines such as SPIN by Marisca et al. NeurIPS 2022. We appreciated your comment, and replied to you with some experimental results about this baseline, but we finally decided not to include it for the following reasons. (1) This method targets transductive setting, which is not the main target setting of our paper; (2) It does not work as well as GRIN; (3) This method costed much in terms of GPU onboard memory, and we met out-of-memory (OOM) problem in PEMS07 dataset (with 883 nodes). In case that you still think we should include these results, we shall include them in a new version of our draft.
  • Q2: Average degree of training and inference graphs. We have included some statistics in Appendix B.1.
  • Q3 (minor): Why removing self-loops in Equation 1. We have replied to you in the previous rebuttal.
  • Q4 (minor): Initialization of virtual nodes' representations. We have re-written Section 3.2 to make it more clear.
  • L1: The possible negative effects of adding noise to training graphs. We previously replied to you with statistics in Figure 5.

W1, Q2: Intuition behind Spatio-Temporal Graph Convolution (STGC) Module.

Thanks for raising this good question! We reply to this question as follows:

1. For Q2 (why the setting is spatio-temporal), we claim that for the $i$-th node $X_i^t$ obtained from timestamp $t$:
   • if it aggregates information from other nodes $X_j^t$ under the same timestamp $t$, where $j \neq i$, then the spatial aspect will be included in the setting;
   • if it obtains information from different timestamps, e.g., $t-1$, $t+1$, then the temporal aspect will be captured in the setting.
2. More details about STGC module:
   • Firstly, we explain the differences between normal GCN (spatial only) and STGC (spatial + temporal).
     • In Figure 3(a), for node-1 under timestamp $T_i$, normal GCN would only aggregate node-1 with node-2,3,5's (node-1's neighbors) information under the same timestamp $T_i$;
     • Differently, STGC would aggregate not only neighbor information within the same timestamp, but also information from each node's neighbors across time.
   • Then, we explain the design and rationale behind STGC module:
     • Forbid each node to aggregate information from itself (including the same and different timestamps):
       • As introduced in the paper, the nodes can be divided into 2 groups, namely observed nodes (with sensors) and target nodes (no sensors). Observed nodes have sensor readings all the time, while target nodes can never have sensor readings. The essence of kriging is to learn the pattern only on observed nodes during training, and directly apply the learned pattern to target nodes for value estimation. Due to the fact that observed nodes not only have sensor readings but also have loss regulations, their kriging patterns can be easily learned, i.e., their learned representations/features can be quite good.
       • With the above viewpoint in mind, if we allow self-loops, then the observed nodes would keep aggregating good-quality features, while the target nodes would keep aggregating bad-quality features from itself across time. Therefore, the pattern learned on observed nodes cannot suit target nodes well, and the performance would be suboptimal.
       • Therefore, we (1) remove self-loop in the adjacency matrix (verified in row 1&3 of Table 11); and (2) do not utilize RNN-like modules to aggregate standard temporal information (verified in row 1&2 of Table 11). For example, in Figure 3, node-1 ($T_i$) will not receive information from node-1($T_i$) and node-1($T_{i+1}, T_{i-1}$).
     • Aggregate each node with its neighbors information (from the same timestamp): The intuition is the same as that of the normal GCN.
     • Aggregate each node with its neighbors information (from different timestamps): This design is less common yet effective (verified in row 4-8 of Table 11). We provide a few examples to explain the intuition as follows:
       • Consider that we have road 1 and road 2, and the vehicles would drive from road 1 to road 2. Suppose at 8:00 am, a traffic jam occurs at road 2, and affected by road 2, road 1 gets crowded at 8:05 am. Therefore, the data distributions (e.g., vehicle speed data) of road 1 (8:05) have high correlations with those of road 2 (8:00).
       • Similarily, suppose a group of vehicles drive on road 1 at 8:00 am, and the same group drive to road 2 at 8:05 am. The driving behavior of the same group might be uniform, thus, the data distributions (e.g., vehicle speed data) of road 2 (8:05) have high correlations with those of road 1 (8:00).
   • Finally, the hyperparameter $m$ is used to define the scope, e.g., if $m=2$, then we consider each node at $T_i$ has high correlations with its neighbors at $\{T_{i-2}, T_{i-1}, T_i, T_{i+1}, T_{i+2}\}$, and would aggregate the former with the latter.

W1: The sparsity of deployed sensors.

Thanks for this comment. Our responses are as follows:

As written in the 1st paragraph of introduction ("Nevertheless, due to the high cost of devices and maintenance expenses, the actual sparsely deployed sensors..."), we target the kriging setting where the sensors are sparsely deployed (mainly for cost saving considerations). This setting has been studied in many existing kriging methods, including IGNNK (Wu et al., 2021a), LSJSTN (Hu et al., 2021) and INCREASE (Zheng et al., 2023). For this setting, the graph gap issue would remain valid. We further provide a few practical application scenarios for justifying the targetted setting.

1. The PEMS07 dataset comes from the Caltrans Performance Measurement System (PeMS) deployed in California (in use). It covers a large district, but the average node degree is 1.79 for the full graph (883 nodes), which means the graph is indeed locally sparse.
2. Similarily, the AQI-36 dataset is collected from an air quality monitoring application, where only 36 stations are installed for the whole Beijing City.
3. In our own business data of traffic, only 71 traffic sensors (to measure traffic flow) are sparsely installed to a large region, centered at Jinghu Amusement Park, Shaoxing, Zhejiang, China. The average node degree is less than 3.
4. In many water quality management systems, the sensors are sparsely installed. For example, the Rhein river in Europe only has 2-3 sensors in each city it flows through.
5. In a landslides monitoring application, there are only 7 landside tilt sensors installed in Yushan mountain in Taiwan.

We are not aware of any existing studies/datasets/practical scenarios, which target the kriging task with desnsely distributed sensors. We believe the kriging with densely distributed sensors would naturally be an easier task than the one targeted in this paper since each node would have more neighbors to leverage for the kriging task.

W2: Non-stationary or neural network-based kernels.

It's a very reasonable and interesting idea! When we worked on this paper, we did not come up with this idea, but we do agree it can potentially be applied to this task. Particularly, we also wonder if the virtual nodes-related graph edges (in adjacency matrix) can be learnable instead of being randomly determined (in the current version). To keep consistency with baselines, we would still use Gaussian-thresholded static kernels in this paper, and consider your idea as a very good future direction. Thanks again for this precious comment!

Q1: Use attention mechanism for global spatio-temporal features aggregation.

Thank you for your suggestion! Our understanding of the suggested idea is as follows - in case we misunderstood the idea, please kindly let us know. Suppose node-1's neighbors are node-2, 3, 4, and their data from 24 consecutive timestamps are available in a data sample. Your suggested idea is to use attention to aggregate node-1's features (in 1 of 24 timestamps) with node-2, 3, 4's features (from all 24 timestamps), so as to capture global dependency (in the temporal dimension).

Our responses are as follows:

• Temporal scope (from local to global): We have studied the effects of the temporal scope, which we represent with a parameter $m$ (i.e., the information over $2 \times m + 1$ timestamps is aggregated). According to the results shown in Table 11, we can observe that larger $m$ (more global temporal information) cannot guarantee better performance.
• GCN v.s. GAT/Attention: When we started this work, we also thought about which to use among GCN, GAT or standard Self-Attention (without adjacency matrix). We finally chose GCN, and we explain as follows:
  • GAT and Self-Attention would firstly perform projection operations to project input features to $Q$, $K$ and $V$ in attention blocks. However, we think such projections cannot work well in spatio-temporal kriging. This is because in kriging, observed nodes would consistently have loss supervisions, while target nodes do not have loss supervisions all the time. Hence, the complex projections learned on observed nodes may not be suitable for target nodes during inference since it would get overfitting on observed nodes.
  • We conduct experiments by replacing GCN (M-0 of Table 5) with GAT and Self-Attention under the same condition. We show the MAE scores on METR-LA dataset in the table below (we've already tuned the hyperparameters). We could see that attention-like methods cannot outperform GCN-like methods.

Dear all,

I believe I have discussed my opinion on this aspect at length. However some clarifications:

1. The experiment in Appendix H.8 simply shows that incremental reconstruction does not provide a performance improvement, but does not show that this so-called "graph gap" is the issue causing poor performance and that the proposed method performs well because it's addressing this specific problem. Actually, the experiment confirms that the improvement might not come from removing the gap.
2. Again, I'm not saying the training methodology proposed in KITS is not effective. My opinion is that the effectiveness of the method does not come from solving a mismatch in the graph used for training and inference as again it could easily be avoided in these settings. The introduced data augmentations are well aligned with the kriging task and support inductive learning, i.e., extrapolation to new nodes and unseen locations, and I believe that this is what could justify the improvement (this should be empirically validated). However, I believe that identifying the "graph gap" (as formulated in the paper) as the main issue of the existing methods is misleading (again it can be avoided very easily).
3. I don't think that any of my comments is "unfair".

That being said, as I already mentioned I believe the paper has merits.