Get Rid of Isolation: A Continuous Multi-task Spatio-Temporal Learning Framework

Dear Reviewer xkVJ,

Thank you for your meticulous review and insightful comments, which are invaluable in refining our approach and enhancing the quality of our manuscript.

W1. Data preprocessing. Thank you for your reminder. We have included the detailed data processing code in our anonymous repository, please check it available, and we explain the preprocessing steps in the Common Issue 4 of our global response. All datasets and the processed results will be made publicly available on a cloud storage platform after the paper is accepted.

W2. Contribution of MSTI and Transformers references. In MSTI, our innovation not only lies in designing a new attention mechanism but in how we use it to capture and refine interactions across different dimensions, such as 'spatial aspect - main observation' and 'temporal aspect - main observation' correlations. By coupling with the RoAda process, we enhance capturing these correlations across multiple tasks, leading to more effective and enriched encoding of spatiotemporal representation over main observations. This multidimensional interaction is specifically designed to exploit the inherent complexities and dependencies in multiple urban elements that standard spatiotemporal models may not fully capture. Additionally, we explain our specific design and the differences from other traditional ST with attention in the Common issue 2. Regarding the specific references to the foundational work on Transformers and the attention mechanism, we now have rectified this by including pertinent references to the original works on Transformers by Vaswani et al. [1], as well as other seminal papers that have shaped the use of attention in spatiotemporal learning [2-4], which provides a clearer context for our contributions.

[1] Attention Is All You Need, NeurIPS'17

[2] Learning Dynamics and Heterogeneity of Spatial-Temporal Graph Data for Traffic Forecasting, TKDE'22

[3] STAEformer, CIKM' 23

[4] PDFormer, AAAI'24

W3. Cold-start problem. Actually, generalization capacities have been empirically validated in Sec 5.2 and Sec. 5.4, where experiments on Tab. 2 can be viewed as imitating the cold-start issue on spatial dimension. Thanks for your good suggestion, we further add the cold-start experiments concerning tasks, i.e., training on two tasks and testing on another task. The details and results can be found in the Common issue 3.

W4. Performance of single task. Thanks for your careful reading, we also confirmed this result and found not seriously inferior to other single-task baselines (The comparison with the baselines have been made over single-task in Tab. 1 of Sec 5.2). Moreover, since in the experiment of Fig. 4(b), the hyperparameters and experimental settings we adopted are based on multi-tasks, investigation over the influence of the task number are also following such multi-task settings to guarantee the fairness of comparisons. This ensures that the only variable is the number of tasks. Considering learning over individual task, we also believe our backbone, MSTI, can outperform other baselines. To confirm this intuition, we have conducted additional experiments following the individual task setting, (one model for one task), and the testing MAE and MAPE are as follows,

(The symbols in the table are explained in the attached PDF of global response.)

Even in single-task setting, comparison with single-task baseline results in Tab. 1, it is observed that performance of MSTI is still better than most other single-task models, showing the effectiveness of MSTI. Additionally, the goal of our task is to collective the intelligence across different tasks, and enhance the individual learning, especially improving resilience on extreme scenarios. For detailed evaluation, it include experiments of Sec 5.2, experiments in Sec 5.4, and additional cold-start challenge investigated in our global rebuttal (Results in Common issue 3). All these results have demonstrated the success of coupling RoAda and MSTI, as well as the well-obtained goal of our work. We will also incorporate these discussion and new results into our revised manuscript.

W5. Justify the data summarization module and potential improvement. For this module, we are aimed to extract a snapshot of the data that captures the representative characteristics and typical pattern summarization of the data, so that it can play the role of task prompt. We utilized a straightforward yet effective MLP with an activation function to capture data snapshots, efficiently reflecting the intrinsic properties of the data. This approach has been empirically validated through ablation studies, demonstrating its capability to capture essential data features effectively. Thanks for your suggestion, and it inspire us for a potential improvement, i.e., we can further introduce contrastive loss, and build a constraint mechanism that is similar among tasks and different across tasks for the abstract representations of different tasks, so as to generate a higher quality task description learner. We are committed to further exploring this improvement and will include the results in revisions of our manuscript.

Thank you once again for your thoughtful feedback. We are committed to addressing these issues thoroughly and improving our manuscript accordingly, ensuring it meets the standards expected by the NeurIPS community.

Authors of Paper 2077

Dear Reviewer GaXf,

Thank you for your thoughtful review and acknowledging the potential and contribution of our work. We appreciate your insightful comments, which have provided us with an opportunity to refine our manuscript and address critical aspects that will enhance the clarity and impact of our research.

W1. Feature inconsistency across Tasks. Actually, the tasks we collected have consistent features ($C=3$: numerical value, time of day, day of week). The investigated space and interval units are standardized, and context factors are mapped into same dimension with an MLP to aviod the diverse dimensions of raw contexts. The detailed data preprocessing process can be referred to the Common issue 4 of the global response. On the other hand, if another task with inconsistent features, we will use task-specific MLP for observation encoding and a task-specific prediction head to transform features into consistent spatiotemporal interaction (MSTI), which is corresponding to unique prompt for each task. This approach ensures the unique features of each task to be handled uniformly and effectively, thus the shared spatial and temporal encoders can enhance spatiotemporal representation during rolling training.

W2. Task rolling details. For the rolling training of each task, we set a maximum of 100 epochs, as this number is based on observations from our training logs, where most tasks typically converge around 90 epochs. Additionally, we maintain a learning rate of 1e-5 during rolling training to prevent catastrophic forgetting. We will include these specific operational details in the revised manuscript to provide a clearer understanding of this process.

W3 & W4. Presentation issue. Appreciate your careful reading, we will check thoroughly and correct the typos in the whole paper and modify the proprietary terms with detailed explanation.

Specifically, PECPM refers to Pattern Expansion and Consolidation on Evolving Graphs for Continual Traffic Prediction [1] proposed in SIGKDD 2023.

[1] Pattern Expansion and Consolidation on Evolving Graphs for Continual Traffic Prediction, SIGKDD, 2023

Q1. Definition of domain. Actually, in our study, various domains correspond to different urban elements collected with different manners in a given city. For instance, in an urban system, it includes diverse elements such as taxi demands, crowd flow, traffic speed and accidents. We collect and organize various urban elements in a city into one integrated dataset. The goal of our work is to explore the integrated intelligence from various domains and enhance learning of each individual urban element. To this end, the concept of multi-task here is to forecast various elements from different domains in an integrated model.

Q2. Explanation of formalization. Initially, $E_s$ represents the spatial embeddings at the beginning of the model's operations, and $H[..., slice(s)]$ refers to the spatial segment within the tensor $H$ that initially matches $E_s$. As the model processes, $H'[..., slice(s)]$ evolves to reflect these updates, making it distinct from the initial embeddings $E_s$ , so we adopt a uniform slice representation. We will clarify this issue and the dynamic nature of these embeddings in our revised manuscript.

Q3. Input features across tasks. For different tasks within the same city, the input features can vary specific to each task, though they share the same dimensionality $C$. Currently, the tasks we have collected features with an input dimension $C=3$, which includes numerical values, time of day, and day of week as answered in W1. In our future work, we plan to collect more diverse datasets and conduct further experiments with various types of input features with different dimensions $C$ to accommodate broader ranges of urban tasks.

We again show our great appreciation of your valuable efforts on our work. We will comprehensively take you and all other reviewers comments into consideration and try our best to polish our manuscript for satisfying the high-level requirement of NeurIPS community.

Authors of Paper 2077

Dear Reviewer SHU9,

Thank you for your valuable feedback and for recognizing the contributions of our work. Your insights are greatly appreciated and will help us further improve the quality of our manuscript.

W1&Q1. Avoiding catastrophic forgetting. Actually, to avoid catastrophic forgetting, we implement several strategies during the Rolling Adaptation (RoAda) phase. Firstly, we set the learning rate for each task to 1e-5. This helps to retain more knowledge from previous tasks and prevents the model from over-adjusting to new tasks. By maintaining a low learning rate, the model can incrementally learn new information while preserving the stability of previously learned tasks. Additionally, we use task-specific weights for each task, such as task prompts. This method allows us to absorb the common features across all tasks while independently preserving and updating task-specific parameters. This approach ensures that when learning new tasks, the model does not forget the knowledge gained from previous tasks, thereby preventing catastrophic forgetting and avoiding overfitting to new tasks.

W2&Q2. More comparison baselines. We appreciate your feedback and agree that comparing against unified spatiotemporal/time series learning (such as UniST [1] and UniTime [2]) will better showcase the generalization capabilities of our model for multi-task learning within same urban system. In response to your suggestion, we have conducted additional experiments comparing our CMuST model with UniST and UniTime. The results of these comparisons are as follows:

(The symbols in the table are explained in the attached PDF of the global response.)

The results of these additional comparisons will be incorporated into the next version of our manuscript, providing a thorough evaluation and demonstrating the practical benefits and advancements introduced by our approach.

Thank you once again for your valuable feedback. Your suggestions and comments will significantly enhance the quality of our manuscript, and we are committed to making the necessary improvements to meet the high standards of the NeurIPS community.

[1] UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction, SIGKDD, 2024

[2] UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting, WWW, 2024

Authors of Paper 2077

Dear Reviewer YiFX,

Thank you for your detailed and insightful feedback. We have carefully addressed each concern below.

W1. Concept, definition and scope of Multi-task learning. Various domains correspond to different urban elements in a given city, and the concept of multi-task is to forecast various urban elements in a same neural network. (More details in Common issue 1).

W2. Contribution of MSTI. Previous attention-based spatiotemporal learners often process spatial and temporal aspects respectively. Different from those, MSTI designs cross-dimension attention, allowing flexible decomposition and spatial-temporal cross interactions. (More details in Common issue 2)

W3. Related work review. TrafficStream [1], PECPM [2] on ST learning, and CLS-ER [3] on task-level continuous learning are investigated in our paper. We supplement some more related works as below.

1. ST learning: DG2RNN [4] designs a dual-graph convolutional module to capture local spatial dependencies from both road distance and adaptive correlation perspectives. PDFormer [5] designs a spatial self-attention and introduces two graph masking matrices to highlight the spatial dependencies of short- and long-range views. TESTAM [6] uses time-enhanced ST attention by mixture-of-experts and modeling both static and dynamic graphs. These solutions focus on accuracy and generalization but neglect significance of continuous learning and fail to capture commonalities across different dynamic elements for collective intelligence.
2. Detailed continuous learners: C-LoRA [7] is low-rank adaptive by continuous self-regularization in cross-attention layer with stable diffusion. Kang et al. [8] develop a customized dirty label backdoor for online settings while maintaining high accuracy. COPAL [9] continuously adapts new data through a pruning approach guided by sensitivity analysis. These researches predominantly centers on the same task. Even CLS-ER addresses different tasks, it still focuses on image classification, trapping into little exploration on task-level streaming data. To model various aspects of commonality, MSTI and RoAda are proposed respectively. Therefore, the proposed task and solution are both novel to ST learning community.

[1] TrafficStream, IJCAI'21

[2] PECPM, KDD'23

[3] Learning Fast, Learning Slow, ICLR'22

[4] DG2RNN, TITS'24

[5] PDFormer, AAAI'24

[6] TESTAM, ICLR'24

[7] C-LoRA, TMLR'24

[8] Poisoning Generative Replay in Continual Learning ...., ICML'23

[9] COPAL, ICML'24

W4. Generalization experiments and theory.

1. Experimental Evidence: We have conducted generalization tests in Tab. 2 of Sec.5.2 and Fig. 4(b) of Sec. 5.4. Tab. 2 shows performance variations when node number reduced on NYC, indicating the robustness of CMuST. Fig. 4(b) shows performance changing with number of input tasks, indicating that task learning benefits from collective intelligence through assimilating common representations and interactive information, supporting the enhanced generalization in continual learning. Additional task-level cold start experiments are added to validate CMuST (Common issue 3).
2. Theoretical Perspective: It can be analyzed from uncertainty and information theory. First, introducing more diverse samples and iteratively repeating model training can reduce the epistemic uncertainty and increase the experience of models [10,11]. From information-theory aspect, continual learning allows the model to maintain useful common information and dynamically updates the model with new data, increasing the mutual information across task-level observations [12,13]. By learning multiple related tasks, the perceived knowledge of model is expanded via increasing patterns and dependencies, leading to enhanced generalization [14,15].

[10] Aleatoric and epistemic uncertainty in machine learning, MachLearn'21

[11] SDE-Net, ICML'20

[12] A Comprehensive Survey of Continual Learning, TPAMI'24

[13] Graph information bottleneck, NeurIPS'20

[14] Improving robustness to model inversion attacks via ..., AAAI'21

[15] Incorporating neuro-inspired adaptability for continual learning ..., NMI'23

W5. Cold-start issue. Generalization capacities have been empirically validated in Sec 5.2 and Sec. 5.4, where Tab. 2 can be viewed as imitating the cold-start issue on spatial dimension. (Task-level cold-start experiments are added in Common issue 3)

W6. Analysis of comparative experiments. 1) CMuST achieves overall good performances with most best results, and only 4 out 18 achieve the second best, showing the superiority against all baselines. 2) Baseline models are usually designed from specific tasks and datasets, then they tend to be tailored and tuned for the specific data and tasks, e.g., PromptST is designed on NYC, thus obtaining best performances on NYC. To this end, baseline model tends to individually achieve best while CMuST achieves overall best results. We will incorporate such discussions into our manuscript.

Q1. Continuous & continual learning. 'Continuous' is equivalent to 'continual'. The uniqueness of our work refers to a novel continuous task learning in ST community, which collects the integrated intelligence and benefits each individual learning task.

Q2. Model training and working details. Given each dataset with different urban elements, we trained separate models for each city (dataset). Regarding the increment on spatial domain, we have conducted the generalization experiments on Tab.2 by node masking. The results suggest that CMuST can ease the data requirements of single task by capturing and exploiting commonalities and diversity among tasks.

Other. The pseudocode of RoAda. We have added pseudocode of RoAda to global response PDF.

Based on your suggestions, we are revising the manuscript to satisfy the high standards of the NeurIPS community. If you have further questions, please feel free to discuss with us.