Investigating Pattern Neurons in Urban Time Series Forecasting

W1: While the paper introduces neuron detection and optimization methods, it lacks in-depth theoretical explanation for why these neurons exhibit distinct behavior for low-frequency events. More analysis and discussion [1,2] on the working mechanisms of pattern neurons, such as potential differences in weight distribution or activation between high- and low-frequency events, is needed, along with supporting theoretical calculations.

[Mechanisms] ComS2T [1], LAPE [2], and PN-Train differ in both methodology and application. ComS2T [1], from a complementary learning systems perspective, disentangles neurons into stable (invariant) and dynamic (adaptive) categories using variation matrices, enabling adaptability to dynamic spatiotemporal data without specifically addressing low-frequency patterns. In contrast, LAPE [2] and PN-Train take a pattern-based perspective, identifying neurons linked to specific patterns. LAPE uses activation probabilities to detect pattern-specific neurons for large language model understanding but does not fine-tune these neurons for improved performance. PN-Train, however, identifies pattern-related neurons using activation values and fine-tunes them to improve performance on low-frequency patterns, while also preserving or even enhancing accuracy on high-frequency patterns.

[Weight Distribution] We have added visualizations of pattern neurons in Figure A.1 (also available at https://anonymous.4open.science/r/PN-Train/figures/vis.pdf), showing the distribution of holiday, non-holiday, and holiday-specific neurons. These visualizations confirm our assumption that holiday neurons include those specific to low-frequency events, which do not contribute to high-frequency events, and those that learn general time series features useful for both low- and high-frequency patterns. Additionally, pattern neurons are primarily located in the transformer's query and key components for pattern capturing.

[Theoretical Explanation] Understanding neurons in deep models remains challenging, with existing strategies [2, 3, 4, 5] relying on empirical validation. Similarly, PN-Train uses validation and visualization to confirm that identified neurons capture distinct behaviors for specific patterns. Empirical results show that deactivating these neurons significantly impairs low-frequency pattern forecasting. Notably, these neurons are primarily located in the transformer's query and key components, which are responsible for pattern clustering [6].

Furthermore, we provide a preliminary explanation for why these neurons exhibit distinct behaviors in response to low-frequency events. The optimization process in deep networks allows neurons to specialize by aligning their weights with specific data features. For a neuron $h^k_i = f(x, w^k_i)$, optimization minimizes the loss through backpropagation, where the weight update follows $\Delta w^k_i \propto -\frac{\partial L}{\partial w^k_i}$. When a specific input pattern $x^p$ significantly impacts the loss, $w^k_i$ aligns with $x^p$ to minimize the error. This alignment enables the neuron $h^k_i$ to effectively capture the pattern $p$ after training with $x^p$. Building on this behavior, our approach identifies neurons by quantifying the influence of the $k$-th neuron on a specific low-frequency pattern $p$, thereby directly pinpointing neurons associated with $p$.

While we acknowledge the importance of theoretical calculations, they often rely on strong assumptions. Given that our primary contribution focuses on identifying and fine-tuning pattern neurons to enhance urban time series forecasting, we prioritize empirical validation in this work, leaving deeper theoretical exploration for future studies.

We sincerely appreciate your insightful review and comments. Below, we provide detailed responses to the specific concerns you raised.

Q1: Is there any way to quantitatively define or model the frequency of events?

A low-frequency event can be defined as a specific type of event that occurs infrequently, with an occurrence rate below a predefined threshold. We quantify this using the frequency rate of the event [1], calculated as $(\text{number of event days} / \text{total observation days}) \times$ 100%.

In this study, holidays are considered low-frequency events due to their limited annual occurrence, accounting for approximately 3.01% (11 days out of 365 or 366) in the metro-traffic dataset and 3.56% (13 days out of 365 or 366) in the pedestrian dataset.

W1: It is better to have a more complete discussion about low-frequency events, e.g., how these events are handled in previous works (Related Works section), how low-frequency events should be technically defined, and how the evaluation metrics are established to verify the effectiveness of the methods.

[Low-frequency events] While existing studies recognize the importance of accurate forecasting for holidays [2, 3, 4, 5, 6], they primarily focus on improving accuracy for all patterns simultaneously, without explicitly distinguishing between low-frequency and high-frequency events. Among these, [3] classifies holidays and non-holidays as special periods and normal periods, respectively. We classify holidays as low-frequency events because they are infrequent, comprising less than 4% of the year [1].

[Related Works] Previous works address low-frequency events, such as holidays, at the network level. Early approaches [2, 3] use feature extraction to encode factors like holidays, but they overlook the imbalance between low- and high-frequency events, leading to degraded prediction performance for low-frequency events. Recent methods introduce adaptive mechanisms to address this. For example, STV [4] embeds holiday names and corrects bias toward normal-day patterns by distinguishing their distributions through a two-stage training process. PM-MemNet [5] uses a key-value memory structure to cluster traffic patterns and dynamically retrieve relevant ones for accurate predictions. TESTAM [6] leverages a Mixture of Experts to select the most suitable expert for low-frequency events, improving predictions through specialized routing. However, these methods rely on additional network components to handle low-frequency patterns. In contrast, our work improves the network at the component level by directly fine-tuning pattern neurons, without the need for extra network structures.

[Evaluation Metrics]: We used well-established metrics [2, 3, 6], including MAE, RMSE, and WMAPE, to verify the effectiveness of the methods. We evaluate performance separately for holidays (low-frequency events), non-holidays (high-frequency events), and overall, ensuring a comprehensive evaluation across all scenarios.

Q2: Is there any theoretical principle that can support the neuron-based strategy?

Understanding neurons in deep models remains challenging, and existing pattern-based strategies [7, 8, 9, 10] primarily rely on empirical validation. Similarly, PN-Train utilizes empirical methods to support its neuron-based strategy through the detection and visualization of pattern neurons. Empirical results confirm that these neurons are successfully detected and are primarily located in the query and key components. This aligns with the theoretical principle on transformers, which suggests that the query and key components are responsible for pattern clustering [11].

Additionally, we provide a preliminary theoretical explanation from an optimization perspective to support such neuron-based strategies. In deep networks, optimization allows neurons to specialize by aligning their weights with specific data features. For a neuron $h^k_i = f(x, w^k_i)$, weights are updated via backpropagation following $\Delta w^k_i \propto -\frac{\partial L}{\partial w^k_i}$. When a specific input pattern $x^p$ significantly affects the loss, $w^k_i$ aligns with $x^p$ to minimize the error. This alignment allows the neuron $h^k_i$ to effectively capture the pattern $p$ after training with $x^p$. Based on this behavior, neuron-based strategies quantify the influence of the $k$-th neuron on a given pattern $p$, enabling the direct identification of neurons associated with pattern $p$.

While we acknowledge the importance of theoretical principles, they often rely on strong assumptions. Given that our primary contribution focuses on identifying and fine-tuning pattern neurons to enhance urban time series forecasting, we prioritize empirical validation in this work, leaving deeper theoretical exploration for future studies.

W2: Lack of comparison with baseline methods that deal with various unbalanced data distribution problems.

We have included an additional baseline, PM-MemNet [5], which can tackle unbalanced data distributions by clustering time series patterns into 100 groups within a memory bank to assist the predictions. We have also compared our model with TESTAM [6], which uses a Mixture of Experts to dynamically select the most suitable expert for handling diverse unbalanced distributions based on input patterns.

The results below show that PN-Train outperforms existing methods in handling unbalanced data distributions. PM-MemNet performs worse than TESTAM and PN-Train because it relies heavily on fixed memory patterns extracted from the training set, which may include noise, reducing its forecasting accuracy. TESTAM performs well on low-frequency patterns like holidays in the Pedestrian dataset thanks to its routing mechanism. However, its overall performance is limited because unnecessary routing for dominant patterns reduces its effectiveness on high-frequency events.

Metro-Traffic:

Pedestrian:

We sincerely thank you for your constructive comments. Your insights have been invaluable in improving our manuscript. We hope our responses have addressed your concerns and have been incorporated into the revised version, with changes marked in RED. If any questions remain, we would be happy to engage in further discussions.

Reference:

[1] Devore, Jay L. "Probability and statistics." Pacific Grove: Brooks/Cole (2000).

[2] Zhang, Junbo, Yu Zheng, and Dekang Qi. "Deep spatio-temporal residual networks for citywide crowd flows prediction." AAAI, 2017.

[3] Lin, Ziqian, et al. "Deepstn+: Context-aware spatial-temporal neural network for crowd flow prediction in metropolis." AAAI, 2019.

[4] Hou, Chenyu, et al. "A deep-learning prediction model for imbalanced time series data forecasting." Big Data Mining and Analytics 4.4 (2021): 266-278.

[5] Lee, Hyunwook, et al. "Learning to remember patterns: pattern matching memory networks for traffic forecasting." ICLR, 2022

[6] Lee, Hyunwook, and Sungahn Ko. "TESTAM: a time-enhanced spatio-temporal attention model with mixture of experts." ICLR, 2024

[7] Tang T, Luo W, Huang H, et al. Language-specific neurons: The key to multilingual capabilities in large language models. ACL, 2024

[8] Dai, Damai, et al. "Knowledge neurons in pretrained transformers." ACL (2022).

[9] Voita, Elena, Javier Ferrando, and Christoforos Nalmpantis. "Neurons in large language models: Dead, n-gram, positional." ACL (Findings) 2024.

[10] Chen, Yuheng, et al. "Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons." AAAI 2024.

[11] Geshkovski, Borjan, et al. "A mathematical perspective on transformers." arXiv preprint arXiv:2312.10794 (2023).

Dear Reviewer 3uYm,

Thank you very much for your thoughtful comments and insightful suggestions. We sincerely appreciate the time and effort you have invested in reviewing our work.

We have made our utmost effort to address your comments and have revised our manuscript to incorporate your valuable suggestions. In the revision, we defined low-frequency events (Section 2), conducted additional experiments (Section 4.2), expanded the related work (Section 5), added future directions (Section 6), and expanded the evaluation metric description (Appendix 5). These updates are highlighted in red.

We are greatly encouraged by the positive feedback from other reviewers, which has led to increased scores. We kindly ask if you have any additional questions or concerns, as we would be happy to engage in further discussions to address them. If we have successfully addressed most of your concerns, we would greatly appreciate it if you might consider raising your score. Thank you once again for your valuable time and effort.

Best regards,

The Authors

W2: The absence of open-source code makes it difficult to reproduce the results.

The code, datasets, and checkpoints are available at https://anonymous.4open.science/r/PN-Train/README.md

W3: The study utilizes too few datasets. It would have been better to use datasets from the PeMS system, such as Large-ST [1], which are well-processed and cover longer time spans.

We have conducted additional experiments on the Large-ST [1] in the Greater Bay Area, focusing on traffic predictions during holidays and anomalous events like parades. The results below confirm that our training method enhances forecasting for low-frequency events without compromising overall accuracy by neuron-level fine-tuning.

During holiday:

During holiday and parades:

We sincerely thank you for your constructive comments. Your insights have been invaluable in improving our manuscript. We hope our responses have addressed your concerns and have been incorporated into the revised version, with changes marked in RED. If any questions remain, we would be happy to engage in further discussions.

Reference:

[1] Liu X., Xia Y., Liang Y., et al. LARGEST: A Benchmark Dataset for Large-Scale Traffic Forecasting. Advances in Neural Information Processing Systems, 2024, 36.

[2] Luo, Xianglong, Danyang Li, and Shengrui Zhang. "Traffic flow prediction during the holidays based on DFT and SVR." Journal of Sensors 2019.1 (2019): 6461450.

[3] Cools, Mario, Elke Moons, and Geert Wets. "Investigating effect of holidays on daily traffic counts: time series approach." Transportation research record 2019.1 (2007): 22-31.

[4] McElroy, Tucker S., Brian C. Monsell, and Rebecca J. Hutchinson. "Modeling of holiday effects and seasonality in daily time series." Statistics 1 (2018): 1-27.

[5] Hou, Chenyu, et al. "A deep-learning prediction model for imbalanced time series data forecasting." Big Data Mining and Analytics 4.4 (2021): 266-278.

We sincerely appreciate your insightful comments. Below, we provide detailed responses to the specific concerns you raised.

W1: The distinction between low-frequency and high-frequency events is primarily between holidays and non-holidays, which may not have substantial practical significance. Other baselines also improve accuracy for both low-frequency and high-frequency events simultaneously. Focusing on anomaly detection or traffic prediction under anomalous events might have been more meaningful.

[Substantial practical significance] Accurate forecasting of urban time series during holidays has been recognized as crucial [2, 3, 4, 5], as holiday and non-holiday periods exhibit distinct patterns that can reduce forecasting accuracy during holiday periods. Effectively predicting both patterns is key for urban planning, optimizing schedules, improving operations, and balancing ride-hailing demand and supply by aligning resources with crowd dynamics.

[Other baselines] While other baselines improve accuracy for both events, we show that fine-tuning at the neuron level further enhances the network's forecasting ability. For example, although STAEformer achieves strong overall accuracy on the Pedestrian dataset by optimizing for both event types, it tends to prioritize dominant non-holiday patterns, weakening holiday predictions. PN-Train builds on STAEformer by further fine-tuning the pattern neurons, resulting in a 7.91% WMAPE reduction for holidays and a 4.73% reduction for non-holidays compared to STAEformer. These results highlight the value of granular fine-tuning in improving the network's performance on low-frequency patterns.

[Under anomalous events] We have conducted additional experiments on forecasting during anomalous events, e.g., extreme weather, with results shown below. The results confirm that fine-tuning at the neuron level significantly improves predictions in these conditions.

Metro-Traffic:

Pedestrian:

Dear Reviewer XvJ7,

Thank you very much for your thoughtful comments and insightful suggestions.

We sincerely appreciate the time and effort you have invested in reviewing our work. We have made our utmost effort to address your comments by clarifying several details, releasing the code, and conducting additional experiments in our rebuttal.

If you have any further questions or concerns, we would be happy to engage in additional discussions. We are truly grateful for your time and comments.

Best regards,The Authors

Thank you to the authors for the rebuttal, which has clarified most of my concerns. The experiments are very well executed. Additionally, I hope that in the subsequent manuscipt revision, more details can be added about anomalous events, such as extreme weather conditions, in the text. This is important in case studies and can also attract the reader's attention. Since further experiments are not mandatory in the discussion phase, consider adding other exceptional events beyond extreme weather (such traffic incident, large gatherings such as concerts) to enhance the highlights of the work. The current content is already quite sufficient; this is just a suggestion that adding more than one would be even better. I will increase my score.

We sincerely appreciate your insightful review and comments. Below, we provide detailed responses to the specific concerns you raised.

Q1: In line2 Algorithm1, why do authors filter out low-frequency samples?

We filter out low-frequency samples because PN-Train needs to fine-tune pattern neurons using a small set of samples related to the target event. By randomly sampling a subset of low-frequency samples from the training dataset, this approach ensures effective fine-tuning without over-training on the same samples. Additionally, this approach also ensures a fair comparison with baseline methods, as all training and fine-tuning samples come from the same training dataset.

Q2: Can we allocate low-frequency patterns by categorizing neuron activations?

Yes, this is achievable. To test the effectiveness of this approach, we introduce PN-Train+, which clusters neuron activation patterns and allocates the cluster with the smallest sample size to low-frequency patterns. Similar to our PN-Train, pattern neurons are identified as those consistently activated under these low-frequency patterns, and the model is fine-tuned using samples from this pattern group.

We compare PN-Train+ with PN-Train* (no fine-tuning) and PN-Train (fine-tuning neurons using explicitly defined low-frequency patterns). Results below show that while PN-Train+ improves forecasting for low-frequency patterns, it underperforms our PN-Train across all pattern types. This is likely due to imprecise pattern categorization from implicit defined patterns, which mixes pattern types and reduces fine-tuning effectiveness.

Metro-traffic:

Pedestrian:

W1: The framework requires explicitly defined low-frequency patterns, i.e., holidays, and a dataset with low-frequency patterns is also required (as in Eq(3)). It is unclear whether the framework could handle implicit low-frequency patterns.

In this study, our primary goal is to investigate whether neurons associated with specific patterns exist within deep networks. By using well-defined low-frequency patterns, such as holidays, as explicit examples, we provide strong evidence supporting their existence.

While the current PN-Train framework does not explicitly handle implicit low-frequency patterns, our findings that fine-tuning explicitly defined patterns also enhances forecasting for other patterns. This suggests that PN-Train improves the model's overall pattern recognition capability, which also benefits the modeling of implicit low-frequency patterns.

Furthermore, our new experiments with PN-Train+ demonstrate that handling implicit low-frequency patterns does not necessarily lead to better forecasting performance. This is because some low-frequency patterns may arise from noise, such as sensor malfunctions, which do not contribute to meaningful improvements. Also, implicit identification can incur misclassification, resulting in suboptimal optimization. In contrast, explicitly defining low-frequency patterns ensures precise optimization for more effective and reliable forecasting performance.

Dear Reviewer x92L,

Thank you very much for your thoughtful comments and insightful suggestions. We sincerely appreciate the time and effort you have invested in reviewing our work.

We have made our utmost effort to address your comments by clarifying several details and conducting additional experiments in our rebuttal.

If you have any further questions or concerns, we would be happy to engage in additional discussions. We are truly grateful for your time and comments.

Best regards,The Authors