Q2.1 How did the baseline models adapted for partially observed data.

Thank you for your question. In our experiments, the missing values are imputed using the mean value, after which the Transformer and Autoformer models were applied. The same procedure is used for the additional experiments with iTransformer, CARD, and PatchTST.

Q2.2 Justification on deliberately omitted methods not designed for non-missing data. Besides, the experimental analysis should include SOTA baselines specifically designed for long-term forecasting tasks, such as iTransformer [1], CARD [2], and Crossformer [3].

Thank you for raising this issue. In our initial submission, we did not include iTransformer and CARD because these methods are not specifically designed for time series with missing values. Comparing them directly with our method might lead to an unfair evaluation. Instead, we focused on comparisons with SOTA methods tailored for time series prediction with missing values, such as BiTGraph, as well as S4-based methods. For transformer-based baselines, we selected two classic architectures: Transformer and Autoformer.

To provide a more comprehensive evaluationaddress your request, we have now included PatchTST, iTransformer, and CARD in our experiments. The results on four benchmark datasets are presented in the table below, with additional results on a real-world dataset. Furthermore, we analyze the computational cost and the performance across different horizon windows.

Among the three additional methods, PatchTST exhibits strong performance in handling missing values, particularly on the Electricity dataset, and also performs well in scenarios without missing values. However, in other settings, its results are less competitive compared to S4M. Additionally, as shown in Table X, PatchTST incurs significantly higher training and inference times than S4M.

W3. Highlighting the existing discussion on hyperparameters K1, K2, tao1, tao2..

We agree on the importance of discussing hyperparameters. Therefore, in our original submission, we provided extensive sensitivity analysis on these hyperparameters in Appendix D. Specifically, we discussed $K_1$ in, $K_2$ in Appendix D.4.1. According to our results, a suggested $K_2$ is between 5 and 10, as this range is effective and relatively insensitive to performance. The recommended ranges for $\tau_1$ and $\tau_2$ are [0.3, 0.6] and [0.8, 1.0), respectively. Both parameters can be selected using the validation set. Most of the experiments are robust to the choice of $K_1=30$ or $K_1=50$. If one finds the dataset exists large amout of clusters over 100 (by running a preliminary experiment with a large value of $K_1$ and let the data adaptively tells the number of clusters), they can also adjust this number accordingly.

Q1. Adding experimental results with various horizons.

Thank you for your question. In response to your comment, we have included new results with a fixed lookback window of size 192 and various prediction lengths. The results show that our model retains top performance across most horizons for the Traffic, ETTh1, and Weather datasets. Specifically, for the Electricity dataset, our proposed model ranks among the top two performers in most cases.

We thank the reviewer for the thoughtful and detailed review. Also, we appreciate that the reviewer acknowledges our proposed model achieves SOTA performance in handling missing data. We address the raised concerns below.

W1. The authors should further explain the motivation for introducing $\overline{\mathbf{E}} E_m(m_{t};\theta_m)$  to the SSM model in Equation 4.

Thank you for your question about our motivation for incorporating $\overline{\mathbf{E}} E_m(m_{t};\theta_m)$ into the model.

To address the missing data problem in S4 models, we aim to (1) distinguish the missing time points, enabling the model to treat them differently from the observed data (e.g., by referring to data in the prototype bank), and (2) ensure that the core properties of the S4 model are preserved. To this end, we seek a term that can flag missing values while preserving the HiPPO structure of S4. We found that integrating additional masking terms $M$, inspired by literature [1], to serve as a simple yet effective indicator for the model to recognize missing values. However, since the elements of $M$ take binary values (0 or 1), they are not naturally on the same scale as the other terms in (4). To address this, we designed an encoder to transform the mask information to an appropriate scale. Incorporating this term still preserves the HiPPO structure of S4, thereby enriching the model with additional information while maintaining its core advantages.

W2. Highlighting the existing discussion between the S4M and existing methods designed for handling missing values

In fact, we discussed the differences between our methods v.s. the other existing methods for handling missing values in both Introduction (see lines 62-73) and Appendix A.2. To recap, traditional approaches for handling missing values use a two-step process: imputing missing values first and then performing standard analysis. This can lead to errors and suboptimal results, especially in multivariate time series with complex missing patterns and high missing ratios. We also refer to methods that directly forecast with missing data. RNN-based methods, such as BRITS and GRUD, typically require long training times and exhibit inferior forecasting performance. Graph network-based models, like BiTGraph, are effective at navigating temporal dependencies and spatial structures but often suffer from high memory usage. ODE-based methods, such as Neural ODE, generally incur high computational costs.

In contrast, we propose S4M, which combines a prototype bank with a structured state space model (S4). Our approach focuses on recognizing and representing missing data patterns in the latent space, thereby enhancing model performance by better capturing underlying dependencies while maintaining the high performance of S4.

Dear reviewer EZj4,

We sincerely thank you for your thoughtful review and valuable feedback. We have carefully addressed each of your questions and provided detailed responses in the rebuttal. We hope to have resolved all your concerns. If you have any further comments, we would be glad to address them before the rebuttal period ends. If our responses address your concerns, we would deeply appreciate it if you could consider raising your score. Your recognition for our novel work means a lot. Thanks again for your time and effort in reviewing our work.

Regards, S4M authors

Dear Reviewer EZj4,

We are at the end of the discussion period, so please take some time to read the response to your review for submission 10665

Also, please indicate the extent to which the response addresses your concerns and whether it changes your score - try to explain your decision.

All the best,

The AC

Q2. I would also like to see Only Masking, i.e. having the mask m_t in (4) but no prototype bank, i.e. replacing o_t with X_t. Is your prototype-bank really needed for irregular time-series forecasting?

Thank you for your valuable feedback. We have included an ablation study for the first module, ATPM. In this study, we compare S4M with and without ATPM, highlighting the improvements brought by ATPM, particularly on the Traffic and ETTh1 datasets. The following experiments were conducted under the same settings as those in the ablation studies presented in the paper.

R

W2.2. Justification on MethodsComparison.

Thank you for your comments on the comparison between PatchTST and our method. There might be a misunderstanding of the comparison. PatchTST does not include straightforward forecasting with missing values. Masking is only used in the context of self-supervised learning. Therefore, the results with 0.06 missing values (~40% missing ratio) are not directly comparable. In our initial submission, we did not include iTransformer and PatchTST because these methods are not specifically designed for time series with missing values. Comparing them directly with our method might lead to an unfair evaluation. Instead, we focused on comparisons with SOTA methods tailored for time series prediction with missing values, such as BiTGraph, as well as S4-based methods. For transformer-based baselines, we selected two representative architectures: Transformer and Autoformer.

To address your request , we have now included PatchTST, iTransformer, and CARD in our experiments. The results on four benchmark datasets are presented in the table below, with additional results on a real-world dataset(https://openreview.net/forum?id=BkftcwIVmR&noteId=qh9vNw3wyX). Furthermore, we analyze the computational cost (https://openreview.net/forum?id=BkftcwIVmR&noteId=8TefCTqULs) and the performance across different horizon windows (https://openreview.net/forum?id=BkftcwIVmR&noteId=bcEimHMmTq).

Among the three additional methods, PatchTST exhibits strong performance in handling missing values, particularly on the Electricity dataset, and also performs well in scenarios without missing values. However, in most of the settings, S4M achieves consistently superior performance. Additionally, as the table on computational cost(https://openreview.net/forum?id=BkftcwIVmR&noteId=8TefCTqULs) shows, S4M is significantly more efficient than these three suggested methods..

Q1. Have you tested S4M when there are no missing values at all? I would be curious whether your prototype bank is also useful if no values are missing.

Thank you for the question. By design, our method is suitable for time series with block-based missing data. The historical features stored in the prototype bank are particularly helpful when the missing ratio is high. If there is no missing data i, as expected, our method will not show a significant advantage over the other methods but will still maintain a very competitive performance. For this experiment, we report the results using a horizon window of 96 and a lookback window of 96, with no missing values in the original dataset.

W1.2 Justification on Datasets Selection.

There appears to be a misunderstanding on the type of data missing problem we are focusing on in this study. Our work considers cases with block missing patterns in regularly sampled time series, where the observed values occur at consecutive time points (see our Fig. 4). This structure enables the design of an informative representation $o_t$, which is crucial for capturing temporal dependencies effectively. In contrast, standard irregularly sampled time series, like MIMIC and Physionet, do not contain such patterns of consecutive observations in the non-missing time points, making them outside the scope of our study.

Although our current design does not consider the general irregular sampled data, we appreciate your encouraging recognition that "I do really like the idea and think that it has potential, even beyond S4 models for irregular time-series forecasting." We also believe in the benefits of introducing a prototype bank beyond the S4 model, and we hope our work lays the foundation to inspire future work along this valuable future direction.

W2.1 Comparison with the linear interpolation method.

Thank you for your advice. We have included simple and standard imputation methods such as mean, forward fill (Ffill), and linear decay interpolation in their original forms. To incorporate your suggestion, we have added linear interpolation methods to the following table, which presents experiments conducted on four datasets with $r = 0.24$ and a horizon window of 96. The results indicate that, linear interpolation’s performance is inferior to both our proposed approach and the decay method in most cases.

We thank the reviewer for the thoughtful and detailed review. Also, we appreciate that the reviewer acknowledges our prototype bank presents an interesting practical way to tackle missing values. We address the raised concerns below.

W1.1 Justification on the comparison baseline selections.

We thank the reviewer for the question. The reasons that we did not directly compare S4M with irregularly sampled time-series methods are multifold: First, here, we focus on block-based missing patterns where the observed values occur at consecutive time points (see our Fig. 4). The irregularly sampled time series problems typically don’t directly consider the properties of such missing patterns. Second, efficiency is an essential consideration for our work, as stated in lines 51-52 in our original submission. Compared with S4, the suggested ODE and graph-based irregularly sampled methods are computationally costly. For instance, experiments with Grafiti on the Traffic and Electricity datasets often result in out-of-memory (OOM) errors in most settings. Similarly, CRU is extremely slow due to its iterative computations over variable dimensions. Detailed comparisons and computational costs for these methods are provided in the table below

In response to your comment, we have two additional SOTA methods for irregular time-series forecasting, Grafiti and CRUD, for comparison. However, due to the high computational costs and significant memory requirements of methods designed for irregularly sampled time-series data, our experiments were restricted to the smaller-scale ETTh1 dataset, as detailed below. The results show that ODE-based models perform suboptimally in this context.

Dear reviwer, thank you for acknowledging that our responses have addressed your concerns in part. We are delighted to engage further and address your remaining questions.

Clarifications on Time-Series Setting: In our revision, we have made explicit the distinction between irregularly sampled time-series and our focus on the missing data setting. This distinction is now emphasized both in the abstract and the instructions, ensuring clarity for all readers. Additionally, to guide understanding, we have highlighted Figure 4, which comprehensively illustrates this setting. Thanks for your suggestions. We hope this addresses your concern.

Focus on S4-based Models: We respectfully request the reviewer to consider that the primary focus of our study is on S4-based models, as stated in our introduction. S4 was selected due to its efficiency, which has been well-documented in the literature. We further demonstrated this efficiency in response to your W2.2 (Part 3), where we compared S4M against the suggested methods and many others. For your convenience, we have included the table from our response, which underscores that S4M is significantly more efficient than the three suggested methods.

We hope the reviewer agrees with the trade-offs inherent in various backbone architectures (no free lunch) and recognizes improving S4 is central to this study, reflected in the title and justified in our introduction.

Response to Interpretation of Reviewer Comments: We appreciate your clarification regarding your request. Initially, we interpreted your comments as two distinct questions, leading us to provide detailed responses in W2.1 (Part 2) and W2.2 (Part 3) separately.

Experimental Comparison with Your Suggested Methods: We appreciate your thoughtful feedback. We conducted experiments incorporating your suggested methods with the mean interpolation approach (specified in W2.2, Part 3). Furthermore, we conducted experiments combining PatchTST with linear interpolation. The results are shown below. In most settings, S4M consistently demonstrated superior performance. Both linear and mean interpolation with PatchTST do not work well. Linear interpolation outperformed mean interpolation only on the ETTh1 dataset. For datasets exhibiting clear seasonality like electricity and traffic, linear interpolation may perform worse than mean interpolation.

We thank the reviewer for the thoughtful and detailed review. Also, we appreciate that the reviewer acknowledges our prototype bank presents an interesting practical way to tackle missing values. We address the raised concerns below.

1. What is the computational complexity of ATPM vs traditional approaches?

Thanks for the question, we provide the complexity analysis below. An empirical comparison of the computational costs of the S4M and other methods is given in the table below. S4M demonstrates superior efficiency in both training and inference compared to other baselines.

We also provide the complexity analysis for the core steps in bank writing and reading operations.

• Bank Writing: Given $B$ training data points in a batch, each with $L$ segments, as we detailed in Algorithm 1 on the page, the core operations in writing the prototype bank involves four main steps: (1) randomly selecting $n$ out of $B \cdot L$ representations, which has a complexity of $O(n)$; (2) computing similarity between $n$ selected representations and $s$ centroids of dimension $R$, leading to $O(n \cdot s \cdot R)$; (3) selecting the maximum similarity for each representation, which requires $O(n \cdot s)$; and (4) updating the clustering via a FIFO-based mechanism, with a cost of $O(n)$. Assuming standard operations for similarity computation and FIFO updates, the overall computational complexity is dominated by $O(n \cdot s \cdot R)$, reflecting the influence of the embedding dimension $R$ and the number of centroids $s$
• Bank Reading: Given $B$ training data points, each with $l$ segments, the procedure involves the following steps: (1) for all $B \cdot l$ segments, compute cosine similarity with $s$ centroids, resulting in a complexity of $O(B \cdot l \cdot s \cdot R)$, where $R$ is the embedding dimension; (2) select the top $K$ centroids for each segment, which takes $O(B \cdot l \cdot s)$ using a partial sort; (3) normalize the similarity values for these $K$ centroids using an exponential function, costing $O(B \cdot l \cdot K)$; and (4) compute the weighted average of these $K$ centroids, which also takes $O(B \cdot l \cdot K \cdot R)$. The overall computational complexity is dominated by $O(B \cdot l \cdot s \cdot R)$, primarily driven by the initial cosine similarity calculations.

4. More comprehensive evaluation.

The datasets for empirical evaluation are widely used in time series forecasting literature [2,3]. We selected these four datasets because they exhibit significant variation in size, number of variables, and the presence or absence of seasonality. We consider cases with block missing patterns in regularly sampled time series, where the observed values occur at consecutive time points ((see our Fig. 4). This structure enables the design of an informative representation $o_t$, which is crucial for capturing temporal dependencies effectively.

Following your comment, we included the real-world USHCN climate dataset [1] in our analysis. We set the lookback window of size 96 and a horizon window of size 96. The results further confirm that S4M outperforms other methods on the real-world dataset.

Reference:

[1] Long-term daily climate records from stations across the contiguous united states, 2015.

[2] Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021

[3] CARD: Channel aligned robust blend transformer for time series forecasting[C]. The Twelfth International Conference on Learning Representations. 2024.

5. How sensitive is performance to prototype bank initialization?

In our experiment, we found the performance is insensitive to the initial cluster configuration, as the clusters are updated continuously throughout the training process. To provide evidence, we presented the experimental results on four datasets using different cluster numbers for initialization, as shown below. In practice, we recommend using 3 to 5 clusters for initialization or determining the optimal number of clusters based on the within-cluster sum of squares.

2. Theoretical justification.

Thank you for the questions. The long-term dependency in S4 is achieved using the HiPPO matrix $A$, as shown in (1). This long-term dependency is evident because the current state can be expressed as a convolution of previous states, with the convolution kernels being polynomial in the HiPPO matrix $A$. Our dual-stream processing maintains the HiPPO structure, as described in line 273 of our manuscript. Specifically, the current state remains a convolution of previous states, with the convolution kernel being polynomial in the HiPPO matrix, just as in S4. Therefore, the theoretical results for S4 remain valid in our approach.

3. Additional ablation studies.

Thank you for your suggestion on additional ablation studies. We have included an ablation study for the first module, ATPM. In this study, we compare S4M with and without ATPM, highlighting the improvements brought by ATPM, particularly on the Traffic and ETTh1 datasets. The following experiments were conducted under the same settings as those in the ablation studies presented in the paper.

Dear reviewer iDLq,

We sincerely thank you for your thoughtful review and valuable feedback. We have carefully addressed each of your questions and provided detailed responses in the rebuttal. We hope to have resolved all your concerns. If you have any further comments, we would be glad to address them before the rebuttal period ends. If our responses address your concerns, we would deeply appreciate it if you could consider raising your score. Your recognition for our novel work means a lot. Thanks again for your time and effort in reviewing our work.

Regards, S4M authors

Dear Reviewer iDLq,

You have indicated that submission 10665 is marginally below acceptance. The authors have provided a detailed response.

Please indicate the extend to which their response addresses your concerns and explain your decision to update (or not update) your score.

All the best,

The AC

We thank the reviewer for the thoughtful and detailed review. Also, we appreciate that the reviewer acknowledges our proposed S4M model is innovative. We address your concerns below.

W1. Source code.

Thanks for the question, the code can be found at the anonymous link.

W2 & Q1. Computational costs of S4M model and its comparison to the baseline models in terms of training and inference time?

Thank you for your valuable feedback. S4M demonstrates superior efficiency in both training and inference compared to other baselines. To evaluate the computational cost of S4M, we conducted experiments using the Electricity dataset under the highest missing ratio setting. The experiments were performed with a batch size of 16 and a hidden size of 512.

We observe that S4M (ours) achieves a lower FLOPS value compared to other SOTA transformer-based methods, including Grafiti. Also, S4M (ours) is similar to the S4-based methods. The results confirm our motivation to focus on S4-based architecture, given their efficiency (see lines 51-52 of our original submission). Furthermore, S4M demonstrates shorter training times than CRUD, PatchTST, BiTGraph, and BRITS. For inference, S4M also outperforms CRUD, PatchTST, and BRITS, making it a more efficient choice for both training and inference. (For clarification, "OOM" in the tables refers to "Out-of-Memory.")

Q2. How does the dual stream processing impact the model's ability to capture temporal dependencies?

Thank you for the question. The motivation behind the dual-stream processing is to take advantage of the strengths of S4—namely, its ability to capture long-term temporal dependencies and its computational efficiency—when addressing block missing patterns in time series. The long-term dependency in S4 is achieved through the use of the HiPPO matrix $A$ as shown in (1). In our dual-stream processing, we build on this structure by incorporating $o_t$ (the representation from a shorter look-back window) instead of $u_t$ (the observation of only at the current time point) into the model. The term $o_t$ inherently captures additional temporal dependency information.

Q3. Can the authors confirm whether they implemented these baseline methods using official code or leveraged existing unified Python libraries.

Thanks for checking the experiment details. For a fair comparison, we used the official implementation for all baselines.

Dear reviewer UFK6,

We sincerely thank you for your thoughtful review and valuable feedback. We have carefully addressed each of your questions and provided detailed responses in the rebuttal. We hope to have resolved all your concerns. If you have any further comments, we would be glad to address them before the rebuttal period ends. If our responses address your concerns, we would deeply appreciate it if you could consider raising your score. Your recognition for our novel work means a lot. Thanks again for your time and effort in reviewing our work.

Regards, S4M authors

Dear Reviewer UFK6,

The discussion period is almost over, so please read the response of the authors of submission 10665 to your review.

Does their response address your concerns? Will you modify your score? Please explain your decision.

All the best,

The AC