MuSiCNet: A Gradual Coarse-to-Fine Framework for Irregularly Sampled Multivariate Time Series Analysis

Thank you for your careful review and helpful suggestions. We note your main concerns about the writing clarity and the insight on irregular relativity. In response, we have made edits to improve readability and further clarified our insights. At the same time, other reviewers have recognized our writing and insights, with comments such as “The paper is clearly written and well-structured” (Reviewer ewtP), “This is a high-quality paper” (Reviewer T9q8), and “The paper generally explains its methodology well, with clear explanations of the key concepts and results.” (Reviewer KxEq). We hope you will also see these strengths in our work. Below is our detailed response:

W1: The insight of irregular relativity. & Q1: The generation principles for the hierarchical set and its advantages.

A1: Here, we address W1 and Q1 together.

We rethink the causes of multivariate irregular time series generation and realized that, due to various external forces or interventions mentioned in the paper’s real-world scenarios, sensors are unable to sample uniformly, resulting in ISMTS data.

However, when we zoom in on the time scale, as shown in Figure 1 of the paper, larger time windows are more likely to contain real sampling values, forming relatively regular series, which helps mitigate the issue of irregular sampling because regularly sampled time series have been widely studied and demonstrate to be easier for models to learn.

The advantage of coarse-grained regular sequences is that they ease the learning difficulty caused by irregular sampling as shown above and provide broad-view temporal information. However, fine-grained details that may be missed at this level still need to be captured from finer-grained series, which is why we adopt a hierarchical learning approach. Moreover, it’s important to emphasize that our hierarchical learning iteratively refines the learned representations rather than simply merging them.

W2: The definition of the symbols is poorly articulated.

A2: In the revised version of the paper, we have updated our symbol notation, highlighted in blue, including but not limited to the dimensions and representation of symbols.

W3: The dimensions of the variables Q, K, A, and C.

A3: Followed by the definition $X_n$ in PROBLEM FORMULATION subsection, the dimensions of the variables can be formulated as $Q \in \mathbb{R}^{|\tau| \times |\tau|}$, $K \in \mathbb{R}^{|\tau| \times T}$, $A \in \mathbb{R}^{|\tau| \times T}$, and $C \in \mathbb{R}^{D \times D}$, where $|\tau|$ denotes the length of reference points, $T$ denotes the length of observed timestamps, and $D$ denotes the number of observed variables in ISMTS.

Q2: The reasons that the proposed framework does not stand out on the P19 dataset?

A4: According to the no free lunch theorem, it is natural that no single method excels universally. While our performance may not be the best on certain datasets, our non-SOTA results remain among the top-tier. Moreover, our model demonstrates superior average performance across the three downstream tasks.

Specifically, as described in subsection 4.1 Time Series Classification, “For the P19 dataset, while our performance is competitive, MuSiCNet stands out due to its lower time and space complexity compared to ViTST. ViTST converts 1D time series into 2D images, potentially leading to significant space inefficiencies due to the introduction of extensive blank areas, especially problematic in ISMTS.” and the computational complexity is shown in subsection4.5 Ablation Analysis and Efficiency Evaluation.

Even so, we are continually working to improve the performance of our model further.

Thank you for your suggestions. We noted your concerns regarding Writing, the Discussion of Limitations, and Insights from Simulations. Below, we provide specific responses to each of your points. We also noticed that most of your reviews are suggestions, such as adding a hyperparameter sensitivity analysis or including toy data to validate the model’s effectiveness. Therefore, we hope you can improve your evaluation of our paper. If any questions remain or you feel unable to increase the paper’s score, please feel free to ask, we would be glad to clarify.

Regarding your comments on writing, our responses are as follows:

1. The term ISMTS, which we use throughout our paper, is common in the field, not coined by us. For example, the paper by Yalavarthi et al. (2024) also uses "irregularly sampled multivariate time series," and this term appears in their Figure 1. Moreover, though we did not use a specific figure to introduce ISMTS, we define ISMTS and its key characteristics and draw Figture 1 to facilitate readers' understanding in the first paragraph of the introduction.
2. In real-world applications, sensors cannot record continuously, so the data collected is necessarily discrete. “Missing ratio” is a common term that describes the percentage of missing observations in a time series (similar to “sparsity” in Yalavarthi et al. (2024), as you noted). Although the “missing ratio” assumes regular sampling, it is widely used to describe ISMTS data characteristics. This term is also explained on the dataset website referenced in our paper and many other related work.
3. Our paper focuses on real-world datasets with substantial missing ratios, which are influenced by various external forces or interventions, leading to irregular sampling. Moreover, the effectiveness of our model can be demonstrated by the performance of the downstream tasks performed on datasets with high missing ratios and is unnecessary to use toy data.
4. We have clarified in the legend of Figure 2 that gray arrows indicate the direction of data flow and the subsection “Encoder-Decoder Framework” also detailly introduces the whole model processing. Additionally, we corrected the error in the subsection “Correlation Extraction”, where we mistakenly referred to Fig.2(c) as Fig.2(b), and have updated this in the revised version.

Based on the discussion of limitations, we conduct sensitivity analysis experiments in subsection F.3 Parameter Analysis in the appendix as follows

“To analyze the hyper-parameters sensitivity, we conducted the experiments for $\lambda_1$, $\lambda_2$, and $\lambda_3$ with grid search. Due to the closer relationship between the hyper-parameters of the adjustment term and the contrastive learning term, i.e., $\lambda_1$ and $\lambda_2$, we jointly analyzed $\lambda_1$ and $\lambda_2$ while separately analyzing the hyper-parameter of the downstream task $\lambda_3$, as illustrated in Figure 4 and 5. From Figure 4, we can find that the adjustment term plays a greater role compared to the contrastive learning term. This phenomenon matches our motivation, where the coarse-to-fine strategy can effectively alleviate the difficulty of representation learning on ISMTS caused by inconsistent sampling rates. In addition, when $\lg \lambda_1$ and $\lg \lambda_2$ take values around 2 and -2, respectively, our MuSiCNet can perform well. From Figure 5, we can find that our MuSiCNet becomes effective with large $\lambda_3$. This indicates that more effective representations will be captured when utilizing downstream tasks, matching the general insight.We also noticed that it becomes less sensitive when $\lg \lambda_3 \ge -1$. Its suitable range may be located at $\left[1e1, 1e2\right]$.“

Regarding the question on Insights from Simulations, the goal of our paper is to learn effective representations of ISMTS, which can be validated through good performance on downstream tasks, demonstrating the quality of the learned representations. Additional toy data is not necessary to illustrate this point. Additionally, the frequently mentioned Yalavarthi et al. (2024) paper also does not employ simulation experiments.

According to the minor suggestion, we have changed the paper title to “A Gradual Coarse-to-Fine Framework for Irregularly Sampled Multivariate Time Series Analysis.”

We sincerely thank the reviewer for their careful reading and insightful comments. Your feedback primarily focuses on the performance in the forecasting task, the details of multi-scale training, and the experimental validation of coarse-grained series. We have addressed these points and hope our responses resolve your concerns, encouraging you to consider raising your rating of our work. If you have any further questions, please feel free to ask. Below are our detailed responses:

Q1: In the TIME SERIES FORECASTING task, the model is not SOTA on 2 out of 3 datasets. Further analysis is needed. Also, model complexity, training, and inference costs compared to baselines should be provided.

A1:

1. According to the no free lunch theorem, it is natural that no single method excels universally. While our performance may not be the best on certain datasets, our non-SOTA results remain among the top-tier. Moreover, our model demonstrates superior average performance across the three downstream tasks. Specifically, as described in subsection 4.3 Time Series Forecasting, the model achieving SOTA results in the forecasting task is specifically designed for that purpose, incorporating priors designed for forecasting. In contrast, our MuSiCNet remains competitive without relying on task-specific priors. Even so, we are continually working to improve the performance of our model further.

2. We conduct experiments on the P12 dataset with a batch size of 50. Empirically, our model MuSiCNet requires 0.240s per batch with 4.2GB memory usage, compared to ViTST’s 2.196s and 40.2GB, Raindrop’s 0.124s and 4.8GB, MTGNN’s 0.1967s and 4.2GB, and DGM$^2$-O’s 0.313s and 9.1GB. Our model achieves significantly lower time complexity than the other methods and uses much less memory than ViTST, which also performs well on classification tasks. We will add this in section 4.5 “Ablation Analysis and Efficiency Evaluation”.

Q2: Regarding the multi-scale learning method, it's unclear if the number of training epochs is the same as without multi-scale training. Also, the relationship between sample numbers at different scales needs clarification.

A2: In our setup, the training epochs remain the same and are not affected by the multi-scale design.

In Appendix subsection F.1 MUSICNET PARAMETERS, we explain the selection method for the number of scales. This approach does not involve manual selection. According to the observed timestamps in each dataset, we ensure that in the (L-1)-th layer, most windows contain at least one sampling point, while the L-th layer represents the original data layer.

Q3: Experimental validation of coarse-grained series.

A3: From the sensitivity analysis in Figure 5, we can find that compared to $\lambda_2$, the hyperparameter of the adjustment term, i.e., $\lambda_1$, plays a greater role, which aims to utilize broader temporal information (larger scale) to guide the effectiveness of detailed temporal information (smaller scale), to boost the representation learning. If $\lambda_1$ did not become a key role, then our MuSiCNet would definitely lose the guidance of broad-view temporal information. However, this did not happen. Moreover, Scale 1 in Figure 1 clearly illustrates what the broad-view trend information. Therefore, the above phenomenon can indirectly validate that the coarse-grained series can help the representation capture broad-view temporal information.

Thank you for your positive feedback in the Strengths section. We noticed that your concerns primarily revolve around related work, certain writing approaches, and experimental details. We have addressed these concerns and revised the paper accordingly. We hope our efforts will enhance your recognition of our work, and we would greatly appreciate it if you could consider raising your score for our paper. If you have any further questions, please feel free to ask—we are more than happy to provide additional clarification. Below are our detailed responses.

W1: The paper's summary and comments on current research work are not accurate and rigorous enough.

A1: 1. When we state, "Most existing methods treat ISMTS as synchronized regularly sampled time series with missing values," we use "most" to acknowledge that not all papers take this approach; otherwise, we would have used "all." 2. In the sentence, "Neglecting that the irregularities are primarily attributed to variations in sampling rates," this phrase directly follows the previous statement 1 above as part of the same sentence, separated by a comma. Thus, we mean that methods treating ISMTS as synchronized regularly sampled time series with missing values often overlook this aspect. Given that many studies do approach ISMTS in this way, but our own method does not follow this method, we found it necessary to briefly point out this potential limitation in the abstract.

As you mentioned, irregularly sampled time series indeed arises from varying sampling frequencies. However, many existing methods either fail to recognize this or do not leverage this information in their modeling. Our approach aims to effectively utilize this characteristic to better model ISMTS.

1. Here, we refer to the fact that many existing ISMTS analysis models are designed for specific tasks, such as prediction, classification, or imputation. We have updated our manuscript to include additional references on this. In contrast, our work focuses on learning a good representation that can adapt to multiple downstream tasks.

W2: The language used in the paper is not precise enough.

A2: The phrase "irregularity is essentially relative in some senses" reflects our perspective on ISMTS based on varying sampling rates. Another perspective considers ISMTS as regularly sampled time series with missing values, attributing the irregularities to missing data. Given these differing viewpoints, it would be inaccurate to state outright that irregularities are relative. However, since we already use the term "mitigate," we have decided to remove "to some extent.”

W3: The paper lists the model's versatility across tasks as one of its three main contributions.

A3: We emphasize here that our primary focus is on learning a good representation, enabling our model to be not limited to a specific analysis task but to serve as a task-general framework for ISMTS analysis.

While some existing works are capable of handling multiple tasks, such cases are relatively rare, and the tasks are addressed differently. For instance, ContiFormer performs classification and event prediction, IVP-VAE focuses on classification and forecasting, and Latent ODE tackles interpolation, extrapolation, and some dataset-specific tasks.

Therefore, we believe that achieving multiple tasks remains one of the notable contributions of our work at this stage.

W4: Figure 2 (a) appears to have poor readability.

A4: (a) represents the overall framework of our model. Based on the title of (b), we can see that it corresponds to the CorrNet Encoder in (a). (c) illustrates the Frequency Correlation Matrix, which is the method used in (b) to compute the inter-series relationships in the Correlation Matrix.

$X$ denotes the input, and the representation $r^{(L)}$ learned by the final layer, $L$, of the CorrNet is considered the model's output, as all subsequent downstream tasks are performed on $r^{(L)}$.

W5: The paper states, "MuSiCNet stands out due to its lower time and space complexity compared to ViTST," yet there is no theoretical derivation or experimental results provided in the paper to support this claim.

A5: We conduct experiments on the P12 dataset with a batch size of 50. Empirically, our model MuSiCNet requires 0.240s per batch with 4.2GB memory usage, compared to ViTST’s 2.196s and 40.2GB, Raindrop’s 0.124s and 4.8GB, MTGNN’s 0.1967s and 4.2GB, and DGM$^2$-O’s 0.313s and 9.1GB. Our model achieves significantly lower time complexity than the other methods and uses much less memory than ViTST, which also performs well on classification tasks. We will add this in section 4.5 “Ablation Analysis and Efficiency Evaluation”.

W6: How were they adapted and applied to irregular time series data?

A6: We share the same setting with [1], so we directly used their results. And they did not mention how they were adapted.

W7: There is a spelling error in the Conclusion section.

A7: We carefully review the entire paper and correct spelling errors.

---

Q1: How to accurately understand “irregularity is essentially relative in some senses”?

A8: This means that, to some extent, irregular sampling can be transformed into regular sampling. Specifically, as mentioned in our paper: "With sampling rates artificially determined from low to high, an irregularly sampled time series can be transformed into a hierarchical set of relatively regular time series from coarse to fine."

Q2: How sensitive is MuSiCNet to the choice of hyperparameters, such as the number of scales and the masking ratio?

A9: In Appendix subsection F.1 MUSICNET PARAMETERS, we explain the selection method for the number of scales. This approach does not involve manual selection. According to the observed timestamps in each dataset, we ensure that in the (L-1)-th layer, most windows contain at least one sampling point, while the L-th layer represents the original data layer.

As for the masking ratio, we set it uniformly to 10%, as mentioned in subsection F.1. This is because the missing rate in ISMTS is already significantly high. Therefore, we follow the smallest masking ratio in MAE [1], which is 10%, and apply it to all the experiments in this paper.

Consequently, we did not conduct sensitivity analysis for these two parameters. However, we conduct sensitivity analysis experiments in subsection F.3 Parameter Analysis in the appendix as follows

“To analyze the hyper-parameters sensitivity, we conducted the experiments for $\lambda_1$, $\lambda_2$, and $\lambda_3$ with grid search. Due to the closer relationship between the hyper-parameters of the adjustment term and the contrastive learning term, i.e., $\lambda_1$ and $\lambda_2$, we jointly analyzed $\lambda_1$ and $\lambda_2$ while separately analyzing the hyper-parameter of the downstream task $\lambda_3$, as illustrated in Figure 4 and 5. From Figure 4, we can find that the adjustment term plays a greater role compared to the contrastive learning term. This phenomenon matches our motivation, where the coarse-to-fine strategy can effectively alleviate the difficulty of representation learning on ISMTS caused by inconsistent sampling rates. In addition, when $\lg \lambda_1$ and $\lg \lambda_2$ take values around 2 and -2, respectively, our MuSiCNet can perform well. From Figure 5, we can find that our MuSiCNet becomes effective with large $\lambda_3$. This indicates that more effective representations will be captured when utilizing downstream tasks, matching the general insight.We also noticed that it becomes less sensitive when $\lg \lambda_3 \ge -1$. Its suitable range may be located at $\left[1e1, 1e2\right]$.“

[1] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Q3: How does the computational complexity of MuSiCNet compare to other ISMTS analysis methods?

A10: Please refer to our response in Weakness 5.

Thank you for recognizing our work and providing insightful feedback. Your comments mainly focus on the need for additional analysis, clarifying the relationship between our work and mTAND, and revisiting Warpformer. In our responses, we have highlighted the sections where these analyses are located, revised some text to emphasize the fundamental differences between our work and mTAND, and included a review of Warpformer along with adding it as a baseline for comparison. We hope our responses address your concerns and improve your evaluation of our paper. If you have further questions, please feel free to reach out. Below are our detailed responses.

W1: There are some key missing analyses:

a) Comparing multi-scale modeling with single-scale modeling to quantify the benefits. An assessment of the optimal number of scales, along with the computational cost added by each scale.

A1: We have already conducted a comparison between multi-scale and single-scale approaches in Table 6 of subsection 4.5. The third row in Table 6 represents the single-scale results, which we found to be inferior to the performance of the full model.

Searching for the optimal number of scales is time-consuming and labor-intensive. In Appendix subsection F.1, MUSICNET PARAMETERS, we explain that our approach avoids manual selection. Based on the observed timestamps in each dataset, we ensure that in the $(L-1)$-th layer, most windows contain at least one sampling point, while the $L$-th layer represents the original data layer. Under this setup, we are able to achieve good performance while saving computational resources.

b) Evaluating the LSP-DTW correlation matrix against inter-variable self-attention, as seen in prior work (e.g., Warpformer), to confirm its advantage over attention-based correlation modeling.

A2: In Subsection 4.4, Correlation Results, we conduct a comprehensive comparison with prior work to demonstrate the necessity, effectiveness, and efficiency of the correlation matrix.

The inter-variable self-attention in Warpformer can be viewed as a learnable correlation matrix derived through attention, as shown in the 5th row of Table 4, which does not perform well in our model. While determining the best correlation calculation method for all models is beyond the scope of this work, our focus is on identifying the most suitable approach for our specific model.

c) The paper mischaracterizes Warpformer as a multi-scale model for regularly sampled time series. And it should be included as an important baseline to be compared with.

A3: We mistakenly cite a paper by an author with the same surname. We have removed this reference from the current sentence and revisited Warpformer appropriately in the Related Work section as follows

“As far as we know, [1] and [2] are among the earlier works on multi-level ISMTS learning. [1] addresses multi-resolution signal issues by distributing signals across specialized branches with different resolutions, where each branch employs a Flexible Irregular Time Series Network (FIT) to process high- and low-frequency data separately. [2], on the other hand, is a transformer-based model that stacks multiple Warpformer layers to produce multi-scale representations, combining them via residual connections to support downstream tasks. These works typically focus on either specific tasks or particular model architectures. In contrast, our design philosophy originates from ISMTS characteristics rather than being tied to a specific feature extraction network structure. Warpformer emphasizes designing a specific network architecture but involves high computational costs and requires manually balancing the trade-off between the number of scales and the dataset. These are challenges that our MuSiCNet avoids entirely.”

Since Warpformer uses a different benchmark than the one followed in our experiments, we reproduced Warpformer's classification results on the benchmark dataset we used. The results are as follows and have been included in the paper.

[1] Singh B P, et al. Multi-resolution networks for flexible irregular time series modeling (multi-fit)[J]. arXiv preprint arXiv:1905.00125, 2019.

[2] Zhang J, et al. Warpformer: A multi-scale modeling approach for irregular clinical time series[C]//Proceedings of the 29th KDD. 2023: 3273-3285.