GITAR: GENERALIZED IRREGULAR TIME SERIES REGRESSION VIA MASKING AND RECONSTRUCTION PRETRAINING

We appreciate the reviewer's insightful comments. Please see below our responses.

W1: Although some modules are integrated, the motivation is missing in the paper, such as why this specific masking approach is useful over others for handling irregular time series. The authors are suggested to give more theoretical discussions and an intuitionistic example to strongly motivate the work. Note that in terms of the results reported in Table 3, the PrimeNet method is able to obtain comparable performance than that of the proposed model in most cases.

A1. The masking aims to preserve the original irregularity patterns which also contain useful information that can be learnt, by keeping dense regions dense and sparse regions sparse. Additionally, dense regions contain richer information, allowing the model to better capture the underlying temporal patterns. In contrast, making dense regions sparse results in the loss of critical information, leading to less effective learning of temporal dynamics. We will include this in the introduction to motivate the work. Further, we will add the empirical results in our ablation study as in the table below to highlight the effective masking strategy we have over random masking.

To provide an intuitive example, consider a clinical ICU setting where data collection is often limited due to resource constraints, leading to variability in sampling rates and differences in the number of sensors deployed across ICUs. For instance, one ICU may have frequent sampling of vital signs from ten sensors, while another might collect data less frequently from only six sensors. Additionally, ICU data are typically irregularly sampled due to interruptions in monitoring or patient-specific factors. These variations pose significant challenges for existing methods: regular time-series models struggle with irregular sampling, while most irregular time-series methods are not designed to generalize across domains with differing numbers of variables or sampling rates. To address these limitations, we propose a novel self-supervised learning (SSL) framework that simultaneously tackles irregular modeling and generalization, enabling effective transfer of representations between such heterogeneous ICU settings.

Our model still shows slightly improvements over PrimeNet in Table 3, and more importantly it shows significant improvements in Table 2. Additionally, with longer time series, our model demonstrates increased improvement over PrimeNet, as shown in Figure 4.

W2: What is the distinction between the two critical research questions the authors propose? They appear to address the same issue, as a robust self-supervised learning framework inherently implies the ability to adapt to different domains. Could the authors clarify how these questions are fundamentally different?

A2.We thank the reviewer for pointing out the potential overlap in the two research questions. We will revise them in the manuscript as follows:

• How can we effectively model latent representations of irregular time series, overcoming the challenges posed by their inherent variability and lack of temporal structure?
• How can we improve the generalization of learned representations to facilitate their application to unseen domains with differing characteristics, such as varying sampling patterns or variable sets?

The first research question focuses on modeling the latent representation of irregular time series, addressing a key challenge that most self-supervised learning (SSL) methods face due to the inherent variability and lack of temporal structure in these datasets. This question is centered on designing effective mechanisms, such as time embeddings and masking strategies, to represent irregular time series in a way that captures both local and global temporal dependencies.

The second research question focuses on improving the generalization of these learned representations to enable their application to unseen domains with differing irregular characteristics. This involves ensuring that the representations can adapt to datasets with varying sampling patterns, missing data rates, and variable sets, making them robust across a wide range of real-world scenarios.

W3. The description for Fig1 lacks clarity, making it difficult to understand the intended meaning. It is unclear whether “v1-v3” represents different variables, tasks, or something else. Additionally, if the figure is meant to illustrate a different domains/transfer learning issue, a more detailed explanation would help readers grasp the specific context and purpose of the comparison.

A3. the v1-v3 represent different variables. We will include more details in the figure caption and text, and label the variables in the figure more clearly.

W4. From the method and the experimental part, I cannot find how the proposed model learns and captures the claimed local semantics (i.e., irregular patterns) and global temporal dependencies. Could the authors provide more detail on how these aspects are addressed?

A4. The model captures local semantics by first dividing each time series into patches and mapping each patch to a latent vector consisting of local information. Within each patch, continuous time embeddings are used to preserve the temporal irregularities, which are then processed by a Gated Recurrent Unit (GRU) network. The GRU effectively learns the irregular patterns and dependencies within each patch, outputting a single latent representation for the patch.

For global temporal dependencies, we use a transformer-based architecture to integrate the latent representations from all patches, capturing the temporal dynamics among the sequence of patches. We will revise the manuscript to include these details in the methods section and reference the relevant experimental and ablation studies that demonstrate the effectiveness of these modules.

W5. In addition to self-supervised learning, there are numerous deep generative models for irregular time series, including GANs and diffusion models [1-3]. The authors have not discussed or compared with these types of methods. Besides, the metrics used in Table 3 are not clearly specified.

A5. We appreciate the reviewer’s suggestion. We focus on self-supervised learning and E2E supervised compared to baselines that fall under this scheme, as outlined in [1,2]. While GANs and diffusion models also process irregular time series with specific mechanisms, they are based on generative modeling approaches, and fall outside the scope of this work.

We will specify the evaluation metrics used in Table 3 (MSE for the forecasting task) and revise the visualization and explanation to make the table clearer.

[1]Chowdhury, Ranak Roy et al. “PrimeNet: Pre-training for Irregular Multivariate Time Series.” AAAI Conference on Artificial Intelligence (2023).

[2]Zhang, Weijiao et al. “Irregular Multivariate Time Series Forecasting: A Transformable Patching Graph Neural Networks Approach.” International Conference on Machine Learning (2024).

Also, while the authors claim “across PhysioNet to MIMIC,” in the caption of Table 3, it does not seem to present results specifically for the PhysioNet dataset.

Table 3 focuses on generalization experiments where PhysioNet is used as the pretraining dataset, and the downstream evaluation datasets include MIMIC-III, Human Activity, and USCHN. The results on physionet itself (pretraining and evaluation within the same dataset) is listed in Table 2. We will revise the manuscript to clarify this experimental setup and improve the description of the results in both tables to avoid any confusion.

Q4. For A4: My question was not about using the framework to explain how local semantics (i.e., irregular patterns) and global temporal dependencies are learned and captured. I was asking how the experiments demonstrate or provide evidence that the model has indeed learned these local semantics and global dependencies.

A4. Local and global information correspond to specific segments of the time series. In our approach, each time series is divided into N consecutive patches, which capture partial information and are referred to as local information, independent of the model structure. These patches are then mapped into latent representations that focus on local details. A subsequent transformer module aggregates these representations into a global representation, which captures long-range temporal dependencies through attention mechanisms. Additionally, our ablation study shows that removing the patches or the global aggregation module leads to performance degradation, underscoring the importance of the information learned. We understand the reviewer’s request for experimental evidence in this regard. However, at present, there are no established theoretical methods to quantify temporal dynamics in the latent representations, nor to directly quantify the relationship between the original time series patches and the latent representations. We would greatly appreciate any suggestions from the reviewer on how to address this and demonstrate these aspects more effectively.

We appreciate the reviewer's insightful comments. Please see below our responses.

W1 & Q3 & Q4 My strongest critique point is the paper writing. The paper on the hand contains a lot of self-advertising sentences which distracts from the "real content". Example: …

A1. We will update the revised manuscript to simplify and make the method description more clean and precise.

The decoder: How does it look like? Is it an MLP? Just one layer?

A1.1 The decoder is a simple MLP with layers designed to reconstruct the original time series at masked positions. We also tried other decoder structures like symmetric encoder-decoder structures and lightweight decoders, which showed similar performance to MLPs. We chose MLPs for their simplicity and computational efficiency. The loss is computed only for the masked positions, ensuring a focus on reconstructing missing data effectively.

What does the following mean: "This pre-trained reconstruction model R will then be fine-tuned for downstream tasks such as ,,," How exactly does that look like? How does the fine-tuning look like? How do you do for example forecasting? With a forecasting MLP- head as in PatchTST? Or do you just assume that the full forecasting horizon is masked and you apply your normal reconstruction?

A1.2 Fine-tuning procedure: Then during the fine-tuning for the specific forecasting task, we fine-tune the encoder only on historical observed data while adding a forecasting MLP head. This head predicts the future horizon values. The fine-tuning procedure is simple and modular, designed to adapt to downstream tasks such as forecasting and interpolation.

Forecasting Task: The whole process is divided into three stages. During the first pre-training stage, we train the model. In the second fine-tuning stage for forecasting tasks, we divide the time series into past observed time series as input and future values as the forecasting horizons, which serve as the labels. Similar to PatchTST, an MLP head is added for fine-tuning. We use the past observed values as input to optimize the model for forecasting future values using the MSE loss. In the third test phase, we evaluate the model by feeding it the past observations and assessing the forecasted values. We will clarify this in the revised version.

How do you mask? Randomly select a time-span? How long is the time-span? Does masking means zeroing out? I would formalize and explain such crucial points in detail as instead of just mentioning them shallowly.

A1.3 For the masking strategy, we preserve the original irregular pattern and then keep the relatively dense part stay dense and the sparse area stay sparse, as discussed in section Method. We select the time span of 3 data points with a fixed masking ratio within the grid of [0.1, 0.2, 0.3 ,0.4,0.5], setting the masked values to zero. We will include the detailed table for hyper-parameters tuning in the appendix.

I was also not able to have a look at the code to answer these questions by myself. I could unfortunately not find a link in the paper and also I could not see any supplementary material.

A1.4 We will provide the code upon acceptance.

W5. Interpolation experiments are performed only on Physionet dataset, and do not have MAE metric in the final comparison.

A5. We followed the experimental setup of the baseline studies and conducted interpolation experiments primarily on the PhysioNet dataset to ensure a consistent and fair comparison. While the current results focus on the metrics used in prior work, we acknowledge the importance of including Mean Absolute Error (MAE) as an additional evaluation metric. We will incorporate the MAE metric into the revised manuscript for a more comprehensive comparison.

W6. The results section on generalization capabilities is promising but limited to draw meaningful conclusions from. It is unclear why training is performed only on the Physionet dataset and tested on the rest. It would be worthwhile to see if the benefits of this generalization hold up for when the model is pretrained on other datasets or on a combination of them and tested on the remaining set. For example, train on MIMIC and test on the rest; train on MIMIC+Physionet and test on the rest would help to evaluate GITaR’s generalization capabilities.

A6. PhysioNet dataset has a relatively high number of variables and a low missing ratio compared to the other datasets. This makes it a strong candidate for pretraining. To address the reviewer’s concerns, we conducted additional experiments as suggested. Specifically, we trained the model on MIMIC-III and evaluated it on the other three datasets, as well as pretraining on a combined dataset of MIMIC-III and PhysioNet and testing on the remaining datasets. The results are summarized in the tables below.

W7.The code necessary for reproducing the reported results is not included in the submission.

Answer: We will provide the code in the camera-ready version.

In summary, our approach consistently outperforms other baselines across all settings, demonstrating strong generalization capabilities. While pretraining on MIMIC-III alone yielded competitive results, the dataset's high missing ratio presented unique challenges, limiting the extent of generalization compared to other configurations.In contrast, pretraining on MIMIC-III combined with PhysioNet showed improved performance, as the additional data from PhysioNet helped train a more generalized and robust representation.

These results highlight the importance of both the quality and quantity of pretraining data in achieving strong generalization across diverse datasets. We will include these additional experiments and results in the revised manuscript to provide a more comprehensive evaluation of our method’s generalization capabilities.

We appreciate the reviewer's insightful comments. Please see below our responses.

W1. The novelty of the patch schema is unclear and very much resembles recent work on T-PatchGNN.

A1. The patch schema that segments long sequences into fixed time horizons is the same as T-PatchGNN, but the processing for each patch is different in terms of channel dependent and independent processing. We will cite the original paper and clarify this in the revised Method section.

W2. It appears that the Global Temporal Encoding E^i is over individual time series and does not capture dependencies across different time series. It is unclear if/how these patterns and dependences are captured in the model.

A2. We intentionally designed the model to avoid capturing dependencies across channels in multivariate time series. This decision is rooted in our goal of enhancing the model's generalization ability to datasets with varying numbers of channels. By adopting a channel-independent learning strategy, the model learns representations for each channel (or individual time series) separately, enabling it to adapt to new test domains with different numbers of channels.

In contrast, approaches like T-patchGNN specifically capture channel dependencies to model the relationships among a fixed number of channels (N). While this strategy works well when the number of channels in the training and test datasets is consistent, it hinders generalization to datasets with a different number of channels (e.g., N+2). Such dependency-specific learning fails to adapt because the relationships among channels change with variations in channel count.

W3. The description of the decoder architecture is missing in the paper. It is understandable that the model transforms the irregular observations to a regular latent representation. It is unclear how the model then makes the predictions of time series variables at irregular times from this regular latent representation.

A3. The decoder consists of simple MLP layers designed to reconstruct the complete regular time series representation from the latent embeddings. While the model reconstructs the full time series, we only calculate the loss for the observed points during training and evaluate performance on the observed points during testing. This approach ensures alignment with the irregular nature of the original time series while leveraging the regularized latent representation for reconstruction.

The choice of MLP as the decoder architecture is supported by our experiments, where we compared it with other architectures, including a symmetric encoder-decoder structure and a lightweight encoder-decoder design. These alternatives showed similar performance to the MLP-based decoder, so we selected MLP for its simplicity and computational efficiency. We will add a detailed description of the decoder architecture and our rationale for choosing MLP in the revised manuscript.

W4. Evaluating the performance of models designed for regular time series (Re) against those specifically tailored for irregular time series (IR) on irregular time series regression tasks doesn’t add much value to the analysis. Perhaps authors intend to just highlight the importance of incorporating time explicitly in the model is important, which makes sense. However, including only 3 IR model baselines in the main results section is limiting. More IR baselines from the families: RNN-based (e.g., GRU-D [1]), Graph-based (e.g., Raindrop [2]), and ODE-based (e.g., ContiFormer [3]) should be considered in the experimental analysis for robust evaluation.

A4. We appreciate the reviewer’s suggestion to include additional IR model baselines, such as GRU-D, Raindrop, and ODE-based methods like ContiFormer. However, as demonstrated in prior works [1,2], GRU-D and ODE-based methods have been significantly outperformed by more recent approaches. Similarly, Raindrop has been shown to be less competitive compared to T-patchGNN in related studies.

To provide a robust evaluation of our method, we focused on comparing it with the most competitive baselines, as shown in Table 1 & 2. We will include these references and a brief justification for selecting our baselines in the revised manuscript.

[1]Shukla, Satya Narayan and Benjamin M Marlin. “Multi-Time Attention Networks for Irregularly Sampled Time Series.” ArXiv abs/2101.10318 (2020): n. Pag.

[2]Chowdhury, Ranak Roy et al. “PrimeNet: Pre-training for Irregular Multivariate Time Series.” AAAI Conference on Artificial Intelligence (2023).

Q1.1 In case of continuous time embeddings, why are specific sinusoidal terms chosen over other possible basis functions?

A1.1 Sinusoidal terms preserve periodic nature, smoothness, and continuity, functioning similarly to other potential basis functions like cosine functions for position inference. In time embeddings, the primary goal is to map the time index to a vector space while preserving the sequential order, which sinusoidal terms accomplish effectively. Furthermore, when the time indices are discrete, these embeddings can subsume the traditional positional embeddings used in transformers, evident in [1].

[1] Shukla, Satya Narayan and Benjamin M Marlin. “Multi-Time Attention Networks for Irregularly Sampled Time Series.” ArXiv abs/2101.10318 (2020): n. pag.

Q1.2 What benefits do $\omega$ and $\alpha$ provide in capturing irregular periodicities?

A1.2 $\omega$ and $\alpha$ control the frequency and phase in the time embeddings (Eq. xx), and this is simply a mapping from the original time index to a vector space, not a learning of irregular periodicities.

The subsequent step, the ITA mechanism, learns the irregular periodicities. Specifically, given a regular sequence of reference points $r$ such as 1 to 128, and the time embeddings of the original time index such as $\phi = [\phi_1, \phi_2, \ldots, \phi_{10}]$, the attention score will be calculated as:

This attention score clearly reflects the similarity in time; the nearer the time points, the higher the score. While some time steps in $\phi$ are missing due to the irregularity, the attention score will be manually set to 0 at those missing positions. Finally, the ITA embeddings are calculated by multiplying the attention scores with the original time series values:

The ITA embeddings therefore capture information only from the observed points, while automatically taking into account the salient information in the time series. We will break down the description in Section Method, to clarify this.

Q1.3 Is there theoretical support indicating they are optimal for this purpose? A1.3 We have shown the empirical results in Table 1&2, for example, patchTST and Crossformer utilize the conventional position embeddings as in the vanilla transformers, and show inferior performance compared to other approaches.

Q1.4 How do these embeddings behave under highly irregular sampling intervals?

A1.4 The time embeddings are designed to be robust to highly irregular sampling intervals, as shown in datasets such as MIMIC-III (see the Statistics Table), where the missing ratio and irregular sampling rates are high and our method still demonstrates promising result as shown in Table 1&2. This robustness stems from both the theoretical construction of the embeddings and their integration with the attention mechanism.

High irregularity in sampling intervals does not distort the embeddings because the embedding function only depends on the observed time points themselves, not on their spacing relative to other points. In detail, The continuous embeddings map each observed time point independently to a high-dimensional vector space, as shown in Figure.ITA_Mapping in Appendix B. Additionally, the irregular sampling intervals are explicitly addressed by the attention mechanism. The attention weights are computed only for observed points, ensuring that missing or sparse time points do not introduce noise into the representation.

We appreciate the reviewer's insightful comments. Please see below our responses.

W1: The theoretical foundations of the proposed methods are weakly articulated. The notation and assumptions behind the continuous embeddings and ITA mechanism are not thoroughly explained. Key concepts, such as the initialization and constraints of embedding parameters, are underexplored.

A1. The continuous time embeddings and ITA mechanisms are designed to replace traditional positional embeddings to better handle irregular time series, and to map the original irregular time series into regularly sampled latent vectors at fixed intervals. We will include the high-level justification about the assumptions in section Method. Initializations are similar to conventional positional embedding in vanilla transformers and there’s no constraints of these parameters, similar to [1]. We will highlight these connections in the revised Method section.

[1] Shukla, Satya Narayan and Benjamin M Marlin. “Multi-Time Attention Networks for Irregularly Sampled Time Series.” ArXiv abs/2101.10318 (2020): n. pag.

W2. The model’s generalization claim would be better supported by comparing it to a broader array of baselines that specifically address irregular time series and variable sampling rates.

A2. None of the baselines addresses the generalization across various sampling rates and irregular patterns, so we included the SOTA baselines that address both regular and irregular time series, under supervised and unsupervised setting for a comprehensive comparison, as shown in Table 3. These experiments address different irregularity patterns and sampling rates across datasets, as shown in the Table below as well as in Appendix B. We can include it in the revised manuscript for clarity.

W3. The model’s main innovations appear to be modest adaptations of existing techniques (e.g., masked autoencoders). Without stronger theoretical or empirical evidence, GITAR’s contribution to the field of irregular time series analysis seems limited.

A3. Our key novelty lies in designing a SSL framework that can generalize across various irregular patterns, ratios, and number of signal channels. Existing methods using masked autoencoders have been limited to a fixed number of channels, and have been fine-tuned and evaluated using the same datasets, failing to generalize to different unseen domains. In contrast, our method incorporates channel-independent design and irregular-sensitive masking encoder to efficiently learn the underlying patterns of irregular time series regardless of the characteristics of irregular training data. We demonstrate its generalizability performance at table 3.

References:

[1]Yang, Yiyuan et al. “A Survey on Diffusion Models for Time Series and Spatio-Temporal Data.” ArXiv abs/2404.18886 (2024): n. pag.

[2]Zhang, Weijiao et al. “Irregular Multivariate Time Series Forecasting: A Transformable Patching Graph Neural Networks Approach.” International Conference on Machine Learning (2024).

[3]Chowdhury, Ranak Roy et al. “PrimeNet: Pre-training for Irregular Multivariate Time Series.” AAAI Conference on Artificial Intelligence (2023).