ProFITi: Probabilistic Forecasting of Irregular Time Series via Conditional Flows

We thank you for your positive review. We address your concerns below:

w1. Figure 1: I'm not sure what this is trying to illustrate, since you're comparing a target bimodal distribution against two other linear gaussian models, and doesn't do much to highlight the novelty of ProFITi.

• Existing models for probabilistic IMTS forecasting predict only Gaussian distributions (ex. HETVAE, CRU, GRU-ODE, Neural Flows). So fig. 1 is a simple illustration of the goals and achievements of our model ProFITi: to quantify uncertainty in a richer manner. We edited the introduciton to make this clearer.

w2. Experiments seem to only consist of ECG based dataset, which is heavily periodic and consists of similar patterns. Would be interesting to see other datasets here.

• We agree. We added experiments on USHCN, an IMTS climate dataset that also has been previously used in the literature (De Brouwer et al. 2019, Schirmer et al. 2022, Yalavarthi et al. 2023) to section 7.

w3. Minor typo: "Both, Leaky" doesn't need a comma.

• Fixed. Thanks!

q1. What activation function is used in ProFITi-Shiesh? Authors also mention Leaky ReLU but do not note where it is.

• ProFITi-Shiesh does not contain any activation function in the flow model. It is used as ablation to demonstrate the use of a non-linear activation function in ProFITi.
• Leaky ReLU with very small slope (0.01) for the negative values suffers from the vanishing gradient problem. Therefore no results are shown. We added this information to Appendix H.1.

Thank you for the detailed review. We address the concerns below:

w1+q8. In my opinion could be interesting to assess the behavior of the proposed model in several conditions in terms of percentage of missing values, sparsity of the time steps. In fact the irregular time series are very important and to give researcher an evaluation of these aspects could help to understand the quality/limitations of the proposed work.

• We agree. We added an ablation study with varying sparsity levels (time sparsity and missing values) in Appendix H.5, H.6. Our findings show that even in an extremely sparse scenario, where 90% of samples (or timepoints) are missing, ProFITi consistently outperforms both Neural Flows and GraFITi+.

w2+q9. The authors use likelihood to assess the quality of the forecast also for marginal and point forecast. In that case I think could be useful to have other metrics as RMSE, MAPE or other that is more related to the prediction error.

• We do use MSE to evaluate models w.r.t. point forecasts (tab. 4).

w3+q10. The computational effort of proposed solution is not provided and is not compared to other possible solutions. Some results are provided in appendix but only for a particular dataset.

• In the updated version, we included the runtime per epoch for all models and datasets in Table 3.
• Our main claim is to provide a more accurate model, not to provide a faster model (contribution 5). Thus we gave more room to evaluation w.r.t. accuracy but w.r.t. scalability.

q1. §3 - IMTS Query section, in the definition of $\text{QA(C)}$, what is the meaning of |x^\text{qu}| = |y|?

• yes, |.| denotes the length of a sequence. We added a brief section on notation (appendix A).

q2. In the IMTS probabilistic forecasting problem (with $\min_k t^\text{qu}_{n,k} > \max_i t^\text{obs}_{n, i}$) --> the reader is lead to think that is the k-th observation of the n-th query

• You are right. We added a respective note to section 3.

q3. What is the meaning of $\pi^{-1}$ in equation 4?

• for permutations $\pi$, one often writes $x^{\pi}$ to denote the application of the permutation $\pi$ to vector x.
• $\pi^{-1}$ is permutations that are inverse w.r.t $\pi$. $(x^\pi)^{\pi^{-1}} = x$
• we added a brief section on notation (appendix A).

q4. In eq. 7 $X_{:, 1:|X|-1}$ is the matrix X except the last column? In eq. 7 $X_{:, |X|}$ is the last column of matrix X ?

• yes. We added this, too, to the notation section (appendix A).

q5. In the protocol authors said that they use the first 36 hours of observations and forecast next 3 time steps. These time step is, looking at Appendix A, 1 hour for Physionet, 30minute for MIMIC-III, 1 minute for MIMIC-IV. Is this right? Moreover, is there a way to indicate the time sparsity of considered timeseries in oder to understand how the timeseries considered are not uniformly spaced?

• Yes, the observation intervals are data set specific.
• We added an additional column "time sparsity" to table 5 that quantifies the sparsity of the time series (as proxy measure for its irregularity).

q6. The section 6 and figure 3 should be improved in order to make clearer the architecture. In equation 11 the $x_k$ is the h of figure 3? What is the meaning of S in fig. 3?

• Yes, $h_k$ are the encoded $x_k$ and replace $x_k$ everywhere (as stated below their definition in paragraph "Query embedding": "take the roles of the $x_k$").
• $S$ is the sorting criterion introduced at the end of section 5.
• We added references for all symbols in fig. 3 to their respective definitions to make navigation between formulas and the diagram faster.

q7. In Fig. 9 the Grafiti+ returns only a region an not the trajectories? Is it possible to compare the confidence regions of ProFITi and GraFITi+?

• Since ProFITi provides non-parametric distribution, it is difficult to plot the confidence regions for multiple points simultaneuously. Hence, although it is interesting, we do not want to pursue the idea.

Thanks for your detailed review. We address the comments below:

w1. The requirement of permutation invariance, as proposed by the authors, is questionable in time series analysis

• Permutation invariance is crucial w.r.t. channels: it should not matter, if you query for channel 1 or channel 2 first.
• Permutation invariance w.r.t. time points is not a necessary property of a good model due to the causal aspects you mention. But it is a very successful property of most state-of-the-art models in vision and time series. This motivated us to search for a novel invertible variant that can be used in conjunction with normalizing flows.
• All queries carry the time point as an attribute ($q_k$ in the paper), like positional encodings in transformers, so that the model can detect and represent temporal dependencies, of course.
• We added a respective paragraph to our problem analysis section 3.

w2a. The rationale behind utilizing self-attention as the vector field in the proposed work is not clear. It seems redundant to introduce self-attention into the vector field, given that neural networks inherently possess permutation invariance properties with respect to the input.

• Standard fully connected layers in neural networks are not permutation invariant: for example, if the weight matrix is W = [[1,0],[0,1]] the identity, then the output Wx = x varies under permutations, e.g., W[1,0] = [1,0], but W[0,1]= [0,1].
• Invariance partially is crucial, partially is highly promising as explained above (see w1).

w2b. existing literature, such as [1], has already proposed the use of conditional normalizing flow models for forecasting the joint distribution of time series

• MAF (Rasul et al. 2020, [1]) is a model for regularly sampled, fully observed time series data, while ours is for irregularly sampled time series with missing values (also see table 1).
• Forecasting queries for fully observed, regularly sampled data always have the same size, thus any (conditional) normalizing flow model can be used.
• But forecasting queries for irregularly sampled time series with missing data have varying sizes (both w.r.t. to missing time points and missing observation values), and thus standard normalizing flow models are not applicable (column "joint" in table 1). This is one of the fundamental research problems we aimed to solve in our paper. Combining attention (allowing flexible size) with invertibility (to be used in a normalizing flow) is how we solved the problem. In consequence, MAF cannot predict joint distributions over channels and time points for any of the datasets in our evaluation.
• We added this delineation to the related work section 2.

q1. In the paragraph “Invariant conditional normalizing flows,” the authors claim that $x$ is the predictor, which can be grouped into $K$ elements and common elements. The statement confuses me. It is not clear why $x$ contains common elements. Could the authors provide a concrete example of the general setting and explain the statement in the paragraph?

• in IMTS forecasting you have both inputs,
  1. a variable number of queries $x^\text{qu}_1,...,x^\text{qu}_K$ (each a time point and channel), for each of which you want an output (the predicted value at that time point in that channel) and
  2. the past observed time series $x^\text{obs}$ (see section 3, "IMTS probabilistic forecasting problem"). So here: $(x_1,...x_K):= (x^\text{qu}_1,...,x^\text{qu}_K)$, $x^\text{com} := x^\text{obs}$.
• we added this example early in section 4.

q2. What is the meaning of the L.H.S. in Eqn. (4)? There is a superscript $\pi^{-1}$ on the L.H.S. in Eqn. (4), what does it mean?

• for permutations $\pi$, one often writes $x^{\pi}$ to denote the application of the permutation $\pi$ to vector $x$.
• $\pi^{-1}$ is also a permutation which is inverse of $\pi$.
• here it means, that $f$ is equivariant: if you permute inputs $z$ and $x$, then the outputs of $f$ are permuted the same way.
• we added a brief section on notation (Appendix A).

q3. The choice of $\epsilon$ in Eqn. (6) should affect the training results. Should it be determined before training? Or should it be termed as a hyperparameter during training? Should there be an investigation about the choice of?

• yes, you are right, it does. It is a hyperparameter.
• we clarified this next to Eq. 6.
• we use $A^\text{tri}$ (Eq. 9 of revised version) as a vector field in all our experiments and set $\epsilon = 0.1$. It is also shown in Figure 2. We corrected the typo in Eq. 15 in the revised version (from ISA to SITA) which could have caused the confusion.
• we added an ablation study for varying $\epsilon$ in $A^{\text{tri}}$ (appendix H.4).

We thank the reviewer for the valuable feedback. Responses to the weakness and questions are as follows

w1. My main concern is that it seems like most improvement comes from using GraFITi embeddings, which are not used by the baselines.

• Many baseline models such as NeuralFlows and GRU-ODE do not use query encoders and thus cannot (easily) be extended to use GraFITi.
• But to disentangle lifts originating from GraFITi from those originating from ProFITi, we added the model GraFITi+: the full GraFITi encoder plus a simple uncertainty quantification model, a Gaussian (see table 4).
• We added a note to section 7 Experiments.

w2+q3. Evaluation covers only three datasets from very similar domain

• We agree. We added experiments on USHCN, an IMTS climate dataset that also has been previously used in the literature (De Brouwer et al. 2019, Schirmer et al. 2022, Yalavarthi et al. 2023) to section 7.

w3+q2. The modeling limitations of the method are not clear. What kind of distributions can be modeled with the proposed flows?

• ProFITi is a normalizing flow model, as such it is not limited to a distribution with a fixed shape like the Gaussians.
• However, quantifying the exact expressive power of normalizing flow models is an active research topic of its own and beyond the scope of our paper (see, e.g., Kong/Chaudhuri 2020):
• We added a note to section 2 Literature review.

q1. Please adjust the abstract in line with ICLR formatting instructions

• Fixed. Thanks!

Kong, Zhifeng, and Kamalika Chaudhuri. “The Expressive Power of a Class of Normalizing Flow Models.” In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 3599–3609. PMLR, 2020. https://proceedings.mlr.press/v108/kong20a.html.