Marginalization Consistent Mixture of Separable Flows for Probabilistic Irregular Time Series Forecasting

Thank you for the insightful review.

W1. Experiments on varying forecasting, observation and spatial range.

We conducted experiments by varying the observation and forecasting times (obs/forc), and we have added these details to the supplementary sec. C.4. Our experiments cover tabular time series with up to 102 channels; spatio-temporal problem are outside the scope of this work.

Here, the evaluation metric is njNLL (lower the better). Similar to results in table 1, ProFITi is the best performing model for predicting joint NLL whereas moses is the second best.

W2. Importance of marginalization consistency

We showed the importance of marginalization consistency with the results in Tables 1 and 2. While our model, moses, has slightly worse results compared to ProFITi in predicting joint distributions — $\sim$5% worse likelihoods (note: in terms of actual density, not NLL; see table 1) — moses’ predictions for univariate marginal distributions are significantly better than ProFITi (~51%, see table 2) which is a consequence of ProFITi not adhering to marginalization consistency (fig. 4,5).

Table 2 demonstrates that moses, trained for the joint NLL, makes very good predictions for the marginal variables, despite only being trained for the joint NLL, which is a direct consequence of the marginalization consistency.

W3. Missing details on the training of the model

Our model is trained using the Adam optimizer, with a batch size of 64 and a learning rate of 0.001. We added these details to the Experiments section. Other hyperparameters used in the model have already been provided in the paper.

Q1. Demonstration of Marginalization consistency on synthetic ODE dataset

Marginalization consistency does not have to be verified for specific datasets, as it is guaranteed by design. While we agree that the chaotic ODE dataset is an interesting dataset for comparisons, we believe that testing marginalization consistency on an ODE dataset does not add significant value beyond the bivariate case. Visualizing joint distributions is feasible in bivariate cases but becomes much more challenging in higher dimensions. Therefore, we do not plan to pursue this suggestion further.

Q2. Forecasting on longer forecasting horizon, possibility of improving forecasting results with marginalization consistency

As noted in response to W1, we added an experiment that varies the observation and forecasting horizons and compared our model’s results with the published results of ProFITi.

Regarding the effect of marginalization consistency on forecasting accuracy, we would like to emphasize that maintaining marginalization consistency imposes certain constraints on the model. This results in a slight decrease in performance of joint NLL compared to models without this consistency, such as ProFITi, when compared on certain queries without their sub-queries. For metrics that penalize violations of marginalization consistency (i.e. aggregating likelihoods over all sub-queries.) we expect our model to be better as seen in Table 2.

Thank you for your detailed review. Our responses are as follows:

W1. Why do we need Marginalization Consistency?

Marginalization consistency ensures that distributions over subsets of variables remain compatible with the full model, which enables coherent and reliable inference. Not satisfying marginalization consistency represents an internal inconsistency of the model. Would you trust a forecaster that makes self-contradictory predictions? Your question can also refer to downstream tasks for which consistent estimates of marginals and joint distributions would play a role: there are robust optimization or control problems for which this is true, e.g. [1].

W2. Comparison of Moses using a multivariate Normalizing Flow?

Replacing univariate flows with multivariate flows poses challenges due to the varying number of variables involved. To the best of our knowledge, only the ProFITi and Tactis models address this scenario, and serve as our baseline models.

W3. Is it possible to create prediction intervals on desired queries and attain the desired coverage due to marginalisation consistency?

The joint and marginal densities enable the calculation of prediction intervals for specific quantiles and allow for coverage testing. However, this process can become prohibitively expensive for large multivariate queries due to the curse of dimensionality.

W4. Comparison w.r.t. Energy Score

In the revised supplementary material, we provide a comparison among the best-performing models: Moses, ProFITi, and Neural Flows for Energy Score (lower the better). The results are as follows:

Similar to Table 1, we see that our model is the best consistent model whereas ProFITi is the overall best for predicting joint distributions.

W5. Worse CRPS and MSE

Our model achieves the best results among marginalization-consistent models and is overall the second-best model. On the other hand, the best-performing model, ProFITi, has worse CRPS and MSE scores compared to our model when evaluated solely on the univariate marginals (ProFITi-marg). This is because ProFITi is not marginalization-consistent.

W6. Motivation for section C.3 in the appendix

We included this experiment to compare our model with other marginalization-consistent models studied. Currently, very few models, such as Probabilistic Circuits, are both tractable and marginalization-consistent. These models are designed exclusively for density estimation tasks over a fixed number of variables. Therefore, we aim to compare the performance of our model, which can handle a variable number of inputs, against these models. We have clarified this in the revised version.

[1] Guo, Yuxue, et al. "AI‐based ensemble flood forecasts and its implementation in multi‐objective robust optimization operation for reservoir flood control." Water Resources Research 60.5 (2024): e2023WR035693.

Thanks for providing detailed feedback.

W1.1: Simple Experimental Settings

Our experimental setup for probabilistic forecasting of irregular time series with missing values aligns with prior works, including ProFITi, GRU-ODE-Bayes, and Neural Flows, to ensure fair comparisons. In the revised version, we also included an experiment on varying observation and forecasting lengths for further analysis (see revised Supplementary Section C.4). Additionally, since there are marginalization-consistent models for density estimation with a fixed number of variables, we have already compared our approach with other such models in Supplementary Section C.6. of revised version (C.4. of original version).

W1.2: Worse results than ProFITi

While our model, moses, has slightly worse results compared to ProFITi in predicting joint distributions - $\sim$5% worse likelihoods (note: in terms of actual density, not NLL; see table 1) - moses’ predictions for univariate marginal distributions are significantly better than ProFITi (~51%, see table 2) which is a consequence of ProFITi not adhering to marginalization consistency (fig. 4,5).

In principle, one could train profiti w.r.t to marginals. However, there are exponentially many different subsets of variables one could marginalize with respect to, so this approach is infeasible if we are interested in building a model that performs well for arbitrary queries.

Table 2 demonstrates that moses, trained for the joint NLL, makes very good predictions for the marginal variables, despite only being trained for the joint NLL, which is a direct consequence of the marginalization consistency.

W2: Comparison with other Normalizing Flows (e.g. Continuous Normalizing Flows, FFJORD)

Since the flow needs to be separable, i.e. applied component-wise, we need an expressive univariate flow. We opted for Linear Rational Splines due to their computational efficiency and improved performance over other univariate flows. Dolatabadi et al. (2020) compared FFJORD to spline functions and showed that spline functions provide better density estimates (table 1,2 of Dolatabadi et al. (2020)), and avoids having to deal with numerical ODE solvers.

W4: Unfair comparison in fig. 1

We agree with the reviewer’s concerns, in the experiment the GMM used 7 parameters per mixture component, whereas moses used 133 (7 + 126 for the NF). Currently, we are rerunning the experiment using a different setup, testing the scalability of the approach with respect to the input space dimensionality, under the hypothesis that GMMs do not scale well to high dimensional distributions. We will report the results the earliest possible.

W5: Number of samples for Wasserstein Distance

We used 1,000 samples for the univariate marginals, which is larger than the sample sizes used in many other studies (e.g., [1], [2]), where sample sizes for evaluation are typically 100. However, we verified that the results are reliable by running 5 repeats per fold. As shown in the table below, the variance between repeats is negligible, showing that N=1000 is sufficient for convergence.

W3,W6: Limitation of Computational Complexity, solving higher dimensional problem

The normalizing flows used by us are actually computationally light: since they are separable, their Jacobian is diagonal, and hence they can be computed in $𝓞(K)$. The actual bottleneck is computing the density of the Gaussian base distributions, which generally costs $𝓞(K^3)$, as stated in the limitations section.

However, this can be reduced significantly: since we parametrize $Σ=𝕀+UUᵀ$ (eq. 13b), one can actually compute the density in $𝓞(M^2K)$ using the Woodbury and Weinstein–Aronszajn identities. Consequently, our model’s complexity is $𝓞(M^2⋅K⋅D)$, where $K$ is query size, $M$ the embedding dimensionality, and $D$ the number of mixture components, showing that the model is quite scalable when limiting $M$ and $D$.

We forgot to mention this, as in our experiments, we do not use this technique since $K$ is relatively small. We revised the limitations section accordingly.

[1] Rasul, Kashif, et al. "Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows." ICLR’21.

[2] Zheng, Zhihao, et al. "Better Batch for Deep Probabilistic Time Series Forecasting." AISTATS’24.