Winner-takes-all for Multivariate Probabilistic Time Series Forecasting

We would like to thank the reviewer for their positive feedback on the paper.

In Eq.5, why is not $x_{t-1}$ inputted to the function gamma?

This is because, with our notations, $\gamma^k_{\theta}$ corresponds only to the head. The full model writes as $\gamma_\theta^k \circ s_\theta$, where $s_{\theta}$ is the backbone (See section 3,4 and Figure A of the rebuttal pdf).

In l.134 of the right-hand side in p.2, why is not $x_{t-1}$ inputted to the function $f^k_\theta$?

Same as above. This is because, with our notations, $f^k_{\theta}$ Only the head. The full model writes as $f_\theta^k \circ s_\theta$, where $s_{\theta}$ is the backbone. This will be made clearer with an illustration as suggested by Reviewer uVhD.

The authors mentioned that the score head can avoid overconfident heads, but how they avoid them is not described.

Overconfidence is a known issue in Multiple Choice Learning (Rupprecht et al., Lee et al), where, some heads associated with very low probability zones may not be distinguishable from plausible hypotheses at inference time. Score-heads allow to solve this issue by learning the probability of each head to avoid this situation.

Can we not replace min in the WTA loss with the gamma and jointly train with the WTA loss, which can be a more straightforward approach? It can be similar to Eq.9.

Does the reviewer suggest training a loss that looks like a weighted sum of the $L_{\theta}^{k}(x_{1:t_{0}-1},x_{t_{0}:T})$ by the $K$ predicted scores? The suggestion of the reviewer is very interesting, as it would when the hypotheses are fixed, encourage the score associated with the lowest distance to increase (and the other to decrease). However, this would make the prediction head loss dependent on the values of the scores, which may produce different training dynamics as the current model. In the current version, the score head objective depends on the position of the hypotheses, but not the opposite. This would definitely be a promising try for further work.

Proposition 5.2. is analyzed only with the binary cross entropy for the score heads. However, s in TimeMCL are shared in the score heads and heads, and TimeMCL is trained based on the compound loss with the WTA loss, which is not the direct case of Proposition 5.2.

Indeed, Proposition 5.2 is analyzed only with the binary cross entropy for the score heads. In this proposition, we implicitly assume that the prediction heads have already converged, so that the task of learning the probability mass of each cell becomes duable for the prediction heads.

Indeed, in according with Letzelter et al. (2024b) (Section C.1.2), we observed the WTA training scheme leads to a fast convergence of the predictions, while the scoring heads are slightly slower to train because they need the prediction heads to have already converged to do so.

This will be made clearer in the assumption of this Proposition 5.1.

Why is the distortion a fair metric ?

The Distortion is known from the quantization literature (Pagès.G.,2015), as a way to assess the quality of a set of $K$ samples $z_k$, $k = 1,...,K$, for quantizing a target distribution $p$, with:

$D_2 := \int_{\mathcal{X}} \min_{k=1, \ldots, K} \left\| z_k - x \right\|_{2}^2 \mathrm{d}p(x).$

In our setup, the distortion we are considering is the generalization of the above where the samples $z_k$ are functions of the context. It writes in the context of time series

$\int_{\mathcal{X}^{T}} \min_{k=1, \ldots, K} \left\| z_k\left(x_{1: t_0-1}\right) - x_{t_0: T} \right\|^2 \; \mathrm{d}p(x_{1: t_0-1}, x_{t_0: T}) \simeq \frac{1}{N} \sum_{i} \min_{k=1, \ldots, K} \left\| z_k\left(x^i_{1: t_0-1}\right) - x^i_{t_0: T} \right\|^2.$

where $(x^i_{1: t_0-1},x^i_{t_0: T})$ are samples from $p(x_{1: t_0-1},x_{t_0: T})$.

It implicitly assesses how well the predicted samples cover a target distribution with a given set of samples. The distortion is a fair metric provided that the same number of hypotheses is used for each baseline, as the distortion is expected to improve with $K$, as per the Rate distortion curve (Gray, 1989) in the optimal case.

Gray, Robert M. Source coding theory. Vol. 83. Springer Science & Business Media, 1989.

We thank the reviewer for their relevant remarks. The rebuttal pdf is attached to the response. Figure A of the pdf will be included in the paper.

Comparison with more SOTA methods

The results will be more validated if compared with more SOTA methods.

We conducted this benchmark using DeepAR and TimeGrad to demonstrate that, with consistent settings across baselines (e.g., backbone, data scaler, training details), our approach offers competitive distortion at a low computational cost. Since we used an RNN backbone, we felt comparing it with other architectures, such as transformers, would complicate conclusions. However, we acknowledge the reviewer’s point that additional baselines could strengthen our work.

To address the reviewer's comment and enhance our evaluation, we added additional models. Specifically, we included Tactis-2 (Ashok et al, 2024), a transformer-based model based on non-parametric copulas, and TempFlow (Rasul et al, 2020), which uses conditioned normalizing flows (with both RNN and transformer backbones). For completeness and as suggested by Reviewer 2HFL, we also included exponential smoothing (ETS) as a simple baseline without neural networks.

Results analysis

Distortion Comparison (Table A).

• TimeMCL outperforms TempFlow, both when using the same RNN backbone and when TempFlow is based on a Transformer.

• Tactis proves to be a strong competitor in terms of Distortion, though at a significantly higher computational cost (see Table H).

• We conducted an ablation study on the number of hypotheses (Table E). We observed that TimeMCL consistently achieves the best performance (except when using only one hypothesis).

Inference run-time (Table H)

• Among neural methods, TimeMCL and DeepAR demonstrate the best trade-off between speed and performance.

• ETS achieves the fastest inference but exhibits weaker performance on other metrics.

Smoothness Analysis (Table B)

• TimeMCL achieves the best smoothness scores, as measured by Total Variation.

• This supports our theoretical claim from Section 5.2, which predicts that TimeMCL generates smoother trajectories.

Additional metrics (Tables C & D)

• • TimeMCL does not significantly improve RMSE, CRPS, or the Energy Score (Table G), as expected, since it does not directly optimize these metrics.

We plan to extend this comparison by implementing timeMCL with a transformers-based architecture as the backbone, and adhering to Tactis's training details for more accurate performance comparison.

How are the graph tranformer methods, e.g. STGNN, compare with the diffusion methods in general and with TimeMCL in particular?

Regarding spatio-temporal graph transformer methods, does the reviewer have a specific method in mind? Most STGNN methods we found are tailored to specific tasks (e.g., traffic prediction in Luo et al., 2023). However, we did identify stemGNN (Cao et al., 2020), which integrates graph and attention mechanisms and is evaluated on similar data.

However, comparing our approach with stemGNN is difficult, as it is non-probabilistic and generates only one prediction per input. As future work, exploring a graph transformer-based approach in place of our RNN backbone could be promising.

Need for datasets from more domains, e.g., Stock markets

How will the proposed approach perform in other domains e.g. Stock Market?

We thank the reviewer for their advice. Accordingly, we performed experiments on a dataset of correlated financial time series, consisting of 15 correlated Cryptocurrencies (See Tables J and K of the rebuttal pdf).

On this dataset, TimeMCL was trained with annealed winner takes all loss, and compared with the previous baselines, and the results, presented in Table I further demonstrate the competitiveness of the method in terms of Distortion and Smoothness, with also good CRPS here.

We also provide a visualization of the predictions, along with the baselines in Figure B, which is akin to Figure 2 of the main paper.

Missing references

I believe other SOTA works must be discussed including the Graph Deep learning methods e.g. STGNN.

To make our benchmarks more comprehensive, we’ve included Tactis2, TempFlow (with its transformer-based variant), and non-neural exponential smoothing (ETS) methods (Hyndman et al., 2008). Additionally, STGNN will be referenced in the paper.

Luo, X., Zhu, C., Zhang, D., & Li, Q. (2023). Stg4traffic: A survey and benchmark of spatial-temporal graph neural networks for traffic prediction

Cao, Defu, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong et al. "Spectral temporal graph neural network for multivariate time-series forecasting." In NeurIPS 2020

Rasul, Kashif, Abdul-Saboor Sheikh, Ingmar Schuster, Urs M. Bergmann, and Roland Vollgraf. "Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows." In ICLR 2021

We thank the reviewer for their feedback. The rebuttal pdf is attached to the response.

Clarification of the theory

TimeMCL is a stationary conditional functional quantizer [...], but it's unclear what value this provides as the underlying clustering problem is NP-hard, and you're dealing with very high-dimensional data.

The reviewer raises a valid point. However, as shown in Figure 1, our toy example qualitatively demonstrates that our training scheme closely aligns with the target conditional quantizer in practice, even when forecasting across 250 time steps (see also Appendix B for details).

Maybe [...] this algorithm converges to the optimal voronoi tesselation [...], but it would help to analyze the rate of convergence theoretically and experimentally.

Theorem 2 of Loubes & Pelletier (2017) provides an asymptotic upper bound on the distortion error with respect to the number of training pairs in quantizer learning. Extending this result to neural networks trained with WTA Loss is a promising direction for future work.

Additional baselines and metrics

A runtime analysis would be nice [...]. It would be good to add in other relevant baselines, e.g. TACTiS and simple baselines You could also assess the models using actual multivariate metrics, e.g. energy score [...] It would be nice to also provide [...] a measure of diversity, [...] and smoothness.

In response to the reviewer, we added experiments with a simple baseline, ETS exponential smoothing (Hyndman et al., 2008), which does not rely on neural networks. We also included TempFlow (Rasul et al., 2021), a normalizing flow-based using both RNN and Transformer backbones, as well as Tactis-2 (Ashok et al., 2024), a Copula method with a Transformer backbone.

These methods were evaluated across the same six datasets. In addition to the metrics from the original paper (Distortion, RMSE, and CRPS-Sum), we followed the reviewers' suggestions and included Inference runtime, Smoothness, and Energy Score.

Results analysis

Distortion Comparison (Table A,E,F).

• TimeMCL outperforms TempFlow, and Tactis proves to be a strong competitor in terms of Distortion, though being slower at inference (see Table H).

• An ablation study on the number of hypotheses (Table E) shows that TimeMCL consistently achieves the best performance (except with $K=1$).

Inference run-time (Table H)

• Among neural network-based methods, TimeMCL and DeepAR demonstrate the best trade-off between speed and performance.

• ETS achieves the fastest inference, with weaker performance otherwise.

Smoothness Analysis (Table B)

• TimeMCL achieves the best smoothness scores, as measured by Total Variation (averaged over predictions).

• This supports our theoretical claim from Section 5.2, which predicts that TimeMCL generates smoother trajectories.

Additional metrics (Tables C & D)

• TimeMCL does not significantly improve RMSE, CRPS, or the Energy Score (Table G), since it does not directly optimize these metrics.

We did not conduct further experiments with Diversity, as we believe Distortion implicitly captures it.

When comparing TimeMCL (using an RNN backbone) with methods using transformer backbones, it's hard to tell if performance improvements are due to the training method or the backbone. To clarify, we plan to implement TimeMCL with a transformer backbone, following Tactis's training details.

Additional datasets

It is difficult to know how much multivariate correlation is present in these datasets. Therefore, performance on synthetic multivariate tasks would be insightful

In response, we conducted experiments on a new dataset of correlated financial cryptocurrencies time series (see Table J), with the correlation matrix in Table K.

TimeMCL was trained with an aMCL loss and compared to previous baselines. Results in Table I show that our method remains competitive, excelling in both Distortion and Smoothness, with strong CRPS performance. Visualizations are in Figure B.

Additional questions

Can the model accept covariates?

The model can accept covariates. In previous implementations these typically serve as additional concatenated input features.

To compute TimeMCL metrics [...], we resample with replacement from the K hypotheses obtained [...]. Can you explain this is more detail please?

Our implementation extends GluonTS (Alexandrov et al., 2020) with minimal code changes. Instead of modifying evaluation functions, we resample from the K hypotheses, weighted by their probabilities, allowing us to use existing evaluation functions (e.g., CRPS) without rewriting metrics for TimeMCL.

Did you look at methods that might enable a variable number of heads?

Indeed, the number of predictions must be predefined beforehand. Exploring dynamic "rearrangements" of hypotheses when adding new ones, without full retraining, is left for future work.

We are grateful to the reviewer for their positive feedback and insightful comments.

Sensibility to initialization

K-means is quite sensitive to initialization. Can you comment on the sensitivity of WTA to the initialization of the hypotheses?

As MCL can be seen as a conditional and gradient-based variant of K-Means, it inherits some of its limitations, in particular the sensibility to initialization. This is also related to the known collapse issue (Rupprecht et al.) in the MCL literature, where some of the hypotheses may never be chosen, leading to suboptimal Distortion performance. This is the reason why we decided to leverage WTA variants (aMCL, Relaxed-WTA), for TimeMCL, which makes the algorithm more robust to these issues.

Note also that previous works (Letzelter et al., 2023; Shekarforoush et al.,2024) have noticed that this collapse issue can, in some settings, be naturally solved by the randomness of the data distribution. As mentioned in the Limitations paragraph of the submission, further work will include enhanced normalization techniques to further improve the quality of the optimum of the vanilla TimeMCL.

Motivation clarification

The motivation behind the applicability of the paper to real life forecasting problems is unclear. What are some important applications for this method in real life scenarios? In what settings would someone want to generate forecasts from multiple hypothesis?

Indeed, compared to generative models, we believe time-MCL has the ability to capture rare events or "modes" in the conditional distribution. This is for instance illustrated in Figure 2 (middle) on the Solar dataset where one of the hypotheses is capturing a rare event. This can indeed also be useful for Stock Market prediction, for capturing trend reversal. For clarification, we included a use case with financial data in the rebuttal pdf (See e.g., Figure B) for which details are provided in the answers of Reviewer 2HFL and Reviewer uVhD.

Letzelter, Victor, Mathieu Fontaine, Mickaël Chen, Patrick Pérez, Slim Essid, and Gaël Richard. "Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis." In NeurIPS, 2023

Shekarforoush, Shayan, David Lindell, Marcus A. Brubaker, and David J. Fleet. "CryoSPIN: Improving Ab-Initio Cryo-EM Reconstruction with Semi-Amortized Pose Inference." In NeurIPS, 2024