The KNN Score for Evaluating Probabilistic Multivariate Time Series Forecasting

Thank you for your feedback! We see from the reviews that the paper needs significant revision and would like to use this forum as an opportunity for further clarification and feedback.

“The paper aligns with a recent empirical investigation (Caroll, 2022)”

Thanks for pointing out the work by Caroll, 2022. We unfortunately could not find the paper based on the name alone. Could you give us the full reference?

“it's unclear whether it qualifies as a proper scoring rule. If it does, it's essential to provide a formal proof. [...] Could you provide a theoretical justification for using the log score to evaluate multiple projections?”

We agree, that a proof of propriety would be ideal, we only measure this property with our experiments. It should be sufficient to prove, that the inner part of the KNN score is a valid density estimate, do you agree? Because replacing $p(x)$ of the strictly proper logarithmic score with a valid density estimate should result in a proper scoring rule.

“Your score's definition involves both a density model and a score. Please clarify this relationship.”

We start with the logarithmic score, which computes the log density of the prediction model at the observation. Then we replace the model density, which is not evaluable for many models, with a KNN density estimate. For $\mathbf x, \mathbf y \in \mathbb R^D$ and if $V_D$ is the volume of the unit sphere in $D$ dimensions, this results in $\log \frac{k}{|\mathcal X|V_D\|\text{NN}_{\mathcal X}^k(\mathbf y) - \mathbf y\|^D}$. For a given evaluation scenario, $D$, $k$, $|\mathcal X|$ and $V_D$ are constant, so to simplify the computation we omit them for our score.

“Please include a reference for your energy score proof in the Appendix.”

What reference are you missing here? We refer to the proof in Appendix A in the second sentence of Section 2.5.1. The proof is our own work, so there is no reference to another paper.

“a recent and important reference, "Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts" (ICML 2023), appears to be missing”

Thanks for pointing out this work! It is indeed highly relevant.

“There seem to be various approximations and challenges in implementing your method, such as not enforcing orthogonality of rows and the issue of sampling enough prediction vectors due to the curse of dimensionality. It's unclear how these challenges and design choices affect your proposed score.”

We have observed in preliminary experiments that omitting the orthogonality requirement has at most very small effects that are overshadowed by other factors. The number of prediction vectors is one such factor; we generally use 100 vectors, as this is often the default in related work. As these certainly depend on the internal data structure and also on the model prediction: do you think a broad experimental setup with various models and data sets would be sufficient to quantify deviations regarding the impact of omitting the orthogonality of rows and the number of samples?

“The paper requires significant revisions for improved clarity, mathematical rigor, and notations. ”

Thanks also for pointing out the problems with the mathematical notation, we really appreciate it! However, $\mathbb{X}$ is defined in the introduction to Section 2.

“The statement, "Since all marginals are evaluated independently, certain properties of the distribution are lost," needs further clarification. If the energy score is a proper scoring rule, explain what specific properties are lost and why.”

We agree that this statement is phrased too strongly. Perhaps "underrepresented" or something similar would be a better choice here.

“The statement, "Only the green one mimics dependencies between time steps correctly," is disputable. Having all realizations in the predictive region does not necessarily imply a correct capture of true uncertainty.”

Figure 1 should be an intuitive example of why a scoring rule should not score each point independently. The observed bivariate time series repeats the same values twice, and there is a clear correlation between the two individual series.The implicit assumption, of course, is that this is always the behavior of the true data. Now, if you look at a particular time step of a particular series, each model has exactly the same deviations around the observation, so the uncertainty around each point is the same. This is why scoring rules like CRPS score them all the same. However, assuming that the temporal and inter-series correlation structure shown by the observation is representative of the true data, only the green forecast properly reflects these dependencies. It is much more likely to observe the green forecast than the orange or red forecast.

“The assertion, "As a result, these metrics should be avoided when assessing probabilistic predictions," needs further elaboration and support. Clarify under what circumstances these metrics should be avoided and why.”

An ideal probabilistic forecast should match the (unknown) distribution of the data. An optimal model for MSE or MAE does not need to predict a distribution at all, predicting the mean or median is sufficient for an optimal score. For example, if the data distribution has two equal-sized modes, the density at the mean could be zero, but would be optimal if evaluated by MSE.

We appreciate the authors' response. However, my evaluation remains unchanged, as I still identify several flaws in the paper.

• "Thanks for pointing out the work by Caroll, 2022. We unfortunately could not find the paper based on the name alone. Could you give us the full reference?"

  • Alexander, Carol, et al. "Evaluating the discrimination ability of proper multi-variate scoring rules." Annals of Operations Research (2022): 1-27.

• "It should be sufficient to prove, that the inner part of the KNN score is a valid density estimate, do you agree?"

  • I believe you should use the definition of propriety and demonstrate that the expectation of your score with the optimal prediction is always smaller or equal to any other prediction.

• What reference are you missing here? We refer to the proof in Appendix A in the second sentence of Section 2.5.1. The proof is our own work, so there is no reference to another paper.

  • You are not the first one to prove this fact. This is a well-known fact in the scoring rule literature. See, for example, "Energy statistics: A class of statistics based on distances.

• “The assertion, "As a result, these metrics should be avoided when assessing probabilistic predictions," needs further elaboration and support. Clarify under what circumstances these metrics should be avoided and why.”

  • It is true that you should not use MSE alone to evaluate probabilistic predictions. However, it is also useful to assess multiple functionals, such as the mean or specific quantiles. Hence, I disagree with this statement.

Thank you for your feedback! We see from the reviews that the paper needs significant revision and would like to use this forum as an opportunity for further clarification and feedback.

“While simple MAD, MSE, CRPS may be oblivious to correlations or regularity of forecasts, they reflect the key desiderata of the forecasts stemming from the task: that they are close to the ground truth. As the model is better able to capture correlations, one expects that simple metrics like MSE will also vanish.”

A probabilistic prediction ideally represents the uncertainty of the true data distribution correctly. Assume a distribution with two modes of equal size, then the MSE would be ideal if the model always predicts the mean of the distribution, where the density could be zero. A probabilistic univariate score would indeed vanish, e.g. an optimal probabilistic multivariate model would receive the optimal CRPS score, but the reverse is not true: a model that is optimal under CRPS does not have to produce the correct multivariate structure. Therefore, more complex scoring rules are needed to properly evaluate multivariate time series forecasts.

“this makes the magnitude of the metric dependent on the size of the sample (in the paper, "ensemble"). i.e., in order to compare two models one would have to compare them on the same number of sampled trajectories or the metric would be invalid otherwise.”

This is true. In practice, these would have to be defined for a particular benchmark, such as the parameters $p$ for the energy score or the variogram score, or $\alpha$ for the MSIS. We agree that deriving a guideline for choosing these parameters will be important for practical applicability. However, such a guideline would by necessity be scenario-dependent, since different applications will have vastly different prediction dimensionalities. We plan to investigate this in future work.

“Finally, the KNN score's intuition is that the best model is able to place a forecast close in L2 distance to the ground truth. Note that for a univariate forecast this is equivalent to saying that the k-th best forecast in an ensemble, measured in squared error, has low squared error. Setting k=1, this is equivalent to choosing the model with the "best in hindsight" MSE. In other words, KNNS does not appear to measure the quality of distributional forecasts, a desired property set out in the paper, but only point forecast error. ”

Evaluating based on a "best in hindsight" MSE is a better option for distributional predictions than the standard MSE, because a method that only produces a point prediction would perform worse than a method that produces multiple predictions based on a distribution. An optimal model under a "best in hindsight" MSE would spread its predictions over the area of possible futures, with more predictions in areas where it expects a higher probability.

“Multiple random projections sampled and with sufficient latent dimension would be justified for only very high dimensionality tasks such as spatiotemporal forecasting in earth sciences, etc. This doesn't appear to be the case with the experiment setup.”

Many multivariate time series applications (e.g., stock market or energy forecasting) consist of tens or even hundreds of individual time series with a forecast horizon of about 10-50 time steps. Flattening such a forecast results in forecast vectors of several hundreds to thousands of dimensions, which is far too much for a valid density estimate with a manageable number of samples.

“The introduction of KNNS is somewhat counterintuitive. The paper first reports that out of sample likelihood (i.e., perplexity) is a desirable metric; but due to its practicality immediately moves to KNN distance. I believe this link should be substantiated.”

We start with the logarithmic score, which computes the log density of the prediction model at the observation. Then we replace the model density, which is not evaluable for many models, by a KNN density estimate. For $\mathbf x, \mathbf y \in \mathbb R^D$ and if $V_D$ is the volume of the unit sphere in $D$ dimensions, this results in $\log \frac{k}{|\mathcal X|V_D\|\text{NN}_{\mathcal X}^k(\mathbf y) - \mathbf y\|^D}$. Given a certain evaluation scenario, $D$, $k$, $|\mathcal X|$ and $V_D$ are constant, so to simplify the computation we omit them for our score. The result is the negative log distance to the kth-nearest neighbor.

“Why didn't you consider using the average of the first k distances to decrease variance?”

We have actually thought about averaging over the closest $k$ neighbors, but we were worried that this would deviate even further from the idea of combining a density estimate with the log score. Instead of computing the score with $k=3$, one could compute the mean distance with $k=5$, or as an alternative the median distance with $k=5$. We would choose the latter, as this corresponds to the original score with $k=3$.

Thank you for your feedback! We see from the reviews that the paper needs significant revision and would like to use this forum as an opportunity for further clarification and feedback.

“A critical determinant of the proposed metric's efficacy is the selection of the number of neighbors; however, the paper falls short of providing a rigorous justification for this parameter choice.”

The number of neighbors and the dimensionality depend strongly on the application. We do not give a guideline for the choice of these parameters, since they may depend not only on the dimensionality of the data, but also on its internal structure. We expect that stable parameter settings can be found for different application scenarios with their specific dimensionality ranges. We will investigate this further in future work.

“Additionally, while employing random projection for dimension reduction, the paper lacks a thorough theoretical analysis concerning the impact of this technique, which could potentially undermine the robustness or interpretability of the findings.”

We measure the impact of random projections only through experiments, where it becomes clear that the scoring rule behaves improperly when no projections are used. We assume that this is due to the large dimensionality, so random projections into a subspace are a natural choice. Our experiments show that this increases the discriminative power. You mention that you would expect a theoretical analysis regarding robustness and interpretability. Do you mean quantifying the effects of the number of projections and number of samples? Would experiments measuring the noise be sufficient here?

“How does this method fare when juxtaposed against the Kernel Density Estimation method, especially in terms of accuracy and computational efficiency?”

We considered kernel density estimation as an alternative, but found it much harder to choose a “good” kernel bandwidth. An ideal bandwidth would depend heavily on the magnitude of the uncertainty in the observed data, a property that is unknown. If the bandwidth is chosen too low, a model only scores if it hits the observation quite exactly, if it is too high, modes get lost in the forecast. Its a parameter that is related to the width of a mode of the data uncertainty. For KNN density estimation, k is independent of scale, but depends only on the relative size of the modes. A small number of k is able to estimate the density for smaller modes of the prediction, but also becomes more noisy. In our case, with many dimensions and relatively few samples, there are only a few plausible options for k.