Error-quantified Conformal Inference for Time Series

Overall, while the proposed method is straightforward and built on a solid idea, there is room for more detailed discussion regarding the rationales, interpretations of the results, and specific hyperparameter tuning.

Thank you for the summary, and we appreciate your suggestions! Please see the main response (in a separate comment) for a summary of the extensive improvements we have made to the paper. We hope our responses answer the questions sufficiently to earn your support. Please let us know how we can improve if you still feel we are missing something.

The authors primarily select the Sigmoid function for smoothing. However, as illustrated in Figure 2, when the extent of miscoverage exceeds a certain level, the EQ term actually decreases (which is also the case for the Gaussian error function). While the authors justify this with the goal of ensuring robustness, the rationale seems somewhat unconvincing.

Thanks for your feedback! Let us clarify our point further. Please see our main response (2).

In connection with the above, if the Sigmoid function is used, the parameter c can be considered a hyperparameter. Also given that Theorem 1 imposes constraints on c for coverage guarantees, a more in-depth discussion on its selection, beyond the brief mention in Appendix D, would be appreciated.

We understand your concern. The constraints we impose on $c$ in Theorem 1 are for the convenience of the proof. Moreover, our experimental results in new Appendix D show that as $c$ varies, the changes in both coverage and width are relatively small and the two metrics are actually a trade-off. It is worth noting that, for fixed coverage, the width of our methods with $c \in$ {0.1, 0.5, 1, 1.5, 2} are consistently shorter than other methods (see Table 1, Figure 8 and 9). We choose $c=1$ that has good performance in all our datasets. A more effective value of $c$ can be obtained by grid search.

The interpretation of the miscoverage bound presented in Theorem 2 is rather ambiguous. The choice of the learning rate is a crucial aspect, yet theoretical discussion on this remains superficial.

Theorem 2 provides the coverage results of ECI under arbitrary learning rates (the worst-case scenario), but it illustrates how different learning rates influence the bounds, which cannot be directly used to choose the learning rates. In practice, we have adopted the same automatic choice of the learning rate $\eta_t$ as conformal PID which has good performance across our diverse datasets. We advocate for its application in practical scenarios, thereby eliminating the need for manual tuning.

Another approach to choosing learning rates is to utilize methods from online learning literature—run multiple versions of the adaptive procedure with different choices of learning rates $\eta$ and take weighted sums to get the final result. We list some related works below. These techniques can also be used in our proposed method.

[1] Gibbs I, Candès E J. Conformal inference for online prediction with arbitrary distribution shifts[J]. Journal of Machine Learning Research, 2024, 25(162): 1-36.

[2] Bhatnagar A, Wang H, Xiong C, et al. Improved online conformal prediction via strongly adaptive online learning[C]//International Conference on Machine Learning. PMLR, 2023: 2337-2363.

[3] Hajihashemi E, Shen Y. Multi-model Ensemble Conformal Prediction in Dynamic Environments[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.

Dear Reviewer,

Thank you for reviewing our paper again! We have posted our responses to your questions and concerns. We are wondering if you have a new evaluation of our paper after reading our responses. If you have other questions, we are very happy to discuss them with you.

Best,

Authors

Thank you for the kind reply. We agree that the selection of the initial learning rate $\eta$ is very important. This indeed is another branch for improving the adaptive procedure. (e.g. re-weighting approaches in
$1,2,3$ ).

Honestly, the reason why we report the best results is to ensure fairness and validity of our experiments, since OGD-based methods are highly sensitive to the learning rate and can easily fail (see OGD with Prophet in table 2 below). To address your concern, we fixed the initial learning rate $\eta$ and conducted our experiments below. Specifically, ACI: $\eta=0.01$, OGD: $\eta=1$, SF-OGD: $\eta=100$, decay-OGD: $\eta=200$, conformal PID, ECI, ECI-cutoff, and ECI-integral: $\eta=0.1$.

As our experiments show, even with a simple choice of \eta \= 0.1, ECI and its variants also outperform other methods in general. In the table below, the italic values represent coverage < 89%, and the bold values represent the shortest width with coverage > 89%.

In practice, we can select the learning rate based on prior knowledge, cross-validation or re-weighting approaches (as discussed in our previous comments). Take the re-weighting approach in $1$ as an example, they deploy a candidate set of learning rates, run multiple versions of the adaptive procedure in parallel (each version corresponds to a learning rate), and construct prediction intervals based on historical performance of each.

If you still have concerns, please let us know so that we can have a further discussion.

$1$ Gibbs I, Candès E J. Conformal inference for online prediction with arbitrary distribution shifts $J$ . Journal of Machine Learning Research, 2024, 25(162): 1-36.

$2$ Bhatnagar A, Wang H, Xiong C, et al. Improved online conformal prediction via strongly adaptive online learning $C$ //International Conference on Machine Learning. PMLR, 2023: 2337-2363.

$3$ Hajihashemi E, Shen Y. Multi-model Ensemble Conformal Prediction in Dynamic Environments $C$ //The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.

Dear Reviewer f97D,

Thank you again for your time and effort in reviewing our paper. Since today is the final day for reviewers to post comments, we would appreciate your feedback on our responses. If you have any further questions, please let us know and we will be happy to provide additional clarifications. If not, we would be grateful if you could consider increasing your score to reflect the fact that we have addressed your concerns.

Best regards,

The Authors

Thank you for the thoughtful review, and we enjoyed your helpful comments!

Please see the main response (in a separate comment) for a summary of the extensive improvements we have made to the paper. We hope our responses answer the questions sufficiently to earn your support. Please let us know how we can improve if you still feel we are missing something.

Derivation on line 266 is difficult to read because it is inline. Minor grammatical errors and occasional typos scattered throughout the manuscript. For Assumption 1 should it also be "∀t".

Thank you for the careful reading! We have separated the inequality on line 266 into a single column for easier reading. We also checked the written and corrected errors.

Do the proofs for the validity of EQ-integral and EQ-cutoff need to be adjusted from the proof of EQI?

Our proof of both Theorem 1 and Theorem 2 primarily lie in quantifying the EQ term $(s_t-q_t) \nabla f(s_t-q_t)$. Considering ECI-cutoff, all inequalities in the proof still hold after we replace it with the cutoff EQ term $(s_t-q_t) \nabla f(s_t-q_t)$. The assumptions made in Theorem 1 are also easier to satisfy.

Hence the proof for ECI-cutoff does not need any adjustment. When it turns to the ECI-integral cases, ways of measuring inequalities in the original proof must be adjusted according to specific $w_i$. The proof will be more cumbersome. Hence, similar to ACI and Conformal PID, we only prove the ECI form that implies the main mechanism of our method.

Line 159: "(1−α)(1+1/t)" is not necessarily in [0,1]. For instance if t=3 and α=0.001. How can this be a sample quantile?

Since $t$ represents the size of observed data, which is generally taken to be large compared to $1/\alpha$, $(1-\alpha)(1+1/t)$ will not exceed 1. If $(1-\alpha)(1+1/t)>1$, the procedure will set the sample quantile to be 1.

The EQI intervals visually are much more jagged. Could this ever result in dangerous overfitting?

We understand your concern. Actually, “jags” come from quick reactions to past mistakes. This is the common property of works based on adaptive procedure in conformal inference, and similar phenomenons can be found in ACI, conformal PID, etc (see Figure 4). If our prediction set correctly covers $Y_t$, we narrow the next prediction interval by a small size (i.e. $\alpha-EQ$). On the contrary, once there is a miscoverage point, we widen the next prediction set by a relatively large value (i.e. $1-\alpha + EQ$) and make the several following points to be covered. Therefore, whenever we widen our interval (due to miscoverage) and then narrow it in the next steps (due to coverages), there is a jag.

Besides, the reason why SF-OGD and decay-OGD seem visually to be more “smooth” is due to the fact that their learning rate is decreasing and close to zero after several iterations, causing the length of predictive intervals to be constant (we can see the jags in the early stages of SF-OGD and decay-OGD in Figure 4). However, when the learning rate is close to zero, the methods cannot react to sudden distribution shifts.

The significance of Figure 2 is unclear to me and the caption does not provide much detail. Why is this important and what is going on in this figure?

We aim to visualize the EQ term and provide readers with a clearer understanding of the role it plays and the shape it should take. Please see main response (2).

Would error magnitude information be more useful for better or worse performing base models f relative to other methods (ACI, OGD, etc)?

The error magnitude is quantified by $s_t-q_t$, which aims to minimize the interval size while ensuring coverage. Taking the symmetric case as an example, the ECI outputs the interval $[f(X_t)-c_t, f(X_t)+c_t]$. A more effective algorithm would bring $f(X_t)$ close to $Y_t$, while ECI reduces $c_t$, the radius of interval.

Our method imposes no specific requirements on the base model; for instance, the Prophet model may exhibit worse MSE, whereas the Theta model performs better. Importantly, we do not have prior knowledge regarding the effectiveness of the models, and our approach applies to both better-performing and worse-performing base models.

Dear Reviewer FWjo,

Thank you again for your time and effort in reviewing our paper. Since today is the final day for reviewers to post comments, we would appreciate your feedback on our responses. If you have any further questions, please let us know and we will be happy to provide additional clarifications. If not, we would be grateful if you could consider increasing your score to reflect the fact that we have addressed your concerns.

Best regards,

The Authors

"Conformal Inference for Time Series" is an extremely broad title, which gives the impression that the authors are the first in doing something in this which is, in fact, not true.

The latter half of our title describes the domain and the methodology, while the former half indicates our insight into the deficiency of existing methods in conformal inference for time series.

I found the article written in a very rough and unrefined way: I have counted at least 10 instances of missing articles (the...). Moreover, the introduction contains claims that are either too bold, or require a deeper explanation. What does it mean, for instance, that "Bayesian recurrent neural networks or deep gaussian processes are difficult to calibrate?", or, why is it meaningful to say that "quantile regression models may overfit when estimating uncertainty"

Thank you for the reminder. We have checked the written and corrected errors in our manuscript.

"Bayesian recurrent neural networks or deep gaussian processes are difficult to calibrate?": The reason why they are hard to calibrate is that they can not provide corresponding prediction sets based on some specified confidence level as conformal prediction does. In contrast, conformal predictions have become popular because they are model-agnostic and able to enhance complex pre-trained models with predictive uncertainties post hoc. Hence, conformal methods can yield well-calibrated prediction sets based on the output of any point estimation model, irrespective of the base model’s structure.

"Quantile regression models may overfit when estimating uncertainty": They may capture the noise in the training data rather than the true data distribution, especially when the amount of data is limited and the model is inaccurately specified. In addition, the non-smoothness of quantile regression and its sensitivity to minor changes in the data can also lead to overfitting.

I am seriously concerned about the "stock market" application. It seems to me that the authors aim at forecasting log stock prices, which are known to be nonstationary. In fact, the usual approach in the econometric literature is to forecast returns (i.e. log differences) which instead, tend to be a stationary time series.

For fair comparison, we follow the stock experimental setup of conformal PID. Besides, our methods have no requirement for data distribution, including stationarity. For practical use, we can deploy a model that offers more accurate forecasts of returns. However, conformal methods are model-free, which means that we quantify uncertainties on top of the forecasting models, regardless of the quality of them.

The "adaptive" CI school is only one of the possible approache to deal with the absence of independence between observations in Conformal Inference, in fact, https://proceedings.mlr.press/v75/chernozhukov18a.html provide a fairly interesting approach, which would be interesting to see compared to the authors proposal.

This paper extended conformal inference to dependent data by proposing a randomization procedure via permutation on blocking structures. However, this method needs transformations of data to be a stationary and strong mixing series (see conditions (A) and (E) in Section 3.2 and Lemma 1 in Section 3.3 therein). Moreover, implementing the blocking and permutation method entails heavy computation, which is not appropriate for online settings.

A very recent contribution by Oliveira et al. https://jmlr.org/papers/v25/23-1553.html shows that the contribution of adaptivity procedures to be negligible. How do the authors comment about this?

Oliveira et al. (2024) studied the split conformal prediction method for non-exchangeable data. However, the theoretical coverage results rely on the assumptions on the distribution of data, such as $\beta$-mixing condition, which is very hard to verify in complex time series data. Besides, we politely disagree that Oliveira et al. (2024) shows the contribution of adaptivity procedures is negligible. First, Oliveira et al. (2024) only conducted the numerical comparison with one adaptive procedure (DtACI) in one real data. Second, even under the same assumptions, Oliveira et al. (2024) did not show that split conformal prediction can strictly perform better than adaptive procedures in theory.

The authors mention difficulties in calibrating the outcomes of complex machine learning models like transformer, along with other models like deep Gaussian processes. However, the proposed techniques were evaluated on simpler time series models. It would be helpful to have some results with more complex models or discuss how the method would impact them.

We are very willing to test the effectiveness of the more complex models like Transformer! The results can be seen in additional Appendix F (see Tables 9 and 10). Consistent with other base models, ECI variants have achieved the best performance on five benchmark datasets with Transformer as expected.

Actually, conformal predictions have become popular because they are model-agnostic and able to apply to any underlying model and enhance it with predictive uncertainties post hoc. Hence, conformal methods can yield well-calibrated prediction sets based on the output of any point estimation model, irrespective of the base model’s structure.

It would be helpful to discuss the tradeoffs between different ECI variants in the context of experimental results.

We fully agree! The ECI-integral can leverage all available past data, thus incorporating a greater amount of information, particularly in stock datasets (see Table 1 and 2). However, in the climate domain where the data is characterized by a shorter temporal horizon (see Table 4), the performance of ECI-integral may be less effective. As for ECI-cutoff, we believe it outperforms ECI-integral when there exists numerous outliers, causing $s_t-q_t$ to be exceedingly large (see Table 4).

For the place where the conformal PID baseline beats the proposed method, it would be helpful to explain the scorecaster term in a couple of lines so that it is more obvious why this baseline could produce a superior performance.

Thank you for your advice! We have added some details in the experiment part.

The scorecaster term of conformal PID in the baseline utilizes the relatively accurate Theta model (which can also be replaced by AR or Transformer) to take advantage of any leftover signal and residualize out systematic errors in the score distribution. It can be regarded as an additional component that “sits on top” of the base forecaster (base model). Therefore, across all test datasets, the performance of conformal PID in the Prophet model consistently outperforms that of the AR and Theta models. This is attributed to the fact that Prophet, being a worse-performing model, is complemented by the superior performance of conformal PID's scorecaster, Theta, which is a better-performing model.

For Table 3: Did the authors try to use a similar scorecaster term within the ECI update and check if that could improve over conformal PID in the Prophet model?

Great point! We maintained the settings of Table 3 to test the performance of ECI combined with the scorecaster (Theta model), and the results (see Table 11) demonstrated that ECI-scorecaster is consistently superior over conformal PID across three base models, thereby validating the effectiveness of the ECI update. Interestingly, for the worse-performing Prophet base model, adding the scorecaster enhanced the performance of ECI, while the scorecaster tended to degrade performance when better-performing base models were used. This observation is also mentioned in Section 2.3 of the conformal PID paper: “an aggressive scorecaster with little or no signal can actually hurt by adding variance to the new score sequence”.

Also, is there any intuition for why decay-OGD baseline beats ECI in Table 5 (Prophet model)?

In the synthetic data of Table 5, there are only three changepoints, and the distribution of data between two changepoints are independent and identically distributed (i.i.d.). After the last changepoint, the distribution of data remains stable and unchanged. Once the optimal value of $q$ is reached, no further adjustments are needed. Hence, the decay mechanism in later stages effectively reduces the learning rate $\eta$, which benefits the performance of decay-OGD.