Quantile-Free Regression: A Flexible Alternative to Quantile Regression

Thank you for your thoughtful comments and suggestions! We address each of the points raised in turn below.

Q1 - Crossing quantiles.

We appreciate this reviewer raising the subject of theoretical guarantees around the avoidance of crossing of the upper and lower bound of the intervals as this should be considered an advantage of our approach that we have not emphasized in the text. While explicitly linking the quantiles to being specific upper and lower bounds is necessary in quantile regression (due to the requirement of estimating quantiles), this is not the case for our method which directly estimates intervals. Our proposed objective is invariant to permutations of the upper and lower bounds, and simply sets the lower bound to be the minimum and the upper bound to be the maximum of the two values. Therefore, which output neuron acts as either bound can even change on a per-example basis. Formally, this can be observed in the $\kappa = (y - \mu_l)(y - \mu_u)$ term in our loss function which is clearly invariant to permutations between $\mu_l$ and $\mu_u$. We appreciate that this may have been obfuscated by our notation using subscripts $l$ and $u$ which we have now changed to better reflect that neither of these values are explicitly tied to a particular bound. We consider this to be a distinct advantage of our direct interval prediction approach which sidesteps the conceptual issue of crossing quantiles which occurs due to the independent estimation of the upper and lower bounds in quantile-based approaches.

Action taken: We have highlighted our advantages in terms of non-crossing intervals in the text while updating our notation to reflect this.

Q2 - Additional theoretical analysis.

We agree that further theoretical analysis of the behavior of our estimator would be a valuable addition to this work. We have derived a new theorem for our proposed QFR loss $\mathcal{L}^{QFR}$ demonstrating that the interval resulting from the minimization of the loss directly covers the right proportion of samples that correspond to the targeted coverage, regardless of the size of the sample (for ease of access, we provide the new proof at this link). This property is called the “quantile property” when true for a quantile estimator loss function [1] and illustrates that our proposed expression is well motivated. This reassuring property reinforces our claim that this expression provides a suitable objective for the goal of generating well-calibrated prediction intervals in the finite sample setting.

Action taken: We have added this new theorem to the manuscript.

Thank you for your thoughtful comments and suggestions! We address each of the points raised in turn below.

Q1 - Non-asymptotic guarantees

We agree that some non-asymptotic guarantees on the behavior of our estimator would be a valuable addition to this work. We have derived a new theorem for our proposed QFR loss, $\mathcal{L}^{QFR}$, demonstrating that the interval resulting from the minimization of the loss directly covers the right proportion of samples that correspond to the targeted coverage, regardless of the size of the sample (for ease of access, we provide the new proof at this link). This property is called the “quantile property” when true for a quantile estimator loss function [8], and illustrates that our proposed expression is well motivated. This reassuring property reinforces our claim that this expression provides a suitable objective for the goal of generating well-calibrated prediction intervals in the finite sample setting.

Action taken: We have added this new theorem to the manuscript.

Q2 - Why is QFR preferable?

We thank the reviewer for raising this point as it is something that should have been made more clear in the text. The key idea of our work is that although several methods might achieve a desired level of marginal coverage with their intervals, this is an underspecified problem with potentially an infinite number of admissible solutions. Given this, any interval-producing method is required to introduce some additional regularization (either implicit or explicit) in order to select among the possible solutions. In the case of quantile regression, the pair of symmetric quantiles are selected. However, given that other attributes are typically desired in these intervals (as illustrated by the large body of work that attempts to minimize interval width [1-3] or maximize conditional coverage [4,5]), there are likely more preferable solutions than quantile regression. One real-world example where this is the case is in renewable energy sources where it has been noted that “the probability distribution of the renewable energy source’s power output is generally skewed, thereby the width of CPIs is often unnecessarily wide” [6].

In our work, we ensure this decision is made explicitly by the user such that the most appropriate intervals can be found for a given application by using the corresponding regularization term. Our experiments in Tables 3 & 4 then illustrate that our objective results in intervals that better achieve these properties when evaluated on standard benchmarks. For Table 3 this improvement is not surprising as symmetric intervals are generally not the most narrow possible intervals (because real-world distributions are unlikely to be perfectly symmetric). For Table 4 this improvement is also expected as it provides greater flexibility on the solution to select one that achieves greater independence between width and miscoverage events.

We appreciate that the uncertainty quantification literature is vast and there exist many differing approaches. In Appendix A we provided an extended discussion on this literature which highlighted that single model approaches such as quantile regression have distinct advantages that make research in this area valuable in its own right. Factors such as their minimal computational overhead and lack of assumptions on the data have resulted in the extensive application of these methods in practice (see e.g. [7] for an overview of just some of the vast number of real-world implementations of this method). We believe it would fall outside the scope of this conference paper to survey and compare the various categories across all of the uncertainty quantification literature while accounting for their relative strengths and weaknesses to the appropriate rigorous standard required for a conference such as ICLR.

Action taken: We have updated our text to better reflect this motivation.

Q6 - QFR outperforming QR at achieving coverage?

This is an astute observation and not something we explicitly addressed in the paper! We have two hypotheses why this might be the case.

The first is simply that quantile regression introduces two sources of error (i.e. estimating each of the two quantiles) in which an error on either estimate can result in miscoverage. This is not the case for methods that estimate a single quantity (i.e. the interval directly). Note that we are not the first to notice this fact, see e.g. Sec. 2 of [12] for a much earlier work that has commented on this limitation.

The second is that miscoverage may also be due to a reduction in the variance of estimation of statistical quantities in high-density regions of a probability distribution. To illustrate this, consider the distribution we presented in Fig 1 of the main text where the ground truth intervals are shown. Now suppose we take a sample from this distribution (i.e. a training set) and estimate both the empirical minimum width intervals and the empirical quantiles. In both cases, there is likely to be some estimation error. If we were to repeat this process multiple times we note that the variance around the estimates in lower-density regions (where the quantiles lie) is likely to be greater than in the high-density regions (where the minimum width intervals lie). As a consequence of this, quantile estimation methods may suffer from larger errors in estimating true quantiles (particularly in small sample regimes) resulting in more difficulties in obtaining exact coverage.

Edit: We have now added an experiment in which we verify that the variance in the estimation of the bounds is indeed greater in the lower-density regions generally estimated by quantiles as described previously. We illustrate this effect on the skewed distribution presented in Figure 1 (see this link for results). This result has now been added to the main text.

Action taken: We have added a note reflecting this in the text.

References

[1] Koenker, Roger, and Kevin F. Hallock. "Quantile regression." Journal of economic perspectives 15.4 (2001): 143-156.

[2] Chung, Youngseog, et al. "Beyond pinball loss: Quantile methods for calibrated uncertainty quantification." Advances in Neural Information Processing Systems 34 (2021): 10971-10984.

[3] Tagasovska, Natasa, and David Lopez-Paz. "Single-model uncertainties for deep learning." Advances in Neural Information Processing Systems 32 (2019).

[4] Brando, Axel, et al. "Deep non-crossing quantiles through the partial derivative." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

[5] Pearce, Tim, et al. "High-quality prediction intervals for deep learning: A distribution-free, ensembled approach." International conference on machine learning. PMLR, 2018.

[6] Landon, Joshua, and Nozer D. Singpurwalla. "Choosing a coverage probability for prediction intervals." The American Statistician 62.2 (2008): 120-124.

[7] Huang, Shih-Feng, and Hsiang-Ling Hsu. "Prediction intervals for time series and their applications to portfolio selection." REVSTAT-Statistical Journal 18.1 (2020): 131-151.

[8] Vovk, Vladimir. "Conditional validity of inductive conformal predictors." Asian conference on machine learning. PMLR, 2012.

[9] Feldman, Shai, Stephen Bates, and Yaniv Romano. "Improving conditional coverage via orthogonal quantile regression." Advances in neural information processing systems 34 (2021): 2060-2071.

[10] Hunter, David R., and Kenneth Lange. "Quantile regression via an MM algorithm." Journal of Computational and Graphical Statistics 9.1 (2000): 60-77.

[11] Dey, Biprateep, et al. "Calibrated predictive distributions via diagnostics for conditional coverage." arXiv preprint arXiv:2205.14568 (2022).

[12] Takeuchi, Ichiro, et al. "Nonparametric quantile estimation." (2006).

Thank you for your thoughtful comments and suggestions! We address each of the points raised in turn below.

Q1 - Coverage levels and monotonicity

We appreciate the suggestion to consider target coverage levels beyond the 90% already included in the text. We focused on 90% coverage as this is the standard level used in previous works throughout the literature [1-3]. However, it is indeed true that different applications may require different coverage levels and, therefore, investigating the impact of adjusting this value may be valuable to future readers. Below we include equivalent results to Table 3 from our main text evaluated at the 50% coverage level instead. We include interval width and the coverage level achieved in parentheses $\pm$ a standard error for both.

Interestingly, both methods have increased variance in their estimates at this level. It appears that as coverage decreases the intervals become narrower and, just as we would expect as approaching point estimates, the variance of the miscoverage rate increases. The mean standard errors for 90% coverage rate are 0.99 and 0.57 for QFR-W and QR respectively while they are 1.9 (QFR-W) and 2.1 (QR) for the 50% coverage rate. Using the same definition of achieving coverage as included in the main text, we find that the only case of not achieving coverage is QR on Boston. Of the remaining 8 cases, QFR-W obtains narrower intervals in 6/8 cases, tying 1/8 and QR obtains narrower intervals in 1/8. Overall this is consistent with previous results indicating that QFR-W also obtains more narrow intervals when we consider alternative coverage levels.

We appreciate this reviewer raising the subject of monotonicity of the intervals as this should be considered an advantage of our approach that we have not emphasized in the text. While explicitly linking the quantiles to being specific upper and lower bounds is necessary in quantile regression (due to the requirement of estimating quantiles), this is not the case for our method which directly estimates intervals. Our proposed objective is invariant to permutations of the upper and lower bounds, and simply sets the lower bound to be the minimum and the upper bound to be the maximum of the two values. Therefore, which output neuron acts as either bound can even change on a per-example basis. Formally, this can be observed in the $\kappa = (y - \mu_l)(y - \mu_u)$ term in our loss function which is clearly invariant to permutations between $\mu_l$ and $\mu_u$. We appreciate that this may have been obfuscated by our notation using subscripts $l$ and $u$ which we have now changed to better reflect that neither of these values are explicitly tied to a particular bound.

Action taken: We have added these results extending our analysis to a new coverage target to the main text. We have also highlighted our advantages in terms of monotonicity in the text while updating our notation to reflect this.

Q2 - Figure 2 intervals.

The reviewer is absolutely correct in pointing out that the target values should not lie entirely within the intervals if they are achieving <100% coverage. The intention of this figure was to illustrate that QFR-W produces generally narrower intervals than QR by comparing their intervals on the test set sorted by target values. For clarity of illustration, we applied a consistent smoothing to both methods and included the unsmoothed version in the Appendix (see Fig. 3) which we referenced in the text. We note that in the unsmoothed version of this plot it is clear that there are several cases of miscoverage, consistent with both methods achieving 90% coverage as expected. However, the key point holds in both cases that QFR-W finds narrower intervals in aggregate.

Q3 - Method name and purpose.

We appreciate the reviewer's feedback on the name of the method. We will spend some time considering alternatives for a final version of this paper.

“The new method basically uses a new loss to estimate the quantiles, so it is not really quantile-free” - This is not quite true but we appreciate this is a subtle point and the reviewer's general sentiment (that we should make a clearer distinction between our method and quantile based methods) may still be correct. Existing methods first estimate quantiles from data and then use those quantiles to construct an interval. Our method never explicitly estimates quantiles and simply finds intervals that achieve coverage. Indeed every interval must correspond to some quantile, but the corresponding quantile could change for every data point in the dataset. Therefore, these intervals are not associated with any particular quantiles in a meaningful way.

“the main purpose of the quantile regression is to estimate quantiles, and constructing intervals is typically seen as a byproduct” - It is correct that the original Koenker & Bassett (1978) introduction of quantile regression was primarily interested in the task of estimating quantiles. Recent works have generally referred to the task of constructing intervals via the estimation of quantiles by the name quantile regression too (see e.g. [2,3,5,6]). Especially in the deep learning context, the latter concept has been more prominent with the primary interest in constructing intervals. We will clarify this distinction by highlighting that by “quantile-free regression” we are specifically interested in “quantile-free interval regression”.

When we referred to our method being a direct replacement for quantile regression we had this second, more recent definition of quantile regression in mind. We appreciate the reviewer pointing out that there is some ambiguity in this definition and have therefore updated the phrasing “a direct replacement for quantile regression [...]” to “a direct replacement for quantile regression based interval construction [...]”.

Action taken: We will add further distinction in the text between estimating a quantile and estimating an interval. We have also updated the aforementioned phrasing in the abstract.

References

[1] Zhang, Yufan, Honglin Wen, and Qiuwei Wu. "A contextual bandit approach for value-oriented prediction interval forecasting." IEEE Transactions on Smart Grid (2023).

[2] Chung, Youngseog, et al. "Beyond pinball loss: Quantile methods for calibrated uncertainty quantification." Advances in Neural Information Processing Systems 34 (2021): 10971-10984.

[3] Tagasovska, Natasa, and David Lopez-Paz. "Single-model uncertainties for deep learning." Advances in Neural Information Processing Systems 32 (2019).

[4] Pearce, Tim, et al. "High-quality prediction intervals for deep learning: A distribution-free, ensembled approach." International conference on machine learning. PMLR, 2018.

[5] Fasiolo, Matteo, et al. "Fast calibrated additive quantile regression." Journal of the American Statistical Association 116.535 (2021): 1402-1412.

[6] Maciejowska, Katarzyna, Jakub Nowotarski, and Rafał Weron. "Probabilistic forecasting of electricity spot prices using Factor Quantile Regression Averaging." International Journal of Forecasting 32.3 (2016): 957-965.

Thank you for your thoughtful comments and suggestions! We address each of the points raised in turn below.

Q1 - Gains over baselines

There is a slight misunderstanding here on the goal of our method. We have updated our text to further clarify our writing as this is the fundamental point of our method which we don’t want future readers to miss. We will also clarify in our rebuttal next.

This reviewer has correctly noted that our core loss function, $\mathcal{L}^{QFR}$ does achieve desired coverage but only performs well at minimizing width when the additional regularization term (resulting in QFR-W) is used in tandem. The separation of the coverage achieving term and the width minimizing term into two separate expressions is the key point of our method (we were actually attempting to highlight this in Table 2!). This is because achieving coverage in finite samples is an underspedified problem with potentially an infinite number of admissible solutions. Therefore, any interval construction method is required to introduce some additional regularization (either implicit or explicit) in order to select among the possible solutions. In our work, we ensure this decision is made explicitly by the user such that the most appropriate intervals can be found for a given application. This is in contrast to QR which simply selects the symmetric quantiles and therefore produces wider than necessary intervals. This has already been noted as a limitation of quantile-based methods in practice on real-world applications (e.g. in predicting energy consumption, research has noted “the probability distribution of the renewable energy source’s power output is generally skewed, thereby the width of CPIs is often unnecessarily wide” [1]).

In fact the baseline methods typically already attempt to minimize interval width (note that e.g. [2-4] all evaluate their method based on the resulting interval widths which are somewhat reduced via various means). However, despite these efforts, all quantile-based methods are fundamentally limited by either (a) choosing symmetric quantiles resulting in unnecessary excessive width or (b) estimating more quantiles than necessary resulting in a significantly more difficult learning problem. This is the key motivation for our method which discards the intermediate step of estimating quantiles and estimates the intervals directly. We have included an extended exposition of this point in Section 3.1 of the main text.

Regarding the improvement in interval width over quantile regression we appreciate that the raw figures may not immediately suggest strong gains but we note that the reduction in interval width is up to 27.7% which is certainly a worthwhile improvement. We also note that our method statistically significantly outperforms quantile regression on 6/9 datasets while tying on the remaining 3. We would also point toward our conditional coverage results in Table 4 where the gains of QFR are compelling. Quantile regression is a very strong baseline and outperforming it by any margin is a result we believe to be useful for practitioners.

We agree that the fact that several baselines fail to maintain the desired coverage level on test data to be notable. Although discounting results that fail to achieve coverage is standard in the literature (see e.g. [3]), we also agree that this is something we should have discussed further in the text as it reflects an important dimension on which methods should be assessed. We note that all methods were trained to obtain the same coverage level, therefore failing to achieve the desired coverage level at test time is (a) not something that we can control and (b) reflects a consistent advantage of our proposed method.

Regarding the example on the dataset “yacht”, we highlight that the goal it to produce the narrowest intervals that contain 90% of test examples which is marginally better achieved by our method. While obtaining empirical coverage that is greater than the desired coverage level may often be less harmful than obtaining less than the desired coverage level, at a minimum this still reflects an inefficiency. Additionally, there are many applications in which we are primarily interested in the miscoverage cases (e.g. extreme events) and, therefore, overcoverage may be problematic in addition to being inefficient. We note that finding a set of intervals that covers 96.37% of test cases (when 90% was the target) still illustrates a failure in estimating the correct quantiles as is the stated goal of QR.

Action taken: We have updated the text to better highlight the benefits of our distinction between the coverage and regularization term. We have also added some discussion emphasizing the goal of achieving target coverage and the distinction between over and undercoverage.

Q3 - Related works

We thank the reviewer for suggesting these works. Although an interesting paper, Rosenberg et al. extends quantile regression to deal with multivariate outputs (whilst overcoming some of the limitations of previous works on that topic) and is, therefore, addressing a different problem to our work.

Brando et al. propose a method that simultaneously predicts a number of quantile levels (similar to Simultaneous Quantile Regression to which we already compare in our work) with the key innovation of ensuring that these quantiles do not cross (i.e. the 90th quantile would never cross the 85th or 95th quantiles). We evaluate this method as an additional baseline in the setup of our Table 3. As this method is a quantile estimation method and does not prescribe how to construct intervals we include two approaches (a) “narrowest” where we bin search the narrowest interval at each new inference and (b) “symmetric” we use the symmetric quantiles (0.05, 0.95). The results are included below where we report mean interval width (and empirical coverage) on the test set.

We note that, as with the previously evaluated methods for estimating all quantiles simultaneously, this approach consistently struggles to maintain coverage at test time. As we discuss in more detail in our discussion on evaluation, we seek solutions that minimize interval width for a fixed level of coverage which is not achieved by this baseline. We attribute this to two limitations. Firstly, when we select the narrowest possible interval from a set of possible intervals that each obtain marginal coverage, it is not a random choice and it is more likely that we are selecting a particular interval in which coverage is not maintained. Therefore, we would expect that picking the narrowest intervals is likely to result in more miscoverage events than selecting a fixed interval (as is reflected in these results). Secondly, learning all quantiles simultaneously is a more challenging learning problem than simply learning two quantiles for a fixed capacity model. This results in poorer predictive performance at the task in hand.

Action taken: We have added these results to the text and extended our discussion on the limitations of estimating all quantiles simultaneously.

Thank you for your thoughtful comments and suggestions! We address each of the points raised in turn below.

Q1 & Q2 - Interval width vs quantiles as an objective.

The general view throughout the uncertainty quantification literature is that although several methods might achieve a desired level of marginal coverage with their intervals, this is an underspecified problem with potentially an infinite number of admissible solutions. Therefore, previous works seek to compare solutions based on additional desirable qualities in those intervals. Probably the most popular example of such a quality is interval width which is evaluated ubiquitously (see e.g. the baseline methods to which we compare [1-3]). The idea is simply that the more narrow the region in which we have $\alpha$% confidence that the target lies, the more informative the intervals are for guiding the practitioner on what the final point value will be. The value of narrower intervals for decision-makers has been expressed more formally in previous works such as [4], which explicitly models the tradeoff between the disutility of wider intervals against the disutility of greater miscoverage rates for decision-makers. One example where minimizing interval width is of value is in financial applications where a prediction interval might be compared to some threshold value for a downstream decision (e.g. if a prediction is with $\alpha$ probability below a threshold it is a profitable buy/sell) where more fine-grained prediction intervals can allow for a higher quantity of profitable actions to be taken (see e.g. portfolio selection [5]).

However, we are generally in agreement with the sentiment of this reviewer. Which additional qualities beyond coverage are most desirable is determined by the downstream application. While narrower intervals are a common desiderata, it is indeed true that intervals that represent fixed quantiles may be useful in some cases. The flexibility to provide this decision to the practitioner was a key motivation for this work. Our proposed method optimizes for empirical coverage with the core $\mathcal{L}^{QFR}$ term and allows the user to determine which additional qualities should be optimized for. We proposed two objectives (interval width and conditional coverage) but we also hope that custom, problem-specific regularization terms could also be designed in practice.

One final point we would make is that while the interpretability of quantile-based intervals might sometimes be an aid to decision-makers it can also be harmful. As noted by the reviewer, correct estimates of quantiles would ensure the count of errors to be symmetric on either side of the interval. However, this provides no information on the magnitude of errors which could be disproportionately large on one side for a skewed distribution. Interestingly, previous work in [6] found that providing standard prediction intervals to decision-makers “may have no benefits and may even have negative effects such as reducing their responsiveness to the direction of asymmetric loss”. As there is no one-size-fits-all best solution for decision-makers, we hope that our work better reflects the value of providing practitioners with the ability to optimize predictive intervals for whatever goal is most useful to the final end user.

Regarding Q2 which is an interesting concept: much of the gains obtained from our approach are due to the conditional intervals not being explicitly tied to a fixed quantile. Therefore, this idea is not applicable to our method. However, it would be interesting to consider this as a motivation for an alternative approach to overcoming the limitation of needing to prespecify quantiles in interval regression in future work.

Action taken: We have updated the manuscript to better reflect (a) the value of interval width as an objective and (b) the value of providing practitioners with the flexibility to choose whichever interval properties provide the most utility.