QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions

We thank the reviewer for the careful assessment of our work and the thoughtful suggestions. Below, we respond to particular concerns.

[Q1: Comparisons with more methods, such as those in Section 3.2]

With respect to baselines, prior PPI variants are incompatible with the measures considered in our experiments; QuEst is the only existing specialized method for rigorously combining labeled and imputed data to estimate and provide CIs on these measures. We will clarify these choices in the final draft, specifying our exact baselines and emphasizing why specialized PPI methods do not directly apply to quantile-based distributional measures. This should ensure readers understand both the rationale behind our baseline selection and the distinct advantages offered by QuEst.

Also, in order to further respond to these concerns, we have completed additional experiments on estimating the $\beta-VaR$ QBDM (Table 1). This experiment does allow for a direct comparison to PPI, since it is estimating a single quantile. We find that QuEst produces higher quality estimates than our usual suite of observed and imputed baselines as well as the PPI method from Angelopoulos et al. Representative figures will be included in the camera-ready version of the paper.

[Q2: The motivation and advantages of our extension in Section 5.]

We here address the reviewer's question from two aspects:

• Motivation: We leverage the fact that our estimator remains asymptotically unbiased regardless of which weighting function $\tilde{\psi}$ is chosen, because the expectations of its two terms cancel out. Thus, by letting $\tilde{\psi}$ be a flexible tuning function---rather than simply setting $\tilde{\psi} = \psi$---we can optimize for better debiasing and variance reduction.

• Advantage: Instead of tuning a single scalar, we introduce a basis-function approach (e.g., polynomial or ``spline'' basis) and optimize the associated coefficient vector $\xi$. This advanced method, backed by new theorems and techniques, can yield more efficient estimators in both synthetic and real-world settings, as it provides added flexibility in controlling bias and variance.

[Q3: Imputed data quality]

With respect to imputation quality, we emphasize that QuEst’s adaptive nature ensures that it remains valid even if the imputed data deviates substantially from ground truth: if the imputed data are of high quality, then we choose a large $\lambda$ to weight more on the imputed data in the estimator. On the other hand, if the imputed data is of low quality, we will set smaller weight to imputed data (if $\lambda=0$, it just uses the gold-standard data alone). Since we are optimizing the tuning parameter $\lambda$, we would expect our estimator can always perform better than only using gold-standard label alone. Empirically (as in Fig. 4 with correlation $<0.1$), the estimator self-corrects by increasingly falling back on the smaller observed dataset if predictions are poor (with the $\alpha$ parameter).

[Q4: Additional assumptions]

As it pertains to assumptions, our estimator achieves asymptotic variance reduction relative to naive estimates without imposing additional assumptions beyond those enumerated in the paper. As mentioned in the previous question, our framework can automatically adapt to the quality of imputed data and we would expect our estimator can always perform better than only using gold-standard label alone.

[Q5: Finite-sample properties of the proposed estimator]

There are two points we want to address. 1. Building central limit theorems actually only require a couple of hundreds gold-standard labeled data, so in practice, our framework works quite well with limited gold-standard labeled data. 2. If we falls into the regime of extreme small data, i.e., we only have less than 100, then the data uncertainty will be too huge and no statistically principled methods would work in the distribution-agnostic setting (this can be proved by using "no free-lunch theorem" in learning theory). We have to impose more assumptions to gain stronger results.

We thank the reviewer for the thoughtful remarks. Please see below for responses to individual concerns.

[Q1: Imputed data quality]

First, we would like to clarify that our remark in the limitations section concerned the problem that if we want to have significant gain in variance reduction, we need relatively high-quality predictions. There are two points we want to address here regarding methodology:

• We want to emphasize, relatively high-quality imputed data is enough to observe a significant gain in variance reduction. We don't need $100\%$ correct annotations for unlabeled data. Actually, in our experiments (Figure 4), we show that QuEst performs effectively with correlation of less than 0.1 (a very noisy regime) between the observed and imputed labels, and we still observe significant benefit by incorporating the imputed data in the estimator.

• Another benefit of this framework is that our framework can automatically adapt to the quality of imputed data: if the imputed data are of high quality, then our algorithm will choose a large $\lambda$ to weight more on the imputed data in the estimator. On the other hand, if the imputed data is of low quality, our algorithm will set smaller weight to imputed data (if $\lambda=0$, it just uses the gold-standard data alone). Since our algorithm is optimizing the tuning parameter $\lambda$, we would expect our estimator can always perform better than only using gold-standard label alone.

We will explicitly clarify this distinction in the final draft and emphasize that QuEst automatically debiases poor imputation quality and gracefully reverts to using the observed data as the primary signal.

[Q2: Hyperparameters]

Also, in response to the reviewer’s concern we will further clarify that our core estimator does not involve any hyperparameters in the closed-form tuning for $\lambda$, which we derived theoretically. We believe that our empirical results across five distinct datasets and a wide range of labeled/unlabeled splits suggest this tuning is robust.

[Q3: Variance in Figure 2]

With respect to Figure 2, we ran 2000 trials, and the resulting variance in Figure 2 is extremely small, rendering the error bars nearly invisible for most of the plots.

[Q4: Prompt design]

Although we did not vary prompt design in our study, QuEst remains valid regardless of how the imputed data is generated—it will simply adapt to the correlations observed between imputed and true labels. Empirically, we show that even inaccurate predictions can be used effectively by QuEst: for example, accuracy on OpinionQA is roughly 50%, but the QuEst estimate still consistently beats baselines.

[Q5: Other baselines]

With respect to baselines, prior PPI variants are incompatible with the measures considered in our experiments; QuEst is the only existing specialized method for rigorously combining labeled and imputed data to estimate and provide CIs on these measures. Thus we compared against the "classic" approach of using only the labeled data, as well as the use of imputed data only. For the confidence intervals, our "classic" baseline still requires our highly non-trivial CLT derivation (with $\lambda=0$). We will clarify these choices in the final draft, specifying our exact baselines and emphasizing why specialized PPI methods do not directly apply to quantile-based distributional measures. This should ensure readers understand both the rationale behind our baseline selection and the distinct advantages offered by QuEst.

Also, in order to further respond to these concerns, we have completed additional experiments on estimating the $\beta-VaR$ QBDM (Table 1). This experiment does allow for a direct comparison to PPI, since it is estimating a single quantile. We find that QuEst produces higher quality estimates than our usual suite of observed and imputed baselines as well as the PPI method from Angelopoulos et al. Representative figures will be included in the camera-ready version of the paper.

We thank the reviewer for their time and consideration in reviewing our work. Here we respond to specific concerns raised in the review. We hope that our answers can resolve the misunderstanding and help to increase our score.

[Q1: Novelty compared to existing work.]

At the first glance, the surface form of Eq. (2) might resemble that of Angelopoulos et al. (2023). However, there is a significant difference between our frameworks.

QuEst advances the prediction-powered inference paradigm beyond M-estimators (e.g., means, single quantiles) to fully encompass L-estimators, which can express diverse distributional measures like VaR, CVaR, and multi-quantile segments. This expansion requires new theoretical machinery, as our estimator cannot be straightforwardly written into sums of i.i.d. random variables. We rely on techniques from L-statistics, e.g., Hajek decomposition, and we also need to prove that $\mathbb{E}Q_\psi(F_n)-Q_\psi (F)=o_p(1/\sqrt{n})$ for certain $\psi$ functions, thereby establishing asymptotic normality under suitable conditions. Furthermore, QuEst integrates the concept of “power tuning” into this broader class of estimators and extends it with more sophisticated (spline-function-based) corrections in Sec.5, improving statistical efficiency in scenarios where a simple scalar multiplier would be insufficient. We also address multidimensional estimation, allowing simultaneous handling of multiple distributional measures (e.g., tail segments of multiple metrics) in a way that is not possible in prior PPI methods.

From a practical standpoint, this generalization to L-estimators is essential for real-world tasks where the distributional properties themselves matter, such as identifying extreme behaviors or analyzing key population subgroups. QuEst provides asymptotically unbiased estimates and valid confidence intervals for quantities of substantial interest in domains like finance, public policy, and healthcare. Alongside the theoretical results, we present new variance formulas, covariance expressions, and performance bounds tailored to the L-estimator setting, offering a blueprint for rigorous inference on an array of tail or segment-based metrics. In sum, QuEst goes far beyond merely adapting an existing correction term; it is a comprehensive framework that systematically enables distributional inference where M-estimator-based prediction-powered approaches do not apply.

[Q2: Use of splines in extension]

Sorry for causing confusion. We are referring to the result in our “Extension” section. There, we introduce a more complicated version of power tuning instead of just tuning a multiplicative scalar lambda. Specifically, we consider tuning the coefficient vector for a given basis function in $\xi^T\phi(\cdot)$ (the tuning parameter is $\xi$ here). We specifically tried polynomial function basis, that is why we call this “spline function based”. Notice that it is a completely new and more advanced tuning method and we introduce new theorems and new techniques to get more statistically efficient estimators.

[Q3: “Classic” baseline for confidence intervals]

With respect to baselines, prior PPI variants are incompatible with the measures considered in our experiments; QuEst is the only existing specialized method for rigorously combining labeled and imputed data to estimate and provide CIs on these measures. Thus we compared against the "classic" approach of using only the labeled data, as well as the use of imputed data only. For the confidence intervals, our "classic" baseline still requires our highly non-trivial CLT derivation (with $\lambda=0$). We understand that this was not clear in the original submission, and would definitely clarify these details in our future revision.

[Q4: Other baselines]

In order to respond to these concerns, we have completed additional experiments on estimating the $\beta-VaR$ QBDM (Table 1). This experiment does allow for a direct comparison to PPI, since it is estimating a single quantile. We find that QuEst produces higher quality estimates than our usual suite of observed and imputed baselines as well as the PPI method from Angelopoulos et al. Representative figures will be included in the camera-ready version of the paper.

We thank the reviewer for the thoughtful feedback and for recognizing the contribution of QuEst. Below we address your concerns and questions.

[Q1: Relation to existing approaches]

This is correct. Existing PPI methods deal with estimators written in the form of sums of i.i.d. random variables. We strictly generalize it by moving beyond M-estimators to a new class of L-estimators (i.e., quantile-based distributional measures). Specifically, if we choose the weighting function as 1, it will become the mean estimator of a loss function, which is the objective studied in PPI. So the family of estimators we study is a strict generalization of the previous result. The family of estimators we investigate are no longer straightforwardly in the form of sum of i.i.d. random variables as those studied in PPI, thus, it incurs new challenges and requires new techniques to build central limit theorems.

For example, in Appendix C.1.1, in order to prove CLT, we need to prove $\mathbb{E}Q_\psi(F_n)-Q_\psi (F)=o_p(1/\sqrt{n})$ for certain $\psi$ functions. Then, we need to use techniques in L-statistics such as Hajek decomposition to build the final CLT. In practice, this shift addresses important real-world needs, such as assessing tail behaviors, population subgroups, or other segments of interest that existing PPI methods do not handle. Our derivations yield asymptotically unbiased estimators and valid confidence intervals for these more complex target measures. Thus, QuEst closes a gap in the literature by providing a unified approach that not only captures means or single quantiles, but also richer distributional statistics of concern in diverse and important fields like biology, economics, and healthcare.

[Q2: Handling L-Estimators]

This is mostly correct but not precise. Because in line 155, what we mean is that if we choose the weighting function as a special one (constant function 1), it will become the mean estimator of a loss function, which is the objective studied in PPI. Instead of claiming we can handle all L-estimators, our claim is that the family of estimators we proposed for QBDMs falls into the category of L-estimators. As for further generalization over QBDM, we can indeed generalize a bit to handle a wider range of other L-estimators (more general weighting function, and special nonlinear functional of $Q$). But a main focus here is on quantile-based distributional measures, as they are especially relevant in high-stakes applications like tail risk analysis and policy decisions.

[Q3: Clarifying baselines]

For baselines, prior PPI variants are incompatible with the measures considered in our experiments; QuEst is the only existing specialized method for rigorously combining labeled and imputed data to estimate and provide CIs on these measures. Thus we compared against the "classic" approach of using only the labeled data, as well as the use of imputed data only. For the confidence intervals, our "classic" baseline still requires our highly non-trivial CLT derivation (with $\lambda=0$). We will clarify these choices in the final draft, specifying our exact baselines and emphasizing why specialized PPI methods do not directly apply to quantile-based distributional measures. This should ensure readers understand both the rationale behind our baseline selection and the distinct advantages offered by QuEst.

To further address these concerns, we have completed additional experiments on estimating the $\beta-VaR$ QBDM (Table 1). This experiment does allow for a direct comparison to PPI, since it is estimating a single quantile, and is thus the only QBDM computable with PPI. We find that QuEst produces higher quality estimates than our usual suite of observed and imputed baselines as well as the PPI method from Angelopoulos et al. Representative figures will be included in the revised version of the paper.

[Q4: CVaR]

We do not fully understand this point. Our focus on CVaR as one of our primary QBDMs of interest was inspired by its use in the financial mathematics literature. Our innovation applies to this measure; our "classic" baseline corresponds to its application based on that literature. To our knowledge, there are no other known techniques for combining gold-standard and imputed data to enhance CVaR estimates.