Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression

We greatly appreciate your review and your suggestions for improving the work. We especially appreciate your thoroughness in reading the paper, and will be sure to fix any typos currently in the draft for the camera ready version of the paper. Below we respond to your individual points.

Model Evaluation vs. Statistical Inference

Throughout this work, we empirically and theoretically consider the problem of mean estimation. Statistical inference in the context of science and model evaluation are two interesting applications of this problem, and so we consider them both. However, since these problems are so related, we believe generalizing both of these as mean estimation better represents our methods' broad applicability.

Error Bars on MSE Plots

Thank you for your excellent point about the inclusion of error bars in our plots. For the camera ready version of the paper, we will re-run the experiments and include appropriate error bars. Here is a small set of 95% confidence intervals for the normalized MSE (see figure 1) of different methods across 350 trials. Here the confidence is represented with a tuple (lower bound, upper bound):

Sigmoid Function Improving Performance

Using the sigmoid function in Sigmoid-PPI can improve performance when the mapping from $f(X)$ to $h(X)$ is not linear. For instance, since $h(X)$ is a binary value in the refusal rate dataset, mapping $f(X)$ to (0,1) with a sigmoid function could lead to a lower MSE in the regression problem described in expression 7, which we showed to be equivalent to the variance of PPI.

Thank you very much for your insightful review. Your questions and comments have helped to make the draft much stronger and clearer. In the camera ready version of the paper, we will improve the related works section and better describe the LLM refusal dataset. We address your individual concerns below.

Use of MAE instead of MSE

Evaluating our method in terms of MSE is an excellent suggestion. MAE was initially used as it is more interpretable. In the camera-ready version of the manuscript, we will be sure to use MSE instead of MAE.

This change produces nearly identical results to that which we presented in the paper. We have included a small set of results to demonstrate this.

Expanding Section 6, Further Theoretical Justification for Ridge-PPI

Several reviewers requested further investigation of the theoretical analyses on where PPI++ succeeds and fails in section 6, as well as an extended theoretical justification of Ridge-PPI.

Towards addressing these concerns, we below produce an expression for the excess risk of Ridge-PPI - an analogous expression to Expression 13 in the paper that makes the same convergence and independence assumptions. We use the shorthand $\hat{\mu}_{PPI_a}$ for the PPI estimate (Expression 3 in the paper) using the Ridge-PPI lambda (Expression 8), as well as $V:= Var$ \hat{Cov}[f(X), h(X)] $(2\mathbb{E}[f(X)]^2 + (\frac{1}{N} + \frac{1}{n})Var[f(X)])$:

$

Var[\hat{\mu}_{PPI_a}] -Var[\hat{\mu}_h]
= \frac{V}{(Var[f(X)] + \alpha)^2} 

$

$

- \frac{Var[h(X)]Corr(h(X),f(X))^2}{(1+\frac{n}{N})n} + \frac{Var[h(X)]Corr(h(X),f(X))^2\alpha^2}{(1+\frac{n}{N})n(Var[f(X)] + \alpha)^2} 

$

Taking the derivative of this expression and setting it to zero yields the solution:

$

\alpha^* = \frac{(1+\frac{n}{N})nV}{Cov[f(X), h(X)]^2}

$

This quantity is difficult to estimate in practice, but supplies more insight into the dynamics of PPI. Notably, this optimal $\alpha$ is smaller in the case of greater covariance, and greater in the case of large V. Seeing as V is a large positive term in the risk, this expression balances between the potential variance and the potential variance reduction permitted by $Cov[f(X), h(X)]^2$.

We can also now calculate the expected risk reduction from using Ridge-PPI vs PPI++:

$

Var $\hat{\mu}_{PPI_a}$

• Var $\hat{\mu}PPI$

$

$

= \frac{\alpha}{(Var[f(X)] + \alpha)^2}(\frac{Var[h(X)]Corr(h(X),f(X))^2\alpha}{(1+\frac{n}{N})n} - V\frac{2Var[f(X)] + \alpha}{Var[f(X)]^2})

$

This expression further demonstrates the potential harms of small $Var[f(X)]$: the smaller $Var[f(X)]$ is, the greater the improvement from using Ridge-PPI over PPI++.

Cross Validating $\lambda$

We performed some initial experiments using cross validation to fit a scalar regression parameter on a synthetic regression task. We found that while it can perform well in some circumstances, it requires two key assumptions: 1) that one knows a priori the possible range of regressor values, and 2) that one can use a dense grid of values to validate over (e.g. a grid of 50 options in the [0, 1] interval works reasoanbly well, a grid of 10 does not).

When these are both true, this cross-validated regression strategy is a constrained function class that can reduce variance. However, our use of cross validation for selecting an $\alpha$ parameter did not have these assumptions, as the $\lambda$ values we were effectively searching over were exclusively ridge regressors. We appreciate this comment, however, as it helped us to improve our understanding of the scalar regression problem and exemplified the computational efficiency of Ridge-PPI.

Creating Confidence Intervals

Reviewer NpF2 shared this concern. Due to the lack of a general rebuttal as well as space restrictions on individual rebuttals, we have included the response to this in the rebuttal to reviewer NpF2. Please read it to see our additional explanation for this issue.

Research Datasets and Var[f(X)]

We investigate whether the trends observed in Section 5.4 are persistent in the research datasets by calculating the ratio of the MSE at $n=5$ for PPI++ to the MSE at $n=5$ for Ridge-PPI for each dataset and taking its correlation with $Var[f(X)]$ for that dataset. We find that there is a strong anticorrelation between these values (Pearson r = -0.69), further supporting our hypothesis that small $Var[f(X)]$ hinders the efficacy of PPI++. While we do not have enough datapoints to make this correlation statistically significant, it serves as motivation for the more thorough analysis we present in section 6.

We would like to express our appreciation for your thoughtful review and constructive critique. We look forward to making the writing more clear following your points. Specifically, we will amend the abstract and make the description of the LLM refusal dataset more clear. We respond to individual points of yours below.

Appendix A

Sigmoid PPI is intended for use with small sets of labelled examples. Within this regime, we see Sigmoid PPI regularly match the original PPI++ or make significant improvements.

However, one potential fix for the results seen in Appendix A would be using the following alternative formula:

$

g'(f(X)) := \frac{1}{1 + \frac{n}{N}}\frac{1}{1 + exp(-\alpha f(X) + \beta)},

$

This would have the estimator behave more similarly to the classical estimator as n -> N and the classical estimator becomes stronger. We are currently experimenting with this modification and hope to include them in the camera ready version of the paper. Below we provide promising results of average MSE at the largest possible $n$ for the research datasets, demonstrating how this method (Sigmoid-Div) improves asymptotic performance:

Expanding Section 6, Further Theoretical Justification for Ridge-PPI

This was a shared concern for reviewer aoGS as well. Due to the lack of a general rebuttal as well as space restrictions on individual rebuttals, we have included the response to this in the rebuttal to reviewer aoGS. Please read it to see our additional analysis for this issue.

Bias of PPI++

Footnote 1 in the paper indicates that for small $n$, PPI++ will be a biased estimator. Prior work only considered the asymptotic unbiasedness of $\lambda$ and there is no existing expression for this bias with small samples. Here, we produce such an expression and are happy to include it in the paper if the reviewer finds it helpful. The expression for this bias is the following (revealed by taking the expectation of PPI assuming $\lambda$ is not independent from the training data):

$

E[\hat{\mu}_{PPI}] - \mu^* = -Cov[\hat{\lambda}, \hat{\mu}_f]

$

We note that this expression can be applied to any estimator of $\lambda$, such as Ridge-PPI. Using the definition of covariance can make this expression more interpretable:

$

E[\hat{\mu}_{PPI}] - \mu^* = \sqrt{Var[\hat{\lambda}]Var[\hat{\mu}_f]}Corr[\hat{\lambda}, \hat{\mu}_f]

$

We note that we generally do not include this bias term as it can be ignored in cases where we assume that the $\lambda$ parameter can be fit independently, such as section 6.1. Furthermore, since our methods focus on variance reduction (in line with prior literature) and were effective at reducing overall MSE/MAE, we consider a complete treatment of this bias to be out of scope for this work.

Other M-Estimators

We chose to focus our work on specific insights into the dynamics of PPI when used with mean estimation. While other M-estimators are interesting as well, we find them out of scope for our work, which is specifically about the small sample behavior of PPI for mean estimation. We believe the small sample focus of our work validates the contribution as novel and useful, despite not considering a wide range of M-estimators.

Creating Confidence Intervals

We note that throughout this work we did not consider the efficacy of constructing confidence intervals for any regime, since the central limit theorem (which past methods rely on) tends not to hold in small data regimes. Since improving the efficacy of PPI in the small data regime was our primary concern, those intervals would not be valid for any of the methods that we compared with.

However, the basic machinery from Angelopoulos et al. [2023] will work to create confidence intervals for Ridge-PPI, as constructing the interval is not impacted by how $\lambda$ is selected. Additionally, for Sigmoid-PPI, assuming that the parameters of the transformation are fit independently as in section 6.1, the confidence interval can be created using a very similar formula to Angelopoulos et al. [2023], with the exception that the transformed predictions are used in place of the original predictions:

$

\hat{\mu}_{PPI_g} \pm 1.96*\sqrt{\frac{\hat{Var}[g(f(X_i)) - h(X_i)]}{n} + \frac{\hat{Var}[g(f(X_i^u))]}{N}}

$

First of all, we thank you for your constructive feedback and positive reception of the work. We look forward to improving the manuscript based on your review. We will be sure to improve readability of the figures for the camera ready draft. Below, we respond to individual points you raised.

Optimal Lambda Risk Expression

While Expression 5 in the paper does describe the risk of PPI for a static lambda, we substitute the optimal lambda presented in Expression 6 below.

$

=  \frac{1}{n}(Var[ h(X) ] + \frac{1}{Var[f(X)]} -  2 Var[h(X)]Corr[h(X), f(X)]^2).

$

Error Bars on MSE Plots

Thank you for your excellent point about the inclusion of error bars in our plots. For the camera ready version of the paper, we will re-run the experiments and include appropriate error bars. Here is a small set of 95% confidence intervals for the normalized MSE (see figure 1) of different methods across 350 trials. Here the confidence is represented with a tuple (lower bound, upper bound):