Adaptive Labeling for Efficient Out-of-distribution Model Evaluation

We thank the reviewer for the thorough review and valuable feedback on our manuscript. Please refer to the global response for our detailed answers to the concerns raised and the changes we intend to make in the camera-ready version based on all the reviews. We also address the concerns raised point by point here.

Computational inefficiency: Their method requires significant computational resources, especially when the posteriors aren’t in closed form. Due to these computational constraints, Section 4.2 only considers a simple toy setting and does not compare the results with active learning and REINFORCE policy gradients. Additionally, the paper only implements one-step lookahead based on their framework, leaving the question of how to efficiently conduct multi-step planning methods unresolved.

Our work introduces a novel adaptive labeling formulation for evaluating model performance when existing labels suffer severe selection bias. As we highlight in Section 2, the adaptive labeling MDP is hard to solve because of continuous states and combinatorial actions. We make initial methodological progress by designing a one-step lookahead planning algorithm and demonstrate that even this approach can outperform existing active learning heuristics. However, this improvement comes at the cost of additional computational resources. Our formulation is thus appropriate in high-stake settings such as clinical trials where labeling costs far exceed computational costs, and we hope our new formulation spurs subsequent methodological innovations for multi-step lookaheads.

We conduct an additional experiment for planning with Ensemble+ as the UQ module to compare the performance of our algorithm with other active learning based heuristics. This comparison is conducted in a more complex setting than previously discussed in the paper. The results can be found in the attached PDF (included with global response). As shown, Autodiff 1-lookahead and REINFORCE outperforms the active learning heuristic, while the performance of the Autodiff 1-lookahead and REINFORCE is similar. Based on our extensive GP experiments in Section 4.1, we anticipate that our algorithm will perform better than REINFORCE in more complex scenarios for Ensemble $+$. This presents an interesting research question for future exploration.

Fixed and equal labeling cost assumption: The paper assumes that the modeler pays a fixed and equal cost for each label. It would be more interesting to consider variable labeling costs, which are more common in practice.

Our framework can seamlessly accommodate variable labeling costs. Specifically, we define a cost function $c$ applied on the selected subsets $\mathcal{D}^{1:T}$ and update the objective accordingly to have a term $\lambda c(\mathcal{D}^{1:T})$ where $\lambda$ is the penalty factor that controls the trade-off between minimizing variance and cost of acquiring samples.

Lack of theoretical guarantees: The paper does not provide any theoretical guarantees for their method.

We provide a theoretical explanation of why pathwise gradients outperform REINFORCE. In a tractable example we detail in the global response, we prove pathwise gradients can achieve a lower MSE than REINFORCE.

Why not evaluate how close $\mu_T$ is to the ground truth ${f^{*}}$?

Note that the modeler never has access to the ground truth $f^*$. Similar to supervised learning losses that use $Y$ as a proxy of $f^*(X)$, our objectives considers the modeler’s belief over $f^*$ based on $Y$’s observed so far.

Even in our experiments (Section 4) the $f^*$ is not known in the closed form—it is only known through an evaluation dataset generated by it and it is difficult to compare $\mu_T$ (a distribution over functions $f$) to $f^*$. In addition to comparing the variance of the L2 loss, we also show the difference between L2 loss estimated under $\mu_T$ to the true L2 loss estimated under $f^*$ (through the available evaluation set) in Figures 5 and 7.

Is $\mu_T$ always guaranteed to be a distribution concentrated around ${f^{*}}$?

If our model is well-specified, that is, the prior $\mu_0$ contains $f^*$, then we are asymptotically guaranteed to concentrate around $f^*$ as we collect more samples. This is due to the well known results from Doob [1]. Empirically, this means that if the ML uncertainty quantification methodology has enough capacity, $\mu_T$ will converge to $f^*$ asymptotically. We will highlight this point in our revision.

How should $n$ be chosen in Algorithm 3? How should the sampling size $K$ be selected, and how do different choices affect the results?

Here $n$ is simply the number of samples in the pool set we want to query labels from. The size $K$ is assumed to be a given that is input from the analyst based on acquisition cost and labeling budget.

Reference: [1] Doob, J. L. (1949). Application of the theory of martingales. In Actes du Colloque International Le Calcul des Probabilités et ses applications (Lyon, 28 Juin – 3 Juillet, 1948), pages 23–27

We thank the reviewer for the thorough review and valuable feedback on our manuscript. Below, we address the specific questions raised by the reviewer. Additionally, please refer to the global response for the changes we intend to make in the camera-ready version based on all the reviews.

The organization of Section 4 could be improved. I suggest introducing the overall description of the proposed method and the pseudo-code for Algorithm 1 after the introduction of the three key components.

We thank the reviewer for their advice on improving the presentation of the paper. As mentioned in our global response, we will incorporate all their suggestions in the final version of the paper.

Thank you for addressing my questions. I appreciate the effort put into clarifying my concerns and keep my score.

We thank the reviewer for the thorough review and valuable feedback on our manuscript. Below, we address the specific questions raised by the reviewer. Additionally, please refer to the global response for the changes we intend to make in the camera-ready version based on all the reviews.

I do not fully understand some parts: -- Can the authors provide a more detailed formulation on: (1) how the $G(\mu)$ is calculated? (2) Algorithm 2 returns $\pi_{\theta}$. Does the pseudo-label-based part also update the policy? How is this updating connected to the updating with ground truth labels acquired in line 6 of Algorithm 1?

(1) As shown in line 164, given any posterior belief $\mu$, $G(\mu)$ is defined as $\mathrm{Var}_{f \sim \mu} g(f)$. Depending on the form of $g(f)$ and the belief $\mu$, $G(\mu)$ can either be derived analytically or estimated by drawing samples $f \sim \mu$ and then calculating the empirical variance of $g(f)$ across these samples.

(2) We refer the reviewer to the diagram explaining our algorithm in the attached PDF (alongside the global response). Broadly, at each time step $t$, the input to Algorithm 2 is the current posterior state $\mu_t$, which is used to generate pseudo labels. These pseudo labels are then used to optimize the policy $\pi_\theta$ in Algorithm 2. Any update of the UQ module/posterior state within Algorithm 2 using these pseudo labels is temporary and the posterior state is reverted back to $\mu_t$ at the end of Algorithm 2. Next, we select batch $X_{t+1}$ according to the optimized policy $\pi_\theta$ from the Algorithm 2 and obtain the true labels $Y_{t+1}$. Finally, we update the UQ module/posterior belief $\mu_t$ using $(X_{t+1},Y_{t+1})$ resulting in the new posterior state $\mu_{t+1}$ at time $t+1$.

In lines 102-108 of the related work, the authors mention several active learning methods in batched settings, highlighting that they focus on classification settings while this paper mainly focuses on regression settings. Given that these methods are not used as baselines, I wonder how different the active learning settings in classification and regression are. Can methods in classification settings be easily adapted to regression settings?

As mentioned in the global response, we refer the reviewer to our active learning-based baselines in Section 4.1. In this section, we adapt common active learning heuristics in classification settings such as uncertainty sampling (static), uncertainty sampling (sequential), and random sampling, to regression problems we focus on.

What is the size of the training dataset?

We direct the reviewer to Section D for the experimental details, particularly noting the training dataset sizes mentioned on lines 485-486 and 511-512.

in Line 270, what do mean by "We just consider a naive predictor, i.e., ψ(·) ≡ 0 for simplicity."

Recall our objective is to evaluate the performance of the model $\psi(.)$. For simplicity, we assume a naive predictor/model which outputs $0$ for all the inputs. We want to emphasize that this problem is just as complex as having a sophisticated model $\psi(.)$, as it merely serves as an input to our algorithm through objective $G(\mu)$.

We thank the reviewer for the thorough review and valuable feedback on our manuscript. Below, we address the specific questions raised by the reviewer. Additionally, please refer to the global response for the changes we intend to make in the camera-ready version based on all the reviews.

Line 102: Why do you mean "Batched settings"?

In a batched setting, we select a group of inputs to be labeled in each step, rather than just a single input. This approach is particularly important because, in practical scenarios, labels are typically requested in batches rather than individually. Additionally, it makes the problem more challenging by inducing a combinatorial action space. We discuss this in Section 1 (Lines 47-48) and will elaborate further on the problem setup in the camera-ready version.

Line 107: What are lookahead policies?

Lookahead policies are a class of policies for approximately solving the MDPs (Markov Decision Processes). These policies make decisions by explicitly optimizing over a shorter horizon of the MDP rather than the entire horizon. Depending on the length of this shorter horizon, they are referred to as 1-step, 2-step lookahead policies. We will include the following relevant references in the main paper.

[1] D. Bertsekas and J. Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996

[2] Y. Efroni, M. Ghavamzadeh, and S. Mannor. Online planning with lookahead policies. Advances in Neural Information Processing Systems, 2020

[3] Yonathan Efroni, Gal Dalal, Bruno Scherrer, and Shie Mannor. Multiple-step greedy policies in approximate and online reinforcement learning. In Advances in Neural Information Processing Systems, 2018.

Line 134: What's the purpose of rewriting the likelihood function $p(y|f, x)$ as $p_{\epsilon}(y - f(x))$? What does this likelihood function mean?

We want to highlight that the only source of randomness in the label $y$, given $f$ and $x$, is the noise $\epsilon$. Furthermore, this randomness depends solely on $y-f(x)$ and the distribution of the noise $\epsilon$.

Line 160: What's "$F_t$-measureable"?

$\mathcal{F}_t$-measureable states that the posterior state $\mu_t$ is known at time $t$. This is a formal way of stating that the posterior state depends only on the prior and data acquired up to time $t$.

Eq 2: Is uncertainty equal to Var? In the intro, you defined epistemic and aleatoric uncertainty, but what's the uncertainty you're trying to minimize in Eq2?

We aim to minimize the epistemic uncertainty, specifically by considering the variance of $g(f)$, where $f \sim \mu_T$. This variance represents the uncertainty due to a lack of knowledge about the underlying $f$, which is the epistemic uncertainty.

Line 168: What's the definition of A?

$A$ is a random variable distributed according to $\pi(\theta)$. It is introduced to highlight the difference between the REINFORCE estimator and the pathwise gradient estimator.

Line 179: Missing subject.

Thanks for pointing this out, we will address it in the main paper.

Line 180: What's the definition of $g(W_i, \theta)$?

A random variable $A$ distributed according to $\pi_\theta$ can always be expressed as $h(W, \theta)$ for some function $h(.)$, where $W$ is distributed independent of $\theta$. We use a smoother approximation $g(W, \theta)$ of $h(W, \theta)$ to enable differentiability with respect to $\theta$. For example, in Algorithm 2, we discuss how the standard Gumbel random variable $W$ can be used to generate a softer version $a(\theta)$ of K-subset samples.

Figure 4: Missing y-axis label.

Thank you for bringing this to our attention. We will add the y-axis labels in the paper. For your reference, we have mentioned the labels in the figure description.

Line 285: Providing intuition of empirical results before presenting the experimental results seems weird to me. I suggest stating the intuition after presenting the results.

Thank you for the suggestion. We will move the intuition after presenting results in Section 4.

Figure 4 and 5: What is the difference in terms of purpose between them? What should readers take away from the comparison on variance of mean squared loss and error of MSE on collected data? Line 314 refers to them but doesn't explain the details.

Figure 4 demonstrates that the variance of the model’s L2 loss is reduced, which is the objective of the algorithm. In Figure 5, besides comparing the variance of the L2 loss, we also compare the accuracy of the updated L2 loss estimation after acquiring new points. This illustrates that the objective is appropriate: by minimizing the variance, we also achieve a better evaluation of the model.

Figure 6 and 7: Same issues with Figure 4 and 5.

This has been addressed above.

There are many related works. Consider making it a separate section, as the current content is too dense.

Thanks for pointing this out. We will make it a separate section.