Escaping the Sample Trap: Fast and Accurate Epistemic Uncertainty Estimation with Pairwise-Distance Estimators

Thank you for the review.

Additional Base Learners:  

We do consider Probabilistic Network Ensembles as an additional base learner in the Appendix.

Writing Suggestions:  

Thank you, these have been implemented and are reflected in the new version of the paper.

Thank you for your thoughtful comments and the time you invested in reviewing our paper.

Thank you for the review.

Baselines:  

Random query is used as a baseline, as detailed in Table 1-5 and depicted in Figures 7, 8, 10, and 13. Our primary contribution lies in presenting a more precise estimator of epistemic uncertainty in higher dimensions. Consequently, we have adopted BALD (Houlsby et al. 2011) as a baseline in our experiments. BADGE (Ash et al., 2019) computes the gradient of the last layer outputs and employs k-means++ on these gradients to select data points for acquisition. In their original implementation, they exclusively focused on classification tasks with gradients of size 256x10. In our experiments, we face the challenge of computing gradients of size 256x257x2 and subsequently applying k-means++. However, k-means++ encounters difficulties in tasks with such elevated dimensionality. Moreover, BatchBALD necessitates the estimation of $p(y_1,...,y_n)$, a task feasible in classification but poses a barrier in high-dimensional regression output where $y_i$ represents a continuous output in a high-dimensional space.

Plot Test Accuracy:  

Please be aware that our work addresses regression tasks, in contrast to the majority of previous studies that primarily concentrate on classification as in your suggested baselines. Consequently, accuracy is not an applicable metric in our context. This distinction is emphasized throughout the paper, including in the Related Works section, where we state, “Our experiments encompassed tasks where the output spans continuous distributions for 1 to 257 dimensions, as opposed to the aforementioned methods that primarily address classification problems with a 1D categorical output.” For detailed results, we provide RMSE and Log Likelihood metrics in the Appendix.

Q1:  

Thank you for pointing this out, we have addressed this small typo in the text.

Q2:  

The function $f$ denotes a transition function: given a state and action, denoted as $s_t$ and $a_t$ respectively, it produces the subsequent state $s_{t+1}$. In our scenario, these functions are intricate and nonlinear due to our consideration of robot dynamics. In the paper, we explicitly note this as a dynamics model, stating, "the dynamics model for each environment was modeled, $f_{\theta} (s_t, a_t) = s_{t+1}$."

Q3:  

Monte Carlo (MC) estimators necessitate the drawing of a certain number of samples to provide accurate estimates; typically, fewer samples lead to poorer results. Figure 3 illustrates this phenomena within a specific environment. To enhance clarity, we have modified the text in the paper accordingly:  

To conduct a more in-depth analysis of our proposed method, we compared PaiDEs to MC estimators with a varying sample size $per$ $x_i$ in the Hopper-v2 environment. We expected that MC estimators would perform on par with PaiDEs with a sufficient number of samples. However, as illustrated in Figure 3,MC estimators fell short of achieving the same level of performance as PaiDEs in this particular scenario. This suggests that, taking into consideration hardware constraints, PaiDEs begin to outperform MC estimators when dealing with 11 dimensional outputs.

Thank you for your thoughtful comments and the time you invested in reviewing our paper. We have carefully considered your concerns and believe that our responses adequately address them. We kindly request you to consider increasing your score, as it would greatly enhance the likelihood of our paper being accepted.

Thank you for the review.

PaiDEs and Nflows Base:  

Please be aware that an introduction to Normalizing Flows is available in the appendix (A.5). To provide more clarity, we have modified the text in the paper as follows:  

Berry & Meger (2023) demonstrate that estimating Equation (2) in the base distribution is equivalent to estimating it in the output distribution. Consequently, by combining Nflows Base and PaiDEs, we construct an expressive non-parametric model capable of capturing intricate aleatoric uncertainty in the output distribution while efficiently estimating epistemic uncertainty in the base distribution. Unlike previously proposed methods, we are able to estimate epistemic uncertainty without taking a single sample. Figure 1 shows an example of the distributional pairs that need to be considered in order to estimate epistemic uncertainty for an Nflows Base model with 3 components

Nflows Base and Base Distribution:  

When fitting a normalizing flow the base distribution is needed to create the flow. There is an introduction to Normalizing flows in the appendix but for more details refer to Papamakarios et al. (2021). To address this confusion we have changed the text to:  

Nflows Base creates an ensemble in the base distribution,

$

p_{y|x,\theta}(y|x,\theta) &= f_{\theta_j}(y,x) = p_{b|x,\theta}(g_{\theta}^{-1}(y,x))|\det(J(g_{\theta}^{-1}(y,x)))|,

$

where $p_{b|x,\theta}(b|x,\theta)=N(\mu_{\theta,x}, \Sigma_{\theta,x})$, $\mu_{\theta,x}$ and $\Sigma_{\theta,x}$ denote the mean and covariance conditioned on both $x$ and $\theta$. These parameters are modeled using a neural network with fixed dropout masks to establish an ensemble and ensemble diversity is created by randomization and bootstrapping.

Thank you for your thoughtful comments and the time you invested in reviewing our paper. We have carefully considered your concerns and believe that our responses adequately address them. We kindly request you to consider increasing your score, as it would greatly enhance the likelihood of our paper being accepted.

Thank you for the review.

Baselines:  

Please be aware that BALD (Houlsby et al., 2011) selects inputs, $x$'s, that maximize the mutual information criterion as described in Equation (2). This framework is applied in all experiments of our paper as stated in the paper though to make this more clear we have added the following text:  

We select data points that maximize Equation (2) as in Bayesian Active Learning by Disagreement (BALD) to improve the model’s performance (Houlsby et al., 2011)  

Our proposal introduces a novel, more efficient method to estimate this criterion, particularly beneficial for high-dimensional outputs, while performing similarly in lower dimensions. It's important to note that Deterministic UQ Methods (ND-UQ) by van Amersfoort et al. (2020) are only applicable to classification tasks, which are not within the scope of our work. Deep Ensembles, such as those by Lakshminarayanan et al. (2017), are discussed in the appendix (A.4). MC Dropout has some challenges at estimating epistemic uncertainty for regression tasks (Berry and Meger 2023).

OOD:  

The performance in active learning suggests that the method can identify data points that are underrepresented in the training set, providing insight into how our approach may perform in detecting out-of-distribution (OOD) samples. There is a considerable overlap between epistemic uncertainty estimation and OOD detection, with many of our results demonstrating the method's proficiency in identifying novel inputs.

Limited Experiments:  

Two of the baselines, claimed missing, are actually included in the paper. One of them is inapplicable to our specific setting, and the last baseline demonstrates poor performance in our context. Please refer to the earlier response regarding the Baselines for more details. It's important to note that Mujoco simulates realistic and complex robot dynamics, with significantly higher dimensions compared to datasets in prior papers. Consequently, our experiments tackle a more challenging problem than those addressed in the existing literature, and this is the specific gap we aim to fill.

Theoretical Analysis:  

The bias of the proposed methods is examined in Figure 6 and 11 in the Appendix and mentioned in Section 7.4 of the paper. Furthermore, we offer a theoretical analysis illustrating when our proposed method may encounter challenges based on the number of components, as illustrated in Figure 4.

Scaling Behavior:  

Scaling behavior is investigated in Figure 4. The estimation of epistemic uncertainty would accurately scale with the number of components, though implementing the experiments you propose would demand a significant computational resource. These resources are typically only available to large technology companies.

Thank you for your thoughtful comments and the time you invested in reviewing our paper. We have carefully considered your concerns and believe that our responses adequately address them. We kindly request you to consider increasing your score, as it would greatly enhance the likelihood of our paper being accepted.