We thank the reviewer for their thoughtful comments and address them in detail below.

1. The difference between RPriorNet and VI, UCE cannot support the claim " the VI loss in Eq. (3) and UCE loss Eq. (4) turn out to be equivalent to the RPriorNet objective [...]"

We acknowledge that this sentence could be misleading, and we will revise it accordingly. What we want to convey is the resemblance in the in-distribution (ID) part of the objectives. Specifically, although VI loss and UCE loss were originally derived from different motivations, they are equivalent to the ID part of the VI objective in disguise. To our knowledge, this observation has not been made before, and we believe it provides a unified way to reason about a class of EDL methods in the literature.

2. EDL’s non-vanishing epistemic uncertainty is not surprising, and a data size dependent $\lambda$ can resolve this issue.

We agree that epistemic uncertainty would vanish with a choice of data size dependent $\lambda$’s. However, we would like to point out a few key considerations. First, the classical Bayesian approach does not require a data size-dependent prior. As long as the data size goes to infinity, the epistemic uncertainty will eventually vanish regardless of the choice of prior. Second, it remains unclear how to properly define the data size-dependent $\lambda$, in particular, its decay rate. Third, while it is true that one can always propose heuristic tricks to make the uncertainty of any algorithm align with the dictionary definition, this does not imply that such an algorithm is a faithful uncertainty quantifier.

3. EDL’s non-constant aleatoric uncertainty is expected, this holds for any uncertainty estimator.

Although it might be correct to say that aleatoric uncertainty is implicitly dependent on the model's capacity, randomness in training, data sampling, and other artifacts, what we are trying to argue is that, even under ideal scenarios (infinite sample size, overparameterized model), aleatoric uncertainty quantified by EDL methods can still be dependent on hyper-parameters ($\lambda$).

4. EDL’s non-vanishing epistemic uncertainty does not affect the practical use of this uncertainty in downstream tasks.

First, in this paper, we do not claim that EDL methods are ineffective for downstream tasks. Instead, we attempt to answer the following question: “Why and how do EDL methods work in practice?” As a first step, we characterize the global optimizer of several EDL methods and highlight several consequences, including the non-vanishing epistemic uncertainty. We believe that a clear understanding of the objectives used for "uncertainty quantification" is crucial to understanding how EDL methods actually behave, how to further improve these methods, and how to lead to a more reliable and grounded UQ approach.

Second, while we agree that non-vanishing epistemic uncertainty might not be directly related to common UQ downstream tasks such as OOD detection, there are practical applications that require accurate quantification of epistemic uncertainty. For example, in active learning, we cannot rely on the EDL model’s epistemic uncertainty as a signal to indicate the model’s actual knowledge at a certain stage because it never vanishes. The $\lambda$-dependent aleatoric uncertainty also may have issues in certain applications. For example, suppose an application relies on multiple pre-trained models to make predictions and identify ambiguous samples. If the learned aleatoric uncertainty of each model is trained with different $\lambda$'s in an EDL framework, then they might not be in a comparable scale.

5. L242 contradicts with L207.

In the paragraph, we consider the standard regime where $\nu$ is sufficiently large, so that $\alpha_0+\nu\eta(x)$ is approximately $\nu\eta(x)$, and the correspondence approximately holds in that regime.

6. The claim “the EDL methods can be better interpreted as an out-of-distribution detection algorithm based on energy-based-models”' is unsubstantiated, as multiple EDL methods do not require any form of OOD data.

We acknowledge that such resemblance applies to not all EDL methods, but to the methods covered by the unified objective (6). The main claim of this section is that the unified objective (6) closely resembles the EBM-based OOD detection objective. When no OOD data is available, the resemblance still remains, if we ignore the OOD regularization term. In that case, they simply promote very large outputs for ID data.

The empirical success of EDL methods that do not assume OOD data, such as PostNet, implies that OOD data may not be required for OOD detectors to distinguish OOD data for some reason. We conjecture that this success is an artifact due to a combination of some optimization phenomena and neural networks’ extrapolation ability. In fact, similar to the eleven out of the nineteen methods in Ulmer et al., there are also many effective OOD detection algorithms in the literature that do not require any OOD data during training. We acknowledge that this point was not made clear in the submission, and we will revise the section accordingly, especially in the case when no OOD data is available.

7. The claim about the benefit of incorporating model uncertainty is not backed theoretically.

Please refer to global response 1.

8. Missing standard evaluation metrics such as ECE.

Please refer to global response 2.(1) and Figure 13.

9. Missing aleatoric uncertainty benchmark.

Please refer to global response 2.(2) and Figure 14., 15.

10. Missing benchmark against a cross-entropy-trained deterministic network.

Please refer to global response 2.(3).

11. L180-181 are already discussed in the referenced survey of Ulmer et al. (2023; TMLR).

We acknowledge that lines 180-181 are also discussed in Ulmer et al., but these were not used to classify the EDL methods in Ulmer et al. We argue that the features mentioned in lines 180-181 can better classify the EDL methods with our unified view on the objective functions, and the dichotomy between “PriorNet-type methods” and “PostNet-type methods” in Ulmer et al. may not effectively contrast different EDL methods compared to our taxonomy. We will clarify our arguments accordingly in the revision.

12. The improvement upon ensemble distillation is marginal; The novelty of the use of bootstrapping is only in the context of EDL.

First, the detailed numbers with standard error are provided in Tables 2, 3, and 4 in the Appendix, which shows that the improvement is statistically significant. Second, we did not claim that bootstrap distillation is superior to ensemble distillation. Instead, we aim to advocate for distillation-based methods in general: ensemble distillation is the existing algorithm, and we propose bootstrap distillation as yet another alternative. Third, the main contribution of this work is to offer an analysis to better understand how EDL methods behave and provide insights on how to improve them. Proposing a novel EDL algorithm that achieves SOTA downstream task performance is beyond the scope of this work and could be considered a separate publication.

13. Question 1.

For RPriorNet, which utilizes OOD data for training, both the training loss and validation loss can fluctuate significantly due to the conflict between in-distribution and OOD data optimization. In this case, validation accuracy is a better signal to ensure the saved model is well-optimized. For other EDL methods, we rely on validation loss and never use any downstream task performance to choose the optimal model

14. Question 2.

We note that bootstrap distillation and ensemble distillation only differ in terms of the underlying stochastic algorithm, and the induced behavior depends on the choice of stochasticity.

15. Question 3.

First, as we mentioned in response 8, we conjecture that the success of EDL methods without OOD data is an artifact due to a combination of certain optimization phenomena and the extrapolation ability of neural networks. Second, the success is relative to other EDL methods, which actually suggests that other EDL methods relying on auxiliary techniques are not reliable for real data tasks (see Section 5.3).

If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.

Dear Authors, thank you for your rebuttal.

The difference between RPriorNet and VI, UCE cannot support the claim " the VI loss in Eq. (3) and UCE loss Eq. (4) turn out to be equivalent to the RPriorNet objective [...]"

I understand the aim to highlight the resemblance in the ID part of the objectives. However, the paper neglects the importance of including or excluding the OOD data from the training objective. This choice is a key characteristic of EDL methods, thus saying that the "VI loss [...] and UCE loss [...] turn out to be equivalent to the RPriorNet objective [...]" is misleading.

EDL’s non-vanishing epistemic uncertainty is not surprising, and a data size dependent λ can resolve this issue.

the classical Bayesian approach does not require a data size-dependent prior

Consider a Bayesian maximum a posteriori learning task using a Gaussian isotropic prior. Then $- \log p(\theta \mid \mathcal{D}) \equiv -\sum_{i=1}^n\log p(y_i \mid x_i, \theta) + \frac{\tau}{2}\Vert \theta \Vert_2^2$ where $\tau$ is the prior precision. Further dividing by $n$ results in an equivalent objective: this now matches Eq. (6) with the first term being an expectation and the second term being $\frac{\tau}{2n}\Vert \theta \Vert_2^2$, which depends on $n$.

As long as the data size goes to infinity, the epistemic uncertainty will eventually vanish regardless of the choice of prior.

The same holds for the EDL objective with a data-dependent $\lambda$. The cited challenges of scheduling the decay of $\lambda$ only affect the theoretical argumentation for the well-behavedness of the objective.

EDL’s non-vanishing epistemic uncertainty does not affect the practical use of this uncertainty in downstream tasks.

The paper does not sufficiently investigate the question “Why and how do EDL methods work in practice?”. Investigating the global minimizer of a unified objective is indeed useful, but the consequences the authors enumerate do not notably affect the practical usability of EDL methods. In particular, active learning also requires an effective ranking of inputs based on the importance of acquiring supervision on them. Shifting the uncertainty values by a constant does not affect this ranking.

The claim “the EDL methods can be better interpreted as an out-of-distribution detection algorithm based on energy-based-models” is unsubstantiated, as multiple EDL methods do not require any form of OOD data.

We acknowledge that such resemblance applies to not all EDL methods, but to the methods covered by the unified objective (6).

The resemblance applies only to EDL methods that leverage OOD data during learning. There is no such resemblance to EDL methods that only utilize ID data: "if we ignore the OOD regularization term", the learning dynamics become substantially different. Please refer to the first point of this reply.

The improvement upon ensemble distillation is marginal; The novelty of the use of bootstrapping is only in the context of EDL.

Consider including the error bars in the bar plots as well.

Additional experiments.

Thank you for adding the requested experiments. They clarify the behavior of different EDL methods and allow for easier comparability.

Conclusion.

The authors have addressed some of my concerns. In light of their clarifications, I am willing to increase my score from 4 to 5. However, due to the remaining points discussed above and the work's limited impact, I do not propose a higher score.

\newcommand{\piv}{\boldsymbol{\pi}}\newcommand{\av}{\boldsymbol{\alpha}}\newcommand{\bv}{\boldsymbol{\beta}}\newcommand{\E}{\mathbb{E}} We thank the reviewer for engaging in the discussion and sharing additional comments. We are glad that some of our concerns have been addressed, and we appreciate the reviewer for raising the score.

Here, we wish to clarify concerns in some of the additional comments as follows.

1. Regarding the resemblance between EDL method without OOD data and EBM-based OOD detection algorithm.

The reviewer is correct that EDL methods without OOD data are not exactly equivalent to the specific method proposed by [24], which explicitly assumes OOD data during training. However, we wish to clarify that we can still argue a close resemblance if we ignore the OOD regularization in both objectives as follows:

If we remove the OOD-data-dependent term in the EBM-based OOD detector’s objective, the objective becomes $-\E\_{p(x,y)}[\log {p_\phi(y|x)}] + \tau \E\_{p(x)}[\max(0,E_\phi(x)-m_{\textsf{id}})^2],$ where $E\_\phi(x)= -\log \boldsymbol{1}\_C^\intercal\bv\_\phi(x)$.

This still has a similar effect to the ID part of the unified objective in eqn. (6), which is in turn simplified in eqn. (8) as

\E\_{p(x)}[D(\mathsf{Dir}(\piv;\av\_\psi(x)) \~\|\|\~ \mathsf{Dir}(\piv; {\av\_0+\nu\boldsymbol{\eta}(x)}))].

That is, both objectives promote $\bv\_\phi(x)$ and $\av\_\psi(x)$ to output large values for ID data, while to be aligned with the underlying label distribution $\boldsymbol{\eta}(x)=(p(y|x))_{y=1,\ldots,C}$ when normalized, provided that $\nu \gg 1$.

We will clarify these points in the revision of the manuscript.

2. Data size-dependent prior in Bayesian.

The example provided by the reviewer, in fact, suggests that $\frac{1}{N}$ term in the Bayesian framework is naturally induced, rather than heuristically crafted. Here, the $\theta$ (model parameter) in Bayesian framework is a “global” random variable dependent on all the samples in dataset $\mathcal{D}$, when the model observes more samples, the prior belief (of model parameter) is naturally dominated by likelihood, and the posterior become more concentrated.

However, EDL methods do not have such a nice property. The reason is straightforward: In the EDL framework, $\pi$ is a “local” variable dependent on each sample $x$ instead of shared by all samples in the datasets, and its uncertainty is induced by its unique second order distribution $p_{\psi}(\piv|x)$ (parameterized by network $\psi$). To make the same analogy as the Bayesian example provided by the reviewer, we need multiple labels for each $x$, i.e., $\mathcal{D}\_x = \\{(x, y\_1), (x, y\_2), \ldots, (x, y\_{N\_x})\\}$, where $y_1, y_2, \dots, y_{N_x}$ are $N_x$ samples from the underlying first-order label distribution $p(y|x)$. Then, we could end up with a similar objective of the form -\log p(\piv|\mathcal{D}\_x) = - \sum\_{i=1}^{N} p(y\_i|\piv) + \lambda (\text{prior\\_term}).

In practice, however, for each $x$, we only observe a single label $y$, as $x$ is continuous and/or extremely high dimensional like images, and thus, in the EDL methods, the effect of prior does not naturally vanish as the sample size increases. A similar intuition has also been offered in some recent works, e.g., see the third paragraph of Section 3.3 in [4] and the second paragraph of Section 6 in [5].

• On the practical usage of EDL methods

We agree that active learning also requires an effective ranking of inputs based on the importance of acquiring supervision on them. Shifting the uncertainty values by a constant does not affect this ranking. We wish to remark, however, that beyond the query strategy (such as ranking), another important problem in active learning is to decide when to stop querying new labeled samples to minimize the labeling budget. It is unclear whether a heuristic trick such as "stopping data acquisition as long as the uncertainty becomes a constant" is reliable, because if the model queries bad samples in the previous iterations or the model is not well-trained, its uncertainty can also be roughly constant compared to previous iterations without improvement. Stopping the model training in such scenarios is not desirable. Uncertainty learned with a relative scale would render such a decision even more difficult.

Ideally, we believe that uncertainty accurately quantified in an absolute scale can better reflect how a machine learning system is truly lacking knowledge, especially for high-stake applications, such as medical diagnosis, and finance, where users need to rely on uncertainty measures to make important decisions.

References:

• [24] Energy-based Out-of-distribution Detection, Liu et al., NeurIPS 2020.
• [4] Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation, Bengs et al., NeurIPS 2022.
• [5] On Second-Order Scoring Rules for Epistemic Uncertainty Quantification, Bengs et al., ICML 2023.

We thank the reviewer for their thoughtful comments. We address them in detail below.

1. The central point of critiquing EDL is not novel.

While novelty is in the eye of the beholder, and critiquing EDL is indeed a primary focus of this work, our contribution is non-trivial compared to existing papers. First, we offer a more advanced understanding of EDL methods compared to existing critiques [4, 5, 6] (see Appendix A). Second, beyond the critique, this work is the first to attempt to remedy the limitations of EDL methods. Third, to the best of our knowledge, this work is also the first to unify multiple representative EDL methods into a single framework for theoretical analysis, supported by empirical justification with real data experiments.

2. Regarding the support and the source of the claim “distributional uncertainty is a kind of epistemic uncertainty”.

We respectfully point out that this is a common claim in the EDL literature. We provide a list of supporting evidences from the literature as follows:

• Malinin et al. (NeurIPS 2018) [7] proposed the "forward Prior Network". In this work, they introduced the concept of "distributional uncertainty," which is different from "model uncertainty" in Bayesian methods. However, similar to "model uncertainty," "distributional uncertainty" also measures the model's lack of knowledge, so it is aligned with the definition of epistemic uncertainty. They mentioned, “Distributional uncertainty is an ’unknown-unknown’ - the model is unfamiliar with the test data and thus cannot confidently make predictions.” The follow-up work of the same group of authors Malinin et al. (NeurIPS 2019) [8] also noted that “Knowledge Uncertainty, also known as epistemic uncertainty or distributional uncertainty, arises due to the model’s lack of understanding or knowledge about the input.”
• Malinin et al. (ICLR 2020) [9] proposed the distillation-based method “END^2”. Therein, the authors again remarked that “Knowledge uncertainty, also known as epistemic uncertainty, or distributional uncertainty, is uncertainty due to a lack of understanding or knowledge on the part of the model regarding the current input for which the model is making a prediction.”
• Charpentier et al. (NeurIPS 2020) [15], where the“Posterior Network” was proposed, uses "epistemic uncertainty" to refer to the "distributional uncertainty." The same usage can be found in its follow-up work (Charpentier et al., ICLR 2022) [20], in which “Natural Posterior Network” was proposed. There is no mention of “distributional uncertainty” in both works.
• Recently, several critiques of the EDL methods such as Bengs et al. (NeurIPS 2022) [4], Bengs et al. (ICML 2022) [5], Jürgens et al. (ICML 2024) [6] have adopted the same convention. In these papers, they all use “epistemic uncertainty” as the terminology to refer to EDL's “distributional uncertainty” and criticize that EDL cannot faithfully quantify epistemic uncertainty.

We will include these sources in the manuscript to avoid any confusion.

3. Regarding paper presentation and clarity.

We acknowledge the content is a bit cluttered in the submission due to the page limit. In the revised manuscript, we will do our best to improve clarity and elaborate on the details with the additional page.

4. Lacking theoretical justification and further investigation of proposed Bootstrap distillation.

Please refer to global response 1.

5. Would the authors consider introducing the bootstrap-distillation method as a separate publication?

The main point of this work is to advocate for distillation-based EDL methods in general, and the proposed bootstrap-distillation method serves as another alternative in the ensemble approach to justify this argument. We agree with the reviewer that conducting a comprehensive analysis (both theoretical and empirical) of distillation-based EDL methods can be an interesting future work.

6. What practical problems will $\lambda$ dependent aleatoric uncertainty cause in downstream tasks?

It is correct that identifying difficult, ambiguous, or mislabelled samples is still possible in practice, as long as the downstream task only relies on a relative scale of uncertainty values. However, there might be practical scenarios where the absolute value of uncertainty also matters. For example, suppose an application relies on multiple pre-trained models to make predictions and identify ambiguous samples. If the learned aleatoric uncertainty of each model is trained with different $\lambda$'s in an EDL framework, then they might not be in a comparable scale, which prohibits any further comparison.

7. Missing the behavior of a Bayesian approach in Figures 1.a, 4.a and 10.

Please refer to global response 2.4.

We thank the reviewer for their thoughtful comments. We address them in detail below.

1. Is the proposed solution theoretically guaranteed to faithfully express the epistemic/aleatoric uncertainties?

Please refer to global response 1.

2. It is unclear if there is any benefit of the $p(\psi|D)$-based EDL framework over Bayesian approaches.

The main benefit of distillation-based EDL is its computational efficiency at inference time. Once the model is trained, it does not require multiple inferences like Bayesian or ensemble methods.

3. The proposed solution (bootstrap-Distil)'s aleatoric/epistemic uncertainty behavior is not well justified.

The bootstrap-distill method does not have any hyper-parameters in its objective, so the aleatoric uncertainty is not model/hyper-parameter dependent. Regarding the behavior of epistemic uncertainty, we provide the figure of the epistemic uncertainty vs. sample size curve on the CIFAR10 dataset in Figure 10. The behavior of uncertainty quantified by bootstrap, ensemble, and Bayesian approaches is similar, as the epistemic uncertainty is induced by model uncertainty. All these methods are well-studied in literature, and they are usually considered as faithful uncertainty quantifiers.

4. Regarding the range of the aleatoric/epistemic uncertainty values. Are the uncertainty values relative, or is there any meaning to the values?

The range of aleatoric/epistemic uncertainty depends on the uncertainty metric. For example, epistemic uncertainty measured using "mutual information" will be non-negative. For typical downstream tasks such as OOD data detection, only the relative uncertainty matters for identifying OOD samples. However, there might be practical scenarios where the absolute value of uncertainty also matters. For example, in active learning, we cannot rely on the EDL model’s epistemic uncertainty as a signal to indicate the model’s actual knowledge at a certain stage because it never vanishes. The $\lambda$-dependent aleatoric uncertainty also may have issues in certain applications. For example, suppose an application relies on multiple pre-trained models to make predictions and identify ambiguous samples. If the learned aleatoric uncertainty of each model is trained with different $\lambda$'s in an EDL framework, then they might not be in a comparable scale.

5. Missing Comparison with SOTA OOD detection methods.

First, the focus of this work is NOT to propose a novel algorithm that outperforms all SOTA OOD detectors. Instead, the main contribution is to offer an analysis to better understand how a specific type of UQ method (EDL) behaves and to provide insights on how to mitigate its limitations. The bootstrap-distill approach serves as one of many possible solutions to address the issues of existing EDL methods. Furthermore, while comparing EDL methods with algorithms specifically designed for OOD detection might not be fair, we include an additional baseline comparison with a classical cross-entropy-trained network (see global response 2.3).

6. how the OOD detection performance works for EDL methods with $\lambda=0$?

We experimented with the $\lambda=0$ case, but observed that this simply encourages the model to output an extremely large value and leads to numerical errors during training. This is consistent with what Theorem 5.1 and Example 5.2 revealed, i.e., $\lambda=0$ corresponds to that the target for ID data is $\infty$.

7. What’s the benefits of three levels of uncertainty approach over Bayesian approaches?

First, our main claim in this paper is that distributional uncertainty alone while ignoring model uncertainty cannot capture uncertainty in a faithful manner. Second, we suggest that distillation-based methods could be a straightforward solution to remedy these issues. While distillation-based EDL methods have less computational complexity than Bayesian methods, we do not claim that distillation-based EDL methods are fundamentally superior, and further studies are warranted to compare the two approaches.

8. if the proposed method will be robust to learning deficiencies issue in EDL literature [1, 2]?

The learning deficiencies issue is mainly related to the activation function in the EDL model architecture, which might be a tangential problem to this work.

9. How does the aleatoric uncertainty behave with an increase in sample size?

Aleatoric uncertainty does not have asymptotic behavior. In practice, aleatoric uncertainty will converge to a constant when the model is trained with a sufficient number of samples.

I thank the authors for the rebuttal response. However many of my concerns remain

2. It is unclear if there is any benefit of the bootstrap-based EDL framework over Bayesian approaches.

Clarification regarding faster inference compared to ensembles

My current understanding is that M different models would need to be trained on subset of training dataset to obtain the distribution p(\psi|D). I believe that the training is significantly more expensive than EDL approaches, and as expensive as ensemble of M models. I feel as if the inference would be as expensive as ensemble of M models. I wonder if my understanding is correct?

3. The proposed solution (bootstrap-Distil)'s aleatoric/epistemic uncertainty behavior is not well justified.

a) The claim: The bootstrap-distill method does not have any hyper-parameters in its objective --> Wouldn't the bootstrap-Distil method introduce the hyperparameter: number of bootstrap samples?

b) The clarifying question is: will it's uncertainties show reasonable trends? For eg. what is the trend of aleatoric/epistemic ucnertainty trend of bootstrap-Distil for 2D gaussian? I think for a model claiming fine-grained uncertainty quantification capabilities, both epistemic and aleatoric uncertainties should show meaningful trends. I may have missed but currently I'm not confident on the superiority of bootstrap-Distil's uncertainty (the claim: the aleatoric uncertainty is not model/hyper-parameter dependent, --> what is the uncertainty trend?)

9. How does the aleatoric uncertainty behave with an increase in sample size?

Theoretically, aleatoric uncertainty should converge to a constant when the model is trained with a sufficient number of samples. The clarifying question was regarding how the EDL and the proposed model's aleatoric uncertainty behaves with change in number of samples. I believe observing aleatoric uncertainty for the EDL model/proposed model across the experiments/datasets should be straightforward and reveal insights into model's uncertainty behavior.

Unanswered claims:

Is the aleatoric/epistemic uncertainty trend of this model accurate on all datasets and settings or only accurate on simple dataset of Cifar10 (the epistemic uncertainty is shown only for one experiment of Cifar10) It is still not clear to me whether the proposed model's uncertainty be reasonable in realistic practical datasets (eg. imagenet/tiny-ImageNet/Cifar100)

We thank the reviewer for their thoughtful comments. We address them in detail below.

1. Whether EDL’s behavior demonstrated in the paper would hold in other modalities and domains.

We totally agree with the reviewer that analyzing EDL’s behavior in other domains and modalities is worth exploring. Since the classification setting for the vision domain is standard in the UQ literature, however, we focus on this setup to clearly demonstrate the implication of our analyses. While we leave such extensions as future work, we refer the reviewer to Appendix F.2, where we provide some analysis for the regression case.

2. Incorporating more recent models and state-of-the-art architectures for evaluation.

This is also a good suggestion. In fact, most EDL methods evaluate their performance on classical neural networks and standard benchmark datasets. Conducting a comprehensive empirical study to benchmark all existing EDL methods under a unified framework would also be an interesting future work.

3. How do the proposed results translate to the regression case? The EBM intuition of OOD wouldn't work for the regression case.

As the reviewer noted, the EBM intuition would not hold for the regression case. However, a similar observation can be made: the existing EDL methods for regression only aim to fit to a “fixed” uncertainty target. This is consistent with the observation made by a recent paper (Meinert et al., 2023). We kindly refer the reviewer to Appendix F.2, where we elaborate further on this case.

4. If stochasticity is a solution for EDL drawbacks, could we apply efficient ensembling alternatives to achieve the same results?

In this work, we argue that the EDL framework can be better used as a computational tool to emulate the behavior of classical ensemble approaches, while reducing their computational burden. In this context, using more advanced ensembling techniques can indeed boost the UQ performance of distillation-based EDL methods. We will add the relevant discussion in the revision.