Rethinking Aleatoric and Epistemic Uncertainty

Thank you for your review.

We are pleased to see positive feedback on multiple aspects of the paper:

1. Strong motivation
2. Interesting use of decision theory
3. Comprehensive view on reasoning and learning
4. Clear writing, proofs and overall presentation

We also appreciate your thoughtful, in-depth conceptual comments along with useful pointers for lower-level improvements.

Loss functions

I am afraid the authors introduce another such (and equally questionable) dichotomy by fundamentally differentiating between objective model evaluation and subjective uncertainty quantification.

You raise an astute point that is more aligned with our perspective than you might think.

While we agree that a loss function is typically an imprecise abstraction of our internal desires for how a system should behave (and thus there is a subjective element in its construction), we argue that this falls under our problem definition because it relates to our definition of optimality in an ideal world, and not how that optimality is achieved. Core to our arguments is the idea that all uncertainty characterisation should start from the problem definition; our technical contributions are then based on how this can be done in a rigorous way. This results in a goal-driven notion of uncertainty that is rigorous given a problem definition (note that this setup is very standard across machine learning and not just the uncertainty-quantification literature).

Another key point is that even if we are unsure about how best to set up our loss function, this is not an uncertainty that arises from a lack of information. There is therefore no notion of changes in uncertainty from having access to more data or knowing the underlying data-generating process, which is ultimately what we wish to characterise in our decompositions. The choice of loss function is thus objective from a statistical-terminology perspective, even if it is a subjective decision (in the lay sense) in practice. We believe this use of terminology reflects a long history within decision theory, perhaps most prominently in the work of Savage (1971), who linked subjective beliefs to externally grounded evaluations through scoring rules.

Given the above, we do not feel it is problematic to condition our analysis on a real-valued loss, but we do agree that the importance and “subjectivity” of choosing the loss function warrants more discussion in the paper, and we will happily add this in our update.

Practical implications

The paper could benefit from some more constructive and practical recommendations for ML practitioners.

Great point. We have laid out some points in “Practical implications” in our response to Reviewer MLbH.

Case study

the paper's accessibility to the (applied) ML community could be improved by e.g. a more tangible case study

This is an excellent idea.

In our code repo (link below) we have added a new practical demonstration of some of the key ideas from the paper. We look at Gaussian-process regression and show how the loss function affects three things:

1. The model’s subjective uncertainty, $h_\ell[p_n(z)]$
2. The model’s discrepancy, $d(p_n, p_\mathrm{eval})$, with respect to a reference system
3. Which data gets prioritised during data acquisition

We also highlight the crucial distinction between model-based uncertainty quantification and externally grounded evaluation using a scoring rule or discrepancy function.

We do this by comparing a standard quadratic loss against a weighted quadratic loss that encodes a preference for accuracy on larger values of $z$ ($z$ might be a variable we need to be large, such as the solubility of a candidate drug molecule).

We plan to add some of these plots to the paper, along discussion relating it to practical applications.

Code

It appears the author did not share code

Thank you for flagging this oversight. We have made our code available at https://anonymous.4open.science/r/rethinking-aleatoric-epistemic-00DF.

Experimental evidence

Experimental evidence is only anecdotal and limited to BALD.

We recognise that our experiment is simple, but we contest the characterisation of our evidence as anecdotal. As explained in the paper, our experiment is carefully chosen to support our empirical claim, complementing results from past work.

Existing results

I also encourage the authors to clearly note some examples… as standard textbook derivations

Good point. We will flag existing results clearly.

Terminology

The paper occasionally uses different terms interchangeably without explicitly stating equivalences clearly.

This is useful feedback. We will standardise the terminology.

Related work

Thanks for these pointers. We will happily add citations.

Typos

We appreciate you pointing these out. We will fix them.

Thank you for your review.

We are happy to see you highlight a number of positive aspects of the paper:

1. Clear writing
2. Reasonable theory
3. Useful experimental results
4. Clear contextualisation within the literature
5. Refreshing style of contribution

We also recognise there are some things you think should be improved.

Clarity and space

Even though I find the whole work compelling and quite well formalized, I am inclined to say that the paper somewhat falls short in its goal to bring clarity. The paper is excellently written, but it is extremely dense and technical with a lot of notation, which is understandable and necessary, but it makes the paper quite difficult to read

We agree that the paper is necessarily idea-dense in order to communicate the full picture with appropriate nuance. Happily we think the paper’s clarity could quite easily be improved using the extra page allowed in the camera-ready paper, some careful rearrangement of the content, and extra efforts to highlight the key takeaways.

A key improvement we can make is to provide visual examples of the points we are making (see “Case study” in our response to Reviewer Atnu). Another thing we will try is presenting Examples A1-A3 and B1-B3 together: then the logic of the main technical content will be uninterrupted and the progression across the examples will be clearer. Finally we will make more use of the appendix to provide more verbose explanations of the trickier concepts covered in the paper.

Paper style

This paper is really different from standard papers… Personally I find this very refreshing, but it is problematic even only fairly review this paper into the new ICML review scheme…

It is great to hear that you find our contribution refreshing. We believe it could provide real value to the community, especially after we update for improved clarity based on your feedback. As noted above, we do feel there should be sufficient space to convey our message more clearly.

Imperfect estimators

I am not sure what to think of this type of issue that "... is only an estimator of [the true uncertainty quantity]".

You are right: estimation is what we have to do in practice. Two points are worth highlighting.

First, past work has often discussed common estimators as if they are the quantities they are in fact only approximating. We think emphasising the potential inaccuracy of the estimators has value in resolving misconceptions and supporting more clear-eyed use of common quantities.

Second, by establishing that these are only estimators, our work reveals the scope for alternative estimators that might be superior. For example, it might be clear that the infinite-step irreducible predictive uncertainty should be effectively zero, which a practitioner can use directly; Figure 2 in https://arxiv.org/abs/2404.17249 shows this can be better than using standard estimators. From a more theoretical perspective, an important implication of Propositions 1-4 is that the standard uncertainty estimators we consider are only optimal if we use a quadratic estimation loss, with other losses yielding different optimal estimators. Our exposition provides a template for deriving better estimators.

Unique uncertainty decomposition

I don't see what is wrong about a decomposition that still involves the expected value of those additional data points.

The key point here is that stochasticity in the data makes the decomposition depend on the data-generating process: any notion of irreducible and reducible uncertainty becomes conditional on it. This means we cannot talk about the decomposition into irreducible and reducible uncertainty: there are effectively endless possible decompositions because the data-generating process itself depends on design decisions we make (eg, how inputs are sampled or selected). Even if we fix a design policy (https://arxiv.org/abs/1604.08320), the corresponding true data-generating process is still unknown and so we are approximating the true expected decomposition with our model of the data-generating process. We will make updates to clarify this in the paper.

Another thing we would be happy to expand on in the paper (if it is of interest) is the existence of families of distinct data-generating processes that produce the same uncertainty decomposition given a long enough rollout of data generation. For example, it is possible for different experimental-design policies to be equivalent in the extent to which they will reduce uncertainty over a given rollout length.

Training-data entropy

line 149 left: Why would the Shannon entropy of the training data be n log 2?

We have $2^n$ possible outcome sequences, $y_{1:n}$, each with probability $1/2^n$. The entropy is

$$\mathrm{H}[p_\mathrm{train}(y_{1:n})] = -\sum_{i=1}^{2^n} p_\mathrm{train}(y_{1:n}^{(i)}) \log p_\mathrm{train}(y_{1:n}^{(i)}) = -\sum_{i=1}^{2^n} 2^{-n} \log 2^{-n} = n \log 2.$$

Thank you for your review.

Your critical feedback is much appreciated. We hope our responses and new demonstrative plots help alleviate your concerns.

Loss functions and uncertainty

"Given this, we argue that a principled notion of predictive uncertainty cannot be detached from this loss"… I remain unable to fully grasp the rationale and necessity for adopting this viewpoint as the foundation.

This is useful feedback—thanks. We will clarify our case.

Our core argument is that notions of uncertainty should be derived from the task we ultimately wish to perform, rather than being chosen in abstract. The loss is the measure of success on our final task, and our analysis shows that we can then directly derive rigorous notions of uncertainty from this loss using a framework of rational actions, namely minimising the expected loss.

The necessity of adopting this viewpoint as a foundation then manifests in a variety of ways, such as ensuring uncertainties are consistent with rational actions, having an end-goal-driven approach so that uncertainties reflect what we actually care about, and allowing data acquisition to be performed in a meaningful way that targets the problem of interest.

Practical implications

The practical applicability of this work remains ambiguous. What tangible benefits might arise…?

Thanks for highlighting this. We will happily add more discussion on the practical benefits. We also plan to add a case study that helps show the implications of our work in a practical setting (see "Case study" in our response to Reviewer Atnu).

One takeaway is to stop using off-the-shelf uncertainty measures and instead derive the one that will be most useful in a given decision problem (eg, data acquisition). We show how to do this.

Another is that we cannot think simply in terms of "Analyzing the sources of uncertainty" in the way the current literature does, with such decompositions themselves being highly subjective in practice. It is hard to characterise the application value of the work in the context of the aleatoric-epistemic viewpoint, as we are ultimately arguing that this is fundamentally flawed.

Other takeaways relate to estimation of uncertainty in practice. Part of this is promoting caution in using common estimators; part is encouraging alternatives. For example, standard estimators can be so inaccurate that we are better off directly using a numerical estimate based on our prior knowledge (see "Imperfect estimators" in our response to Reviewer 43ee).

Concrete algorithms and empirical evidence

If the authors aim to substantiate the superiority of their decision-theoretic perspective over conventional approaches, the development of concrete algorithms accompanied by comprehensive experimentation is indispensable

the experimental section of the paper is severely lacking, with insufficient empirical validation to support the claims

We feel that this misses the primary contribution of our work, which is not algorithmic but instead in showing the inconsistencies and flaws in the current foundations on which uncertainty-quantification algorithms are usually based, as well as providing a more rigorous and principled foundation. As noted by other reviewers, this makes it an unusual paper, but we do not believe every paper should be about proposing a new algorithm.

By extension, it is not clear what empirical experimentation it would actually make sense to add to the paper, beyond confirming the one empirical claim we make about how the BALD estimator should be understood. We do not agree that there claims being made that are lacking empirical validation, but if there are any experiments you think are missing we will do our best to add them.

To try and provide more clarity on how the work can be used for concrete algorithms, we have also added a new case study in our code base (see "Case study" in our response to Reviewer Atnu). This shows how using a non-standard loss can significantly impact how the uncertainty should be measured.

Title

This title lacks distinctiveness

We believe the broad span and foundational nature of our work actually requires a title like the one we use. Our paper brings together ideas from much of the ongoing discussions of aleatoric and epistemic uncertainty in the literature and aims to change the way people think about these concepts themselves.

Key concepts

Is the author intending for the concepts in this section to be treated as assumptions that must be satisfied, or as prerequisite knowledge that readers are expected to possess?

The concepts in the section are intended to be didactic rather than assumptions we are making. The section is also covering some of our problem formulation and synthesising relevant facts from past work, but its core is to introduce the key ideas that underpin our formulations.

Citations

We will revisit all of the citations you raise.

Thank you for your review.

We appreciate your positive feedback on a number of points:

1. Careful writing
2. Theoretical rigour
3. Convincing empirical evidence
4. Clear coverage of prior work
5. Potential benefit to the community

We are also grateful that you highlighted some ways to improve the paper.

Paper style

The paper's focus on conceptual discussion… could potentially position it as more of a position paper… However, the message it conveys is likely to be beneficial for the broader machine learning community.

We strongly support your emphasis on the benefit the paper could bring to the community, which we believe is ultimately what matters. While we agree that the paper would not have been totally out of place in ICML’s position-paper track, we ultimately felt that the objective and technically precise nature of its core contributions made it a better fit for the standard track, even if it does not fit the common mould within the field, as you note.

Future work

It might be valuable to consider including in the conclusion a brief exploration of any key open questions or potential research directions that this discussion brings to light.

Great idea. We will happily add this.

One exciting direction is deriving new problem-driven data-acquisition objectives using the decision-theoretic approach we demonstrate. In particular, we think our work lays foundations for developing loss-calibrated active-learning methods, which are underappreciated in the literature.

A key open question is how reducible uncertainties should best be estimated. Propositions 1-4 reveal existing estimators to be optimal only if we use a quadratic estimation loss, with further work required to establish optimal estimation strategies in other contexts.

Experimental setup

I would have preferred if more explanation of the setup was given in the main body, as I had to consult the appendix to be able to understand Figure 3.

This is useful feedback—thanks. We will happily revisit it.

Related paper

The paper's discussion of aleatoric and epistemic uncertainty perspectives shares some themes with this ICLR 2025 paper: https://arxiv.org/abs/2412.18808. In particular, Section C.2 of that paper appears relevant to the decision-theoretic perspective in Section 5.1 here.

Thanks for drawing our attention to this work. We agree that it is relevant and will cite it.

Wording of propositions and proofs

I felt the proofs and statements of propositions 2-5 were a bit too brief, sometimes making the discussion hard to follow. It would be useful to concretely restate the quantities referenced in the proposition statements in mathematical terms.

A number of the proofs (Propositions 2, 3, 5) use the phrase "...follows from the same working as in..." perhaps a more standard word choice would be "reasoning" or "argument" rather than "working"?

We appreciate the feedback. We will update the wording and mathematical statement of all propositions and proofs with a view to making things easier to follow and using standard language.

Expected information gain

Perhaps I missed it, is $EIG_{\theta}$ ever defined in the main body?

Thanks for flagging this. It is the expected information gain in $\theta$, where $\theta$ represents stochastic model parameters, which is the same thing as the BALD score we refer to throughout the paper. We will take care to clarify this.