ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation

Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:

W1. It is regrettable that component C0 was fixed (for each dataset) and not explored for at least another architecture. However, I might say that such a weakness is well within the realm of usual limitations expected in a paper.

• We appreciate your reflective feedback, and agree with you that ablating different segmentation backbones would be an interesting extension. However, as you agree, given the extent of the provided experiments, we had to restrict ablations at some point.

W2. The decisions pertaining to component C1 involve numerous hyperparameters, with some known to have a significant impact on performance. Consequently, this represents another weakness in the study, as it appears that these hyperparameters were not thoroughly investigated.

• Thank you for pointing this out. We are aware that selecting the right hyperparameters can influence the results for the different prediction models.
• Therefore, we state the possible impact of hyperparameters in Appendix C (“It is important to note that we did not do extensive hyperparameter searches for all the hyperparameters that we state here but rather used standard values if they worked reasonably and if not, we performed sweeps over the hyperparameters to find appropriate settings.”).
• However, in response to your feedback and to make the potential effects of hyperparamteres even more prominent in the main paper, we added a respective paragraph in our uploaded revision in Sec. 4.3., where we also provide direct links to respective hyperparameter specifications in Appendix C.

W3. Figure 2 uses different colours to distinguish distinct parts of the figure. This is not friendly for people with colour deficiency, and the authors could try to include patterns on top of the colour scheme.

• Thank you for making this important point. To make the barplots in Fig. 2 and Fig. 3 (formerly Fig. 2) more accessible, we added patterns on top of the bars that help to distinguish the categories as suggested.

Thank you once more for your feedback and for highlighting the importance of this kind of reflective work and its potential benefits for the community. Following up on your statement regarding the scoring, please let us know in case there are any remaining concerns that would hinder you from giving a higher score.

Thank you again for your time to review our work. There might be misconceptions regarding the scope of ICLR as well as the provision of experiments. We believe our reply below resolves the related concerns.

W1. The novelty is limited. The primary contribution seems to conduct additional experiments using existing methods.

• We think there is a misunderstanding about the scope of ICLR. We specifically submitted to the primary area “datasets and benchmarks”, a dedicated area where the novelty does not lie in new methods.
• The legitimacy is also evident in numerous ICLR papers sharing our format and type of contribution: E.g., Jaeger et al. ICLR23 (Top 5%), Galil et al. ICLR23a (Top 25%), Galil et al. ICLR 2023b, Wiles et al. ICLR 2022 (oral), Zong et al. ICLR23 (Top 25%).
• Motivation: In contrast, the exaggerated emphasis on methodological novelty is a driving factor behind the problems of current literature we identify in our work (see Sec. 3). The resulting inconsistencies call for a reflective and rigorous understanding of existing methods.
• Novelty: our study is the first to:
  • systematically reveal profound shortcomings in the practices of uncertainty estimation for segmentation. E.g. we identify 1) contradictions about the feasibility of separating uncertainty types, 2) a disconnection between theoretic work and concrete downstream applications, 3) and a widespread neglection of essential components.
  • present a framework that is rigorously designed to overcome the pitfalls by following concretely stated requirements for meaningful benchmarking.
  • conduct a large-scale empirical study that compares prevalent methods on an unprecedented spectrum of data sets, tasks, and uncertainty settings - generating novel insights.
• Concrete impact:
  • prior contradictions in literature are resolved, fostering a consistent picture, e.g. the ability of TTA to model epistemic uncertainty or the importance of score aggregation strategies.
  • methodological researchers are equipped with an easy-to-use framework to test their methods against potent baselines and under meaningful settings.
  • practitioners can access a knowledge base enabling informed decisions about the appropriate uncertainty method for their task.

W2. The experiments [..] in the main text is not persuasiveness, [...].

• To clarify, the main results of Sec. 4.5, where we benchmark methods on downstream tasks, are fully contained in Fig. 3 (formerly Fig. 2).
• Regarding Sec. 4.4 (“separation study” / “Q1-Q4”), please see the detailed response in our general reply above. In the revision, we provide all results in the main paper and clearly indicate where to find them.
• In general, all reviewers acknowledge our extensive experiments: “thoughtful experiments to study AU/EU”, “large amount of experimentation covering different aspects”, “this study represents the first attempt to systematically [..] evaluate these pitfalls across not only a toy dataset but also two real-world datasets”. This includes your comment: “The experimental results are pretty strong.”

W3. Please provide detailed analytically experimental or theoretical proofs

• Regarding experiments, we refer to our reply in W2.
• Regarding theoretical proofs, where necessary, we provide theoretical derivations (see Appendix D and E).

W4. It would be better to conduct more ablation studies using different backbones (e.g., VIT [..]).

• Thank you for pointing this out. We agree that studying backbones would be an interesting extension.
• However, given the extent of our work, we have to cut-off on ablations at some point. Or, as reviewer u7mE excellently phrases it: “I have to highlight that in this type of works it is always possible to think about more architectures, more datasets and more analysis, but [...] the authors chose a good scope for their analysis.”; “It is regrettable that component C0 was fixed [...] However, [..] such a weakness is well within the realm of usual limitations expected in a paper.”
• Further, We argue that ViT is no general new SotA architecture in segmentation that would in some way diminish our CNN-based results. Instead, CNNs are still competitive [a,b] and the clear SotA in many domains, e.g. [c].

[a] Wang et al. “InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions”
[b] Liu et al. “A ConvNet for the 2020s”
[c] Isensee et al. “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation”

Q1. What is the main innovation[..]? [..]

• Please see our response to W1.

Q2. Please [..] explain why this motivation is [..] important.

• Please see our response to W1.

Q3. For Q1-Q4 questions, please supply [..] analyses.

• Please see our response to W2.

Thank you for your constructive feedback. As we believe to have resolved your comments, please let us know if there are any remaining concerns that would hinder a recommendation for acceptance.

We thank the reviewer again for taking the time to review our paper. However, it is unfortunate that there was no acknowledgment of our response and, thus, no resulting discussion. This is especially noteworthy, as we believe to have resolved the concerns stated in this review, and the respective score of 3 is a clear outlier among all reviews.

Thank you again for your valuable comments and for taking the time to read our general reply, as well as considering our point-by-point comments here:

W1. I am not sharing the enthusiasm of the authors regarding the need to separate AU and EU. [..] Perhaps authors should provide a strong justification as to why one must care about this separation…

• Thank you for bringing up this important point. We think this is a misunderstanding because we totally agree with you: we are not enthusiastic about separation, in contrast, we take the disconnect between the prevalent enthusiasm and a lack of evidence as our motivation to challenge the separation by asking
  1. Is it feasible? We investigate this through our separation study and check whether the proposed methods actually perform the separation that they are stated to perform.
  2. Is it important? We systematically test on various downstream tasks whether isolating uncertainty types brings actual benefits.
• Since numerous papers, some of which are highly cited, are based on the assumptions that separation is feasible and beneficial without providing convincing evidence for either of the two, we hope you agree that scrutinizing the current enthusiasm in the field by means of this study is an important contribution.
• That said, in some downstream tasks like AL or OoD-detection, there is a theoretic case that isolating samples featuring epistemic uncertainty is preferable to a general predictive uncertainty containing a measure for ambiguities in the data.
• In Fig. 3 (formerly Fig. 2), in the column “uncertainty measure”, we provide empirical evidence indicating that for some tasks like OoD detection or ambiguity modeling, the isolation of specific uncertainty types seems beneficial, but for others, such as calibration, it is not.
• In response to your valuable feedback we now, 1) updated the introduction to highlight our objective stance and state a need to question the feasibility and importance of separation. 2) We updated the respective conclusion (Sec. 5 “R1”) to now state a clear demand for providing evidence about the feasibility and benefits of separation in future research.
• We hope our study and benchmark can help to foster more evidence-based research in this field in the future.

W2. I find the presentation style not optimal. Authors mostly provide summary results in the main text and the quantitative results in the appendix. Appendix should be there to support an otherwise self-contained main text. Here, I do not think the main text is self-contained.

• Thank you for this very helpful comment.
• To clarify, the main results of Sec. 4.5, where we benchmark prevalent methods on a wide variety of downstream tasks, are fully contained in Fig. 3 (formerly Fig. 2).
• Regarding Sec. 4.4 (“separation study”), please see the detailed response in our general reply above. In the revision, we provide all results in the main paper and clearly indicate where to find them, ensuring the self-containment of the main paper.

Q1. What is the aim for separating AU from EU?

• Please see W1 for a detailed answer.

Thank you once more for your constructive feedback. As we believe to have resolved your comments, please let us know if there are any remaining concerns that would hinder a recommendation for acceptance.

Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:

W1. The authors mention that “Overall, surpassing the ‘random’ AL baseline appears challenging [...]. However, [...] works have obtained significant improvements like in [1]. [...]

• We agree that Deep AL is improving and there are more advanced strategies that do not only rely on uncertainty, but also diversity [1, a] or different training schemes [1].
• However, the consensus is that outperforming the “random” baseline consistently is still challenging for both segmentation (“state-of-the-art active learning approaches often fail to outperform simple random sampling…” [b], [c Tab.6] shows that “random” performs best given small labeled datasets) and classification (see “In contrast to previous findings, well-regularized random baseline in our study was either at par with or marginally inferior to the sota AL methods.” [d], and sub-random performance of multiple AL methods in [e Fig. 2]).
• Similarly, in [1], there is no direct comparison to a “random” baseline with an identical starting budget and no standard deviations are given, making it hard to assess the performance improvement obtained with AL [1 Fig. 5].
• In response to your feedback, we added a paragraph with references supporting our statement.

[a] Sreenivasaiah et al. "Meal: Manifold embedding-based active learning."
[b] Mittal, et al. "Parting with illusions about deep active learning."
[c] Mittal et al. "Best Practices in Active Learning for Semantic Segmentation."
[d] Munjal et al. "Towards robust and reproducible active learning using neural networks."
[e] Lüth et al. "Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment."

W2. The authors mention that “The choice of aggregation method exhibits dataset-dependent variability” and “[...] yields mixed results [...]”. Is there any recommended guidelines [...], or, should users try all combinations?

• While we have some working hypothesis on the pros and cons of applied aggregation strategies, there is no clear signal in our empirical results, thus, given testing the limited number of potential strategies is a cheap post-hoc analysis, our recommendation for reliable performance is empirical testing.
• One advice for AL tasks could be to test the performance of aggregation strategies on proxy tasks before relying on a single strategy throughout the expensive AL training cycle.
• We did identify one “red flag”: using sum-aggregation when there is a potential correlation with the object size in the image. Thus, another recommendation is to reflect on such correlations when choosing an aggregation strategy.
• In response to your feedback, and to communicate these recommendations more prominently, we added a paragraph in Sec. 4.5.

Q1. It seems the authors have considered sampling-based methods [...]. How would the discussion be if the method is sampling-free like [2]?

• Since the method in [2] is proposed as an alternative to Monte Carlo-Dropout (MCD), and it is further compared to MCD showing similar behavior, we would assume that the behavior in our setting would be similar as well.
• Evidence for this hypothesis is also given in [2 Fig.5], where the structural similarity of the uncertainty maps between Bayesian SegNet and their sampling-free method is shown.
• It might also be interesting to discuss the performance of the uncertainty method in conjunction with the computational overhead, similarly to the cost-benefit tradeoff between ensembles and TTA discussed in our manuscript.

Q2. As the authors have considered SSNs, would the discussion for probabilistic (Prob.) methods like Prob. UNet [3] be similar or different?

• Both methods use random variables to model the spatially coherent distribution of outcomes given a sample (while the SSN does so by assuming a multivariate Gaussian over the distribution of logits, the Prob. U-Net utilizes a latent variable model during the forward pass). Therefore their behavior is expected to be similar.
• However, SSNs are a more lightweight alternative to the Prob. U-Net (no separate network to estimate the distribution, no variational inference training), rendering SSNs more applicable for users, which is why we preferred this method in our benchmark.
• In the SSN paper, the authors claim to outperform the Prob. U-Net in terms of GED. However, in theory, the Prob. U-Net is more expressive in its abilities to model ambiguity. This can be seen in [f Tab. 2] when both models are conditioned on modeling a specific style.

[f] Zepf et al. “That Label's Got Style: Handling Label Style Bias for Uncertain Image Segmentation”

Thank you once more for your constructive feedback. As we believe to have resolved your comments, please let us know in case you have remaining suggestions to further increase the quality of our work.