Segmenting the Unknown: Discrete Diffusion Models for Non-Deterministic Segmentation

We thank you for your comments.
We wanted to emphasize that, while one can see some connections between our work and uncertainty estimation, our work is not about uncertainty estimation. We introduce a concrete novel way to leverage discrete generative diffusion models for segmenting ambiguous contexts, and we assess their performance. We expand more on that point below.

Detailed Answers:

Given the lack of clarity on the type of uncertainty captured, can the authors specify whether it is aleatoric or epistemic uncertainty?

We consider aleatoric uncertainty, as we use experimental setups where the input has multiple possible valid segmentations.

The title suggests the capability to segment unknown classes. Can the authors clarify this claim?

No, we do not make such a claim. The ”unkwown” refers to the ambiguity present in the contexts we consider. In particular, when doing future prediction, the future state is unknown, yet we propose a way to segment it.

The paper discusses the potential for forcing diversity in the sampling process. Can the authors elaborate on possible methods to achieve this?

In our concluding remarks, we indeed identified encouraging diversity as a future research direction. A feasible initial approach would be to slightly penalize the previously predicted classes when generating the new segmentation. This serves as a baseline in our ongoing research.

Why is there a lack of comprehensive comparisons with existing uncertainty quantification methods in semantic segmentation?

Considering the importance of understanding and reporting both types of uncertainties, why was this aspect not heavily emphasized in the paper? Furthermore, can the authors shed light on how uncertainty is being evaluated and calibrated?

We will add a section in our literature review on the most related uncertainty estimation methods for segmentation.
However, uncertainty estimation is not the focus of our work. We do not evaluate it.
We present a concrete way to leverage discrete generative diffusion models for ambiguous contexts, and we assess whether these models follow the data distribution and have the capacity to generate different predictions. We have chosen our experimental setup to be contexts where there is true uncertainty so that multiple different predictions could be equally valid. Estimating the model uncertainty in these contexts would be a proxy for estimating the dataset uncertainty, but we believe it would not reveal more useful information about the model.

With the heavy reliance on existing ideas, can the authors provide further technical details or novel contributions that differentiate their approach from previous work?

Our work builds on existing ideas, but we propose an adaptation of these to segmentation, both as an image-to-segmentation model with LIDC and as an autoregressive segmentation-to-segmentation model with our two other experiments. We proposed these modifications and implemented and evaluated the use of discrete diffusion segmentation models in 3 diverse ambiguous setups to showcase their viability for this purpose. In fact, a very recent work [1] proposes a quite similar method, but applied only on LIDC, and was recently accepted at ICCV.

We also refer you to our answers to Reviewer 1 (qH1j) regarding discussions with some similar and related works.

[1] Stochastic Segmentation with Conditional Categorical Diffusion Models (https://arxiv.org/abs/2303.08888)

We thank you much for your comments and pointing to relevant literature we missed!
We are happy that you noted our diverse experimental setting and found it was well-motivated.
We took into account your general notes: we will better reference the interesting supplementary results and clarify the difference with Austin et al.
Additionnaly, we will include in the paper our discussion below regarding the Cityscapes results.

Detailed Answers:

Regarding the motivating example, the definition of the rectangles and the prediction of the deterministic model:

Differentiation of Rectangles: The rectangles within the images are not assigned fixed identifiers such as ”Rectangle 1” or ”Rectangle 2”. For each image, we create 2 rectangles, randomly assigned to one of 4 classes. However, the classes 1 and 2 have the same color in the input image, just as the classes 3 and 4. In practice, on the generation plots, the color mapping we use for each class is: {1: blue, 2: orange, 3: green, 4: red}. In the input image, the class 1 and 2 are in red while the class 3 and 4 are in green.
Categories Predicted by the Deterministic Model: The deterministic model was trained to predict categories based on the colors of the rectangles. However, due to the inherent randomness in our dataset, where each color can correspond to two different categories, the deterministic model tends to predict only one of the two possible classes, namely classes 1 and 4 as seen in Figure 2b.

Regarding the difference with [1]:

We were unaware of [1] and apologize for failing to cite the other relevant related papers. We will include these in the related works and the LIDC results table when relevant. The work in [1] is indeed very close to ours. They also use a discrete diffusion model, which they name categorical, and add image conditioning in a similar way. Our methods are very similar, however the two papers overall have multiple differences: (1) we additionally propose an autoregressive scheme for doing future prediction. In this case, our network is performing ‘segmentation to segmentation’, which means the input to the network is only one or multiple previous segmentation maps. (2) we provide a motivating example that clearly shows the sampling ability of the model and in particular the fact that it properly follows the data distribution. (3) we evaluate on different datasets: LIDC aside, we introduce a new car intersection simulation dataset designed to offer a finite set of valid future trajectories; we evaluate on the future prediction task on the Cityscapes Dataset.

Regarding the applicability of Chen et al. (2022a):

Their approach would also be applicable but is significantly different. First, their way of handling the discrete aspect of segmentation is different. In fact, they do not use discrete diffusion, they are instead mapping discrete classes to a continuous vector representation and then doing continuous diffusion. Also, note that their work focuses on panoptic segmentation, which might necessitate modifications to the network structure. Nonetheless, leveraging instance data from datasets where it exists could prove advantageous.

For each car in each validation example 10 samples are generated. The best of the 10 samples is selected according to FDE. Then the mean of the best FDE values (hit and miss) is computed. Is this correct? I am not sure the mean is very informative as the distribution seems to be quite skewed.

You are correct, we select the best FDE of the ten samples and then perform the mean over the validation set. The mean is informative, but you are correct in saying that a high miss can influence it strongly, which is why we also report the miss rate and consider it as our main ranking measure, and we will emphasize it more in the updated paper.

Regarding the Cityscapes experiment:

Thank you for this relevant question. We noticed that there is not as much ambiguity in the Cityscape's future segmentation task as one could expect. Indeed, given 3 past frames, the long-term future (1.5s) has little space for unexpected events, and we should note that Cityscapes does not feature a very chaotic environment, with disciplined German pedestrians. Therefore, a deterministic model can already do a very good job at predicting future segmentation, and the differences will mostly lie in slight speed variations, which will result in slight displacements of object boundaries. Multiple sampling allows to cover more of these possible speed variations, but do not generate completely different scenarios, as in the car intersection simulation. While we would ideally be able to answer a question such as ”will the child cross the road”, the nature of Cityscapes and the relatively short time delta of the prediction with the current frame does not allow us to showcase such things happening, but we hope our work is a step in that direction.

We thank you for your comments, and for pointing to uncertainty estimation literature.
We are pleased that you noted the simplicity of our solution and its ability to work in diverse settings, and that you found it interesting.
We wanted to emphasize that, while one can see some connections between our work and uncertainty estimation, our work is not about uncertainty estimation. We introduce a concrete novel way to leverage discrete generative diffusion models for ambiguous segmentation contexts, and we assess their performance. We expand more on that point below.

Detailed Answers:

The experimental section is limited. I expect to see the results of more baselines. For example, how the proposed method is compared with GAN-based methods?

We will add more related works on LIDC experiments. To our knowledge, no GAN-based method has been applied to this problem, and our work focuses on showing the ability of diffusion models to handle ambiguity. GANs are known to have mode collapse issues when learning distribution and can additionally be particularly hard to train. While it would be interesting to check how a particular GAN architecture would perform in our experimental setup, this is not the focus of this study and could be viewed as another contribution in itself.

There are lots of works regarding uncertainty estimation in (semantic) segmentation [1,2,3], just list a few. I would like to see the authors do a thorough review of these kinds of methods and provide a comprehensive comparison with the proposed method.

Thank you for the pointers to relevant literature! We will definitely add a section to our literature review of the most related uncertainty estimation methods for segmentation. We want to emphasize though that the focus of our work is not on uncertainty estimation, but rather on presenting a concrete way to leverage discrete generative diffusion models for ambiguous contexts, and on assessing whether these models are able to follow the data distribution.

Beyond the segmentation evaluation metrics, I am expecting the see more empirical results regarding uncertainty estimation. Some metrics can be found in [4,5].

As said above, we focus on the segmentation task and the capacity of the network to generate different predictions. We have chosen our experimental setup to be contexts where there is true uncertainty, so that multiple different predictions could be equally valid. Estimating the model uncertainty in these contexts would be a proxy for estimating the dataset uncertainty, but we believe it would not reveal more useful information about the model. However, any metric measuring the model uncertainty at the image level to assess the model confidence on a specific image could still be applied. In the case of the LIDC dataset, we could consider adding a correlation analysis of the model uncertainty per image with the ground-truth variations.

If the authors argue that this submission is about using the diffusion model for segmentation instead of uncertainty estimation, I would suggest the authors reorganize the paper and clarify the point. And also, if this is the case, the value of this submission to me is even more limited.