Pancakes: Consistent Multi-Protocol Image Segmentation Across Biomedical Domains

We thank the reviewer for their thorough and useful feedback. We are glad they found the paper interesting, and appreciate the number of tasks we train and evaluate on. Below, we address the reviewer’s questions on the use case, the protocols M and candidates K, the choice of baselines, and the metric used.

Use case clarification. How can we utilize the segmentation maps in the downstream or clinical tasks?

We will clarify that Pancakes is not intended to replace existing clinically validated segmentation protocols. If there is an existing tool for a particular protocol, we would advise using it. Rather, our goal is to support a new capability: enabling biomedical researchers and clinicians to explore a diverse set of plausible, semantically consistent segmentations for an unseen collection of images. After Pancakes has generated the label maps, a researcher or clinician selects which of the proposed protocols best aligns with their intended downstream use (e.g., anatomical volume analysis). We envision two broad classes of use:

(1) Rapidly segmenting new protocols. New protocols are frequently introduced [133, 136, 137], and there is a need to produce corresponding segmentations. If a scientist has a particular protocol in mind, but there are no existing tools for segmenting it, they can choose the protocol from Pancakes that best aligns with their intended use. This choice will often suffice. In cases where Pancakes’ segmentations are not sufficiently accurate–for example, when targets are more out-of-distribution than those seen in training–the segmentation maps can offer excellent initializations to interactive segmentation systems such as ScribblePrompt [120]. To demonstrate this, we performed a preliminary experiment and found that:

• On average, Pancakes’ predictions can be improved by 5 Dice points with a single interactive click, for set sizes larger than one.

• On average, using ScribblePrompt with a Pancakes-initialized segmentation reduces the number of required interactions by half. If used alone, it takes ScribblePrompt 5 to 8 clicks to reach optimal quality predictions, while, when initialized with Pancakes’ predictions, ScribblePrompt can reach them in 3 to 4 clicks.

Since figures are not permitted in the rebuttal, we cannot provide plots for this experiment, but will include them in the manuscript.

(2) Exploratory population analysis. Pancakes will support users in discovering or selecting segmentation strategies appropriate for their scientific or clinical questions. For example, a clinical scientist who studies how anatomy relates to some outcome (e.g., progression of a disease) or predictor (e.g., genetics) can use Pancakes to quickly extract segmentations of multiple candidate anatomical regions that have never been segmented, compute their volumes, and test correlations with clinical outcomes thereby identifying promising candidate regions.

Is this assessment clinically verified by the corresponding specialists? Who should confirm the clinical meaning of these protocols?

Pancakes is intended for biomedical experts. We assume that as they use the tool, they can identify the mapping between a label and a known structure. To evaluate the accuracy of our predictions, we used expert-provided segmentations for the held-out datasets (unseen at training time). We did not conduct a user study in which clinical experts evaluated the distribution of protocols. Such a study would be useful, but presents design challenges and would require a substantial investment of time by independent experts.

Did you split the dataset based on images or based on patients?

We split the dataset based on patients, and ensured that there is no train/val/test cross-contamination. We will clarify this in the paper.

Surprisingly, the original UNet can perform better than other baseline approaches.

We are unsure of what the reviewer means by Dice performance Table 1, as Table 1 relates to speed at inference instead of performance on Set Dice score. We will, therefore, address both:

• High UNet performance: There is strong evidence that UNet-based architectures match transformer-based architectures in terms of performance for medical image segmentation [134, 135]. We chose to demonstrate that Pancakes works with even a simple solution, which enables fast throughput and a low number of parameters helps limit overfitting, and especially task overfitting, where the model remembers the structures to segment instead of proposing diverse segmentations. We also highlight that our proposed framework is not dependent on the choice of the model $f_{\theta_f}(.)$ that infers the distribution parameters $\phi$. The UNet implementation of $f_{\theta_f}$ could be replaced with another network, such as the SAM featurizer. We will discuss this in the revised manuscript.
• High Set Dice: Since our method tackles a new task, other models are not designed to perform well on the aspects captured by these metrics (accuracy and consistency). We find that when just looking at accuracy, Pancakes is comparable to SAM and outperforms the other baselines. However, SAM cannot provide semantically consistent segmentations, which leads to substantial performance degradation when considering several images jointly.
• Inference Speed: Unlike SAM-type models in automatic mode, Pancakes remains unburdened from transformer-based architecture and heavy post-processing scripts. It can run inferences for M protocols (of K labels max) in a single forward path, which guarantees much faster inference than our baselines.

Intuition behind Set Dice

Most traditional metrics fail to capture what we care about: simultaneous accuracy and consistency. Given a known label, we want to identify if it is present in our protocols consistently. For example, if we identify that label (m=3, k=5) corresponds to a given structure in one image, we want to ensure that the label (m=3, k=5) corresponds to the same structure in other images from the same dataset. The proposed Set Dice captures this effect by selecting the same label map for all images in the set and taking the best performing label averaged across the set. This metric choice closely approximates real-world use-cases where the user selects a preferred label consistently for all images. We also highlight that Best-Candidate Dice metric–which assumes that a user identifies the most accurate label–is frequent in the medical image literature, especially when labels are uncertain [9, 57, 83].

Additional metrics

• Note that the value for set size S=1 in Figure 8 corresponds to the traditional Dice score metric.

• Following the reviewer's feedback, we included additional metrics. In addition to Set Dice, we now include Set Intersection over Union (IoU) and set surface distance (percentile 95%),

Experimental results. We find that for set surface distance, Pancakes outperforms all the baselines. For set IoU, Pancakes performs similarly to SAM for S=1 (looking at a single image, equivalent to the traditional IoU metric) and outperforms the baselines in all the other cases.

[133]: Merdietio Boedi, Rizky, et al. "3D segmentation of dental crown for volumetric age estimation with CBCT imaging." International Journal of Legal Medicine 137.1 (2023): 123-130.

[134]: Isensee, F. et al. (2024). nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15009. Springer, Cham.

[135]: Carsen Stringer and Marius Pachitariu. Transformers do not outperform cellpose. bioRxiv, pp. 2024–04, 2024

[136]: TopBrain 2025 Grand Challenge

[137]: Greve, Douglas N., et al. "A deep learning toolbox for automatic segmentation of subcortical limbic structures from MRI images." Neuroimage 244 (2021): 118610.

We thank the reviewer for their thorough and useful feedback. We are glad they found the task novel and appreciated the strong performance and efficiency of our model, and found the paper clear. We carefully address the remaining questions on the protocols and the choice of baselines.

Could there be a mechanism (e.g., visual summaries, textual prompts, or clustering) to help users choose the most relevant protocol?

Pancakes produces multiple label maps that are consistent for a set of images to segment. Currently, if the user cares about a specific region, they can choose the most relevant protocol by clicking on that corresponding label map prediction for one of the images. That protocol gets used for all the images in the set.

User Interface 1: During the rebuttal phase, we built a simple user interface that contains 2 slider bars (one for M and one for K). As the user varies them, the segmentations proposed by Pancakes can adjust in nearly real time (enabled by our lightweight architecture) and propose consistent protocols for all the elements in the set.

User Interface 2 -- Going further: If the user is interested in a specific region, we have designed an alternative UI where the user can specify a region they are particularly interested in by clicking, and only the labels for this specific region will be displayed.

The rebuttal guidelines prohibit links, but we will make a Jupyter Notebook notebook supporting this available with the final paper to demonstrate the ease of use.

While we have not explored multimodal prompts, we agree that including visual summaries and text prompts is an interesting opportunity for future research.

How can users directly specify or guide the type of segmentation they want

Beyond the strategies mentioned above, choosing fewer labels per map (lower K) will tend to produce coarser labels. Figure 6, Figure 7 and Figure 10 (in the supplemental material) provide an intuition of how the label maps predicted change with M and K. During the rebuttal period, we built a simple user interface to demonstrate this effect, containing 2 sliders (for M and K). As the user adjusts them, the segmentations proposed by Pancakes adjusts in near real time (possible due to our lightweight architecture). Once M and K values Pancakes segments consistent segmentations on all images in the set. The desired protocol m or label (m, k) can then be selected through a simple click for a single image. While the NeurIPS change in rebuttal policy to disallow pdfs prohibits us from showing the interface, we will show it in the final manuscript.

Has a clinical expert evaluated the protocols?

To evaluate the accuracy of our predictions, we used expert-provided segmentations for the held-out datasets (unseen at training time). We did not conduct a user study in which clinical experts evaluated the distribution of protocols. Such a study would be useful, but presents design challenges and would require a substantial investment of time by independent experts.

How many of the protocols are useful?

This is an excellent question. To simulate this, we evaluate Pancakes on a subset of Total Segmentator, a held-out dataset that contains segmentations for many structures, with maximum 8 protocols (i.e., M = 8 ). We find that for each set, all 8 protocols would have some label maps that serve as the best candidate for some ground truth labels, meaning all protocols are used if we consider a holistic collection of possible target labels.

We thank the reviewer for their thorough and useful feedback. We are glad they found the task novel, and appreciated our training and evaluation framework and Pancakes’ strong performance. We are also pleased that they found the paper well-motivated and clear. Below, we address the reviewers questions about the protocols M and candidates K, the choice of baselines, and the metric used.

Limitations of traditional metrics.

A distinguishing attribute of our metric is that it takes into account the consistency of the labels produced across images of the same set. This aspect is not taken into account by other metrics. This means that the same structure (for example, a ventricle) could be classified satisfactorily, but be assigned different labels. Because structures are very rarely aligned, identifying the matching labels across different images is a challenge that is ignored with the common metrics. We will clarify this point in the paper.

Additional quantitative metrics.

• Note that the value for set size S=1 in Figure 8 corresponds to the traditional Dice score metric.

• As the reviewer suggests, we included additional metrics. In addition to Set Dice, we now include Set Intersection over Union (IoU) and Set Surface Distance (percentile 95%),

Results

We find that for Set Surface Distance, Pancakes outperforms all the baselines. For Set IoU, Pancakes performs similarly to SAM for S=1 (looking at a single image, equivalent to the traditional IoU metric) and outperforms the baselines in all the other cases.

Qualitative visualizations for the baselines.

Since figures are not permitted in the rebuttal, we cannot provide additional qualitative visualizations. Figure 2 shows examples of the segmentations produced by the baselines (left to right, SAM with 2 different hyperparameter choices and UnSAM). The labels produced are plausible but unlike the Pancakes labels, these labels are not consistent across images. Moreover, they are not organized, many of them overlap or some pixels are left without a label assignment. These findings are consistent across datasets and we will include them in the manuscript as well as the corresponding visualizations.

In Figure 2, what are the three methods?

These are all automatic segmentation foundation models. Left to right, we include SAM for different hyperparameter values in automatic mode and unSAM in automatic mode. In our evaluation, we select different hyperparameter values in an attempt to collect as many valid masks as possible.

Impact of the synthetic data.

We will add an ablation study comparing training Pancakes with real data only, synthetic data only, or both. When trained with both real and synthetic data, Pancakes consistently outperforms the other variants in Dice score, (p<0.05, paired Student-t test), for M=16:

Why is SAM a consistent second while not trained on medical data?

This is an excellent question, for which we don’t have a definitive answer though similar behavior has been observed in the literature[120]. We hypothesize that SAM is better because it was trained on a wider variety of labels and images. Therefore, SAM is less prone to task-overfitting compared to other models. SAM might also be more adapted to the type of interaction used to produce the predictions: clicks. We will make this point in the paper.

How were M and K selected for the main results?

We will clarify our choices of M=8 (number of maps) and K=20 (number of candidate labels) in the main paper. As Appendix Figure 15 shows, performance on the development datasets saturates at M=8 and K=20, with only marginal gains achieved by further raising M. We therefore chose to use these parameters as a balance between performance and the number of structures clinical users might actually expect in practice. Importantly, these parameters were chosen independently of the held-out test datasets in Figure 12.

How does specifying M and K compare to providing a hierarchy of protocols to existing multi-protocol segmentation models?

Specifying M and K is faster than defining a hierarchy of protocols. To demonstrate this, we have built a simple user interface that contains 2 slider bars (one for M and one for K). As the user varies them, the segmentations proposed by Pancakes can adjust in nearly real time (enabled by our lightweight architecture) and propose consistent protocols for all the elements in the set. A protocol can be selected through simple click. Since figures are not permitted in the rebuttal, we cannot share a figure showing this interface, but we will release it with the paper. This alleviates the need to manually annotate many labels and define a hierarchy a priori, especially when there is no clear hierarchy between labels.

Clarification on comparison to other universal and multi-protocol segmentation methods

We compare our model to [45] and [98] in automatic segmentation mode. Although we cite as much relevant work as possible, most are not directly applicable to the new problem setting that we tackle. For example, baselines that are data-specific or that require an explicit hierarchy are not applicable because of their limitations or heavy annotation requirements. As is often the case in medical imaging, we assume that only binary labels are available in our data. Similarly, baselines that require multiple interactions (clicks, bounding boxes, scribbles) for each image to be segmented are not applicable to our problem because our goal is to alleviate the burden on the user. We will clarify this choice of baselines in the revised manuscript.

Changing Bar plot to Table

Thank you for pointing this out, we will include a table for a clearer distinction between the different methods.

We thank the reviewer for their thorough and useful feedback. We are glad they found the task novel, and appreciated our training and evaluation framework and Pancake’s strong performance. We are also pleased that they found the paper interesting and clear. Below, we address the remaining questions on the protocol use, method applications, baseline selection, and the evaluation setting.

Could labels leak across protocols?

The primary objective of Pancakes is to propose a diverse set of consistent protocols for an image group. We do not enforce constraints across protocols produced, but we agree that at times it would be good to do so. For example, symmetric labels (left and right ventricles or posterior and anterior hippocampus) in the same protocol may be desirable. One way to incorporate this into the Pancakes framework is to apply the loss function to two randomly chosen segmentation targets in a multi-label dataset, rather than one. To validate this, we ran an experiment on the OASIS brain dataset (2 protocols with 5 and 25 labels respectively) during the rebuttal period, and found it effective. We thank the reviewer for raising this important point and will include the method and results in the revised manuscript.

Use case clarification.

We will clarify that Pancakes is not intended to replace existing clinically validated segmentation protocols. If there is an existing tool for a particular protocol, we would advise using it. Rather, our goal is to support a new capability: enabling biomedical researchers and clinicians to explore a diverse set of plausible, semantically consistent segmentations for an unseen collection of images. After Pancakes has generated the label maps, a researcher or clinician selects which of the proposed protocols best aligns with their intended downstream use (e.g., anatomical volume analysis). We envision two broad classes of use:

(1) Rapidly segmenting new protocols. New protocols are frequently introduced [133, 136, 137], and there is a need to produce corresponding segmentations. If a scientist has a particular protocol in mind, but there are no existing tools for segmenting it, they can choose the protocol from Pancakes that best aligns with their intended use. This choice will often suffice. In cases where Pancakes’ segmentations are not sufficiently accurate–for example, when targets are more out-of-distribution than those seen in training–the segmentation maps can offer excellent initializations to interactive segmentation systems such as ScribblePrompt [120]. To demonstrate this, we performed a preliminary experiment and found that:

• On average, Pancakes’ predictions can be improved by 5 Dice points with a single interactive click, for set sizes larger than one.

• On average, using ScribblePrompt with a Pancakes-initialized segmentation reduces the number of required interactions by half. If used alone, it takes ScribblePrompt 5 to 8 clicks to reach optimal quality predictions, while, when initialized with Pancakes’ predictions, ScribblePrompt can reach them in 3 to 4 clicks.

Since figures are not permitted in the rebuttal, we cannot provide plots for this experiment, but will include them in the manuscript.

(2) Exploratory population analysis. Pancakes will support users in discovering or selecting segmentation strategies appropriate for their scientific or clinical questions. For example, a clinical scientist who studies how anatomy relates to some outcome (e.g., progression of a disease) or predictor (e.g., genetics) can use Pancakes to quickly extract segmentations of multiple candidate anatomical regions that have never been segmented, compute their volumes, and test correlations with clinical outcomes thereby identifying promising candidate regions.

How can a user identify structures and know which protocols make sense?

Pancakes is intended for biomedical experts. We assume that as they use the tool, they can identify the mapping between a label and a known structure. Because Pancakes provides consistent segmentations for groups of images, this provides some context on what labels represent. In future work, we plan to investigate automating the mapping between a structure and known protocols.

What would be the advantage of using Pancakes rather than 3 separate models trained specifically on these protocols?

Pancakes is not intended to replace clinically validated segmentation protocols. If the user knows which targets they wish to segment, and good models exist for those targets, those models should be used. Pancakes is meant for scenarios where the targets are unknown or for which there are no existing segmented scans or segmentation models.

While the task introduced and being evaluated on is novel, most of the technical components are not. As an example, the loss being optimized on the best predicted mask at every existing iteration is done in SAM (this is also stated in the paper).

We agree that the individual components of our framework have been explored in prior contexts. Our key contribution is in formulating an interesting new problem, and in carefully adapting and integrating techniques to solve it. This includes developing a version of the loss function that supports consistency, adapting the synthetic data so that the produced samples are consistent and proposing a lightweight architecture that can efficiently sample protocols. However, we emphasize that our main contribution lies in introducing a fundamentally new framing to medical image segmentation, and alleviating the annotation requirements on the user.

Small model and resource constrained setting

There is strong evidence that UNet-based architectures achieve performance similar to transformer-based architectures in medical image segmentation [134, 135]. Using a relatively small number of parameters limits overfitting, and especially task-overfitting, where the model remembers the structures to segment instead of proposing diverse segmentations. An additional benefit of using a small model is inference time efficiency. We did not have the time to fully train larger models during the rebuttal period. However, preliminary results indicate that increasing the number of parameters does not lead to an improvement in performance. We also highlight that our proposed framework is not dependent on the choice of the neural network that captures the function $f_{\theta_f}(.)$, which infers the distribution parameters $\phi$. The UNet implementation of $f_{\theta_f}$ could be replaced with another network, such as the SAM featurizer. We will discuss this in the revised manuscript.

[133]: Merdietio Boedi, Rizky, et al. "3D segmentation of dental crown for volumetric age estimation with CBCT imaging." International Journal of Legal Medicine 137.1 (2023): 123-130.

[134]: Isensee, F. et al. (2024). nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15009. Springer, Cham.

[135]: Carsen Stringer and Marius Pachitariu. Transformers do not outperform cellpose. bioRxiv, pp. 2024–04, 2024

[136]: TopBrain 2025 Grand Challenge

[137]: Greve, Douglas N., et al. "A deep learning toolbox for automatic segmentation of subcortical limbic structures from MRI images." Neuroimage 244 (2021): 118610.