SegVol: Universal and Interactive Volumetric Medical Image Segmentation

Thank you for your valuable comments and kind words. We answer your questions as follows.

Q1: Figure 2, the violin plots seem confused and it is not cited in the paper.

Figure 2 is cited on Page 7 Line 205: ‘We visualize the Dice score distributions of all methods in all the tasks as violin plots, depicted in Figure 2.’

Figure 2 on Page 5 and Table 2 on Page 6 describe the results of the same experiments from two different perspectives. Table 2 presents the precise quantitative results (the median value), while the violin plots in Figure 2 illustrate the distributions of the results for intuitive comparison.

Q2: Will some of the 25 public volumetric medical segmentation datasets share the same data? Can the authors provide more detailed information about the datasets?

There are no duplicate cases in our processed 25 public datasets. We used the hash code to validate all cases (image-label pair) during the data processing phase. It was found that the LiTS dataset contains many duplicate cases which also exist in other datasets, thus the whole LiTS dataset was removed. Besides, one duplicate case was found and removed in the remaining 25 public datasets.

For more detailed information about these open-source datasets, please refer to their official websites which are provided in Page 17 Table 5.

Q3: How do the authors process the collected data with inconsistent annotations?

It is an important advantage of SegVol that the model is compatible with inconsistent but reasonable annotations. There is no need for additional data processing. For example, the whole ‘kidney’ is annotated in one hospital, while more specifically ‘left kidney’ and ‘right kidney’ are annotated in other hospitals. SegVol can learn to comprehend such inconsistent annotations and generalize well to users’ rough or precise semantic+spatial prompts in the testing phase.

Thank you for your constructive comments. Below we answer the specific questions.

Q1: As shown in Appendix Table 5, the number of train-set volumes is very unbalanced, which will affect the performance of the proposed method.

It is supposed that the unbalanced training set may cause poor performance on the minor categories/domains in some tasks. However, this problem does not matter in our work. We studied this problem by training SegVol on those small sub-datasets and comparing its performance to that trained on the joint of all datasets. Specifically, BTCV, MSD-spleen, and FLARE22 are used as the minor sub-datasets (domains), which have 24, 32, and 40 training samples respectively, and account for only 0.5%, 0.7%, and 0.9% of the whole training set. The following table shows that SegVol trained on the whole dataset, which consists of 25 sub-datasets, achieves much better performances than those trained on the small sub-datasets respectively. The results indicate that SegVol, as a foundation model, performs well on minor sub-datasets (domains) and does not suffer from the problem of unbalanced training data. We will provide more analysis in the final version.

Q2: For unseen anatomical categories, what is the segmentation effect of SegVol?

In Page 6 Table 2 and Page 7 Line 202, we presented the experiments on ULS23, where most lesion categories are unseen for SegVol. The unseen categories include most abdominal lesions, all bone lesions, some lung lesions, and all mediastinal lesions. For unseen categories, we use spatial prompts and general semantic prompts, e.g., ‘lesion’ or ‘tumor’, to drive the model. The results show that SegVol can achieve good segmentation performance (Dice: 0.7046) on these unseen categories.

After the review phase, we will host an online running model for users to test various cases from unseen categories.

Q3: What are the complexity, number of parameters and running speed of the SegVol?

The following table shows the Total Parameters, the average Multiply-Accumulates(MACs), and the average Time required to process a case of the different methods. The comparison indicates that our method takes less computational cost while achieving much better performance.

Note that when calculating MACs Per Case and Time Per Case, the slice-by-slice calculation of the 2D method and the scanning process of the 3D method are accumulated respectively. SAM-MED3D only processes volume with a size of $128\times128\times128$. The experiments are implemented on the validation set of AMOS22, and the setting is the same as that in Sec. 3.2.

Q4: Minor: The reference format should be unified.

Thanks for your reminder. We will unify the reference format in the final version.

Dear Reviewer rczM,

We hope our response addresses your questions. Please feel free to reach out if you have any further inquiries. We would greatly appreciate it if you could consider raising your rating.

Thank you very much!

Thank you for your high recognition of our paper and the detailed feedback. We answer your questions as follows.

Q1: Clarity of the training algorithm. Describing the steps with a paragraph or diagram.

Thank you for the suggestion. We will provide the detailed text description of the Training Algorithm in the final version. Here, we give a simple diagram, Figure 1 in the attached PDF, and a brief description for your reference. Specifically, each case (training sample) consists of an Image x, a Ground Truth(GT) Mask Set Y, and a Pseudo Mask Set Z. The training loss of each sample consists of the ground-truth loss and the pseudo loss. The ground-truth loss is computed by inputting the image, the ground-truth mask (label) and the sampled prompt into the model, while the pseudo loss is computed by inputting the image, the pseudo label and the fixed prompt into the model. Finally, the model is updated by minimizing the weighted sum of the two losses.

Q2: Why is FH algorithm to generate pseudo masks? How is it supposed to remove spurious correlation between datasets?

The FH algorithm is an unsupervised segmentation method that can generate segmentation masks purely based on the voxel similarity of regions in CT scans. In the beginning, we also tried several candidates, including morphological_chan_vese, morphological_geodesic_active_contour, quickshift, slic, etc. Finally, we found the FH algorithm performs best, and thus chose it to generate pseudo masks.

Since in most datasets, the 3D CTs are annotated with only a few segmentation categories, which causes a spurious correlation between the labeled categories and the specific dataset. Using pseudo masks can supplement unlabeled categories in a dataset, therefore relieving the spurious correlation.

Q3: Some details are lacking such as the number of layers of each network, the number of heads in the ViT, etc.

Thanks for reminding, we will add the network details in the paper. Here are main hyper-parameters of the ViT in SegVol:

patch_size=(4, 16, 16), num_layers=12, num_heads=12, hidden_size=768.

The code and model weights of SegVol will be released after the review period.

Q4: Why is the computational cost reduced using the zoom-out-zoom-in principle compared to the sliding window?

The traditional sliding window method requires scanning the entire 3D-CT, processing hundreds of windows. In contrast, the proposed zoom-out-zoom-in mechanism only requires one global inference of 3D-CT and scanning the ROI with dozens of windows.

Q5: How are prompts generated from the coarse segmentation masks?

The process of prompts generated from the coarse segmentation masks is the same as that of prompts generated from ground truth masks, where point prompts are created in or beside the targets, bbox prompts are formed near the target boundaries, and text prompts are derived from category names. The detailed strategies are presented in Page 4 Sec 2.3.

Q6: Could you indicate the training time (SimMIM and finetuning)?

SimMIM pre-training time: ~20$\times$8 GPU hours on NVIDIA A100-SXM4-40GB.

Finetuning time: ~300$\times$8 GPU hours on NVIDIA A100-SXM4-40GB.

Q7: Why did you choose these 3 specific datasets as external test sets?

AMOS22 is a popular dataset for medical image segmentation. Many works are trained and evaluated on it. The validation set of AMOS22, as a representative dataset of abdominal organs, has a large number of samples and includes 15 major abdominal organs.

The ULS23 dataset is a novel large-scale lesion segmentation dataset with thousands of cases covering various lesions, which is a challenging benchmark of lesion segmentation.

SegTHOR dataset focuses on the thorax, which can be a supplement of AMOS22 (abdomen) and ULS23 (lesions). On the whole, the three datasets cover most of the segmentation tasks of organs and lesions in the thorax and abdomen.

Q8: Why is data salability a good thing? Why is this part in the discussion?

The scaling law of foundation models has been verified in multiple CV and NLP tasks. We achieved the success of scaling law in 3D medical segmentation by the design of universal prompts and pseudo masks for joint learning on datasets with inconsistent annotations. We provided a preliminary experiment of the data scaling law in Figure 3 (a) Page 7, which shows that 1) the performance improves significantly with more training data, 2) and SegVol has not yet reached its ceiling if more training data is provided. We will provide more detailed discussion and comparison to other methods in the final version.

Q9: The amount of data required to train the model (SegVol) is not discussed. Would a few-shot finetuning method work on small datasets (10 samples)?

Figure 3 (a) in Page 7 demonstrates the relationship between the model performance and the amount of finetuning training data. Generally speaking, more training data leads to better performance.

We conducted the few-shot finetuning experiment on small datasets, FLARE22 (40 training cases) and MSD-spleen (32 training cases), to study the few-shot learning ability of SegVol. The table below demonstrates that 1) finetuning SegVol on dozens of samples works well on easy datasets such as MSD-spleen, in which the few-shot performance is close to the joint finetuning on all datasets; 2) for challenging datasets such as FLARE22, finetuning on all datasets can achieves much better performance than few-shot finetuning.

Note: SegVol* represents the model finetuned on all datasets.

Thank you for your detailed answers to each of my interrogations. All the questions have been answered by the authors.

Q1 : the algorithm is clearer with this small text, it should be included in the article. Q2 : the choice of FH is then arbitrary, it should be specified in the article. Q3 : For the network details, maybe a results of a torchinfo summary, or something in this idea could be added in appendix so that every hyperparameters could be seen visually (it would be be a bonus, but optional). Q9 : The results of joint finetuning compared to few-shot finetuning is relevent, and could be at least mentionned.

I will maintain my grading, considering the clarity (presentation) will be improved, but is not yet done.