Thank you very much for your constructive suggestions. Below is our detailed response to answer your concerns.

W1&W5: Motivation and Details of textual prompts

The motivation for using textual descriptions was to provide the model with the general concept of each category. While descriptive texts encompass intricate and rare anomalies using domain knowledge, learning alignments between textual and visual representations remains challenging, particularly for fine-grained details such as tumors with variations in shape, size, density distribution, and blurred boundaries. In real-world scenarios, acquiring specific details like quantitative measurements for each sample prior to segmentation is often impractical. Considering the vast diversity of tumors, we utilize general knowledge to convey the concept of each tumor type. Given the vast diversity of tumors, we employ general knowledge to convey each tumor type's concept. Utilizing GPT-4, we generated descriptions with medical domain knowledge for each category over 20 times. A board-certified physician then refined these descriptions based on the results. Further details on our textual prompts are discussed in the Text Descriptions. For example, the description of colon tumors ("On CT scans, colon tumors can present with varying densities and might be associated with surrounding inflammation, adjacent organ invasion, or regional lymph node enlargement.") contains an abstract notion of invading target boundaries. During the training process, we use the long description for the positive categories and randomly sample short phrases for those negative categories. In the inference stage, we apply the long description for all categories. To verify the effectiveness of textual descriptions, we replace them with short phrases during the inference stage, the result can be seen as follows:

The observed declines demonstrate that textual descriptions, which cover the general concept of potential cases, significantly benefit the tumor segmentation process. To provide an intuitive understanding of these results, we present the qualitative results in the Figure (Figure 6 of the rebuttal PDF). Additionally, we enlisted a physician to annotate the tumor regions. The figure shows that a lack of detailed knowledge leads to overlooking crucial details. Importantly, the results from CAT are more closely aligned with those delineated by the expert, who possesses professional medical knowledge. This validates our approach of enhancing the segmentation process by incorporating comprehensive textual knowledge from the medical domain.

We hope the updated clarification in the Author Rebuttal could provide a clear understanding for you.

W2: Details of Anatomical Prompt

We are very grateful for your serious reading. In our paper, we leverage the bounding box derived from masks in the public dataset and relevant anatomical structures to prepare a set of prompt volumes for each category, which we then standardize to a uniform size. The process begins by cropping the image according to the bounding box. Subsequently, we employ two different strategies to adjust the cropped volumes to the required input size of the Anatomical Prompt Encoder ($96 \times 96 \times 96$): 1. For volumes larger than the target size, we resize them to $96 \times 96 \times 96$. 2. For smaller volumes, we re-crop the image using a center-crop paradigm to achieve the size of $96 \times 96 \times 96$. It is important to note that all anatomical prompts used for tumor categories are processed using the second strategy. We exclude cases where the tumor size exceeds $96 \times 96 \times 96$, as these instances have been effectively addressed by previous methodologies. Our paper primarily focuses on the integration of anatomical and textual prompts for challenging segmentation tasks.

W3: Clarification of Figure 2

Sorry for any misunderstanding caused by the omission of image descriptions. As shown in the Figure (Figure 7 of the rebuttal PDF), the part highlighted by a red box is the output. We chose this method of illustration to provide a more intuitive understanding; we regret any confusion this may have caused. The segmentation maps are obtained by a multiplication operation between the decoded segmentation query features $\mathbf{O}_S$ and the pixel embedding map $O$. During inference, both prompt inputs need to be used.

W4: Experimental Details

Sorry for any confusion caused by missing details in our experimental descriptions. In our experiments, the results for the Universal model were derived from the official pre-trained weights, which were also trained on the same datasets as our study. In contrast, the SAM-based models were trained on a broader set of datasets, including the ten datasets discussed in our paper. Regarding the CT-SAM3D model, the detailed organ-wise segmentation results were sourced from the original CT-SAM-Med3D paper. It is important to note that the results presented were obtained using a prompt number ($N=5$), which is derived from the ground truth. As indicated in the subsequent Figure (Figure 3 of the rebuttal PDF), there is a significant performance drop in the average Dice score when the number of prompts decreases from 5 to 1, falling below 85%. Thanks to the design of our CAT model, it demonstrates superior results in organ segmentation without the need for human interaction

Thank you for your thorough review. Below, we will address your concerns on each point.

W1&W3: Details of textual descriptions and Reasons for superior performance

As detailed in Section 3.4, for each category, we curate long descriptions. As highlighted in the motivation section, learning the alignments between textual and visual representations is challenging, particularly for fine-grained details. In real-world scenarios, obtaining specific details for each sample prior to segmentation is impractical. Considering the vast diversity of tumors, we leverage general knowledge to convey the concept of each tumor type. Details can be in Textual Prompts. To better support our claim about the utility of textual prompts, we conducted ablation studies in Table 3 of our paper. The observed improvement in the third row (compared to the first row) underscores the effectiveness of textual prompts in helping deal with the diversity and variance of tumors.

As shown in Table 2, CAT shows strong capabilities in dealing with tumors at different stages. The reasons can be attributed to several factors: 1. The textual prompt for colon tumors provides context for 'surrounding inflammation, adjacent organ invasion, or regional lymph node enlargement,' which are critical in different stages. 2. The visual prompts offer intuitive and direct examples to highlight the appearance features. 3. The carefully designed mask in the PromptRefer helps to handle tumors that invade nearby organs or tissues.

W2: Organ segmentation results

1. The detailed organ-wise segmentation results are derived from the original CT-SAM-Med3D paper. It is important to note that the results presented were achieved with a prompt number ($N=5$), which is derived from the ground truth. As illustrated in the cited Figure (Figure 3 of the rebuttal PDF) from CT-SAM-Med3D, reducing the number of prompts from 5 to 1 leads to a significant performance decrease, with the average Dice score falling below 85%.

2. The average organ segmentation scores for nn-UNet and Swin UNETR are presented below. Our design enables CAT to outperform most SAM-based models in the medical domain and established state-of-the-art (SOTA) segmentation models.

• nn-UNet: Pan.-79.36, RAG.-68.37, LAG.-73.31, Eso.-76.16, Duo.-72.43, Avg.-83.38
• Swin UNETR: Pan.-87.05, RAG.-66.17, LAG.-74.18, Eso.-75.65, Duo.-48.51, Avg.-82.78

W4&W6&W7: Details of our method

Factually, CAT significantly diverges from ZePT in several key aspects. Besides the introduction of visual prompts, CAT utilizes more descriptive texts as textual prompts to convey general concepts, while ZePT only uses knowledge for the final alignment. For handling two types of prompts, CAT employs distinct strategies: we use soft assignment to gather all potentially relevant visual features for textual prompts while applying hard assignment to secure discriminative visual regions without overlaps. This is markedly different from ZePT’s unified refinement process. Furthermore, in PromptRefer, we carefully design attention masks according to clinical knowledge. In summary, CAT mainly focuses on combining anatomical and textual prompts to enhance segmentation tasks, whereas ZePT delves into identifying anomalies from original visual features.

Figure 2(c) illustrates how segmentation queries interact with a mixed group of refined prompt queries via a cross-attention mechanism in the PromptRefer module. For example, Stage-IV colon tumors often invade adjacent organs such as the intestines. In such cases, the segmentation query $\mathbf{Q'}_{Si}$ for colon tumors is specifically directed to attend to the refined query set of anatomical $\mathbf{Q'}_A$ and $\mathbf{Q'}_T$ encompassing Colon, Stomach, Duodenum, Intestine, and Rectum. We hope the Figure (Figure 4 of the rebuttal PDF) clarifies you more clearly.

W5: Effectiveness of contrastive alignment

Contrastive alignment is utilized to further push segmentation queries to be close to the referenced prompt for segmenting the corresponding category. To validate the effectiveness, we trained without utilizing contrastive alignment. The results are shown in the following table. Eliminating contrastive alignment leads to a performance drop.

We also use t-SNE to visualize the distribution of decoded segmentation query features $\mathbf{O}_S$ in Figure (Figure 5 of the rebuttal PDF). We can observe that segmentation queries are more separated in the feature space with contrastive alignment.

Questions:

Thanks for your careful reading. As you mentioned, the Linear function is to transform the dimensions of the embedding space. The "snowflake" icons in Figure 2 indicate that these are frozen models that have not been trained. Following previous work, the learnable segmentation queries are initialized randomly. Each of them is responsible for segmenting one category.

The reason for not reporting HD95 scores for SAM-based methods: The HD95 metric is used to assess the accuracy of image segmentation by measuring the largest distances between predicted and actual segmentations. When applying SAM-based methods that require predefined target key points as the prompts for segmentation, the predefined keypoints could artificially enhance segmentation accuracy near those points. Hence, reporting HD95 scores for SAM-based methods against others without such provisions is considered inequitable.

Thanks for the response. I still have some doubts:

• I understand that adding generic definitions has led to some improvement, but it doesn't fully align with the motivation of your work. For example, it's unclear if your text encoder can effectively differentiate between different cancer stages in real-life scenarios. Given that your paper focuses on the challenges of long-tail distribution (typically for tumors), this should have been demonstrated experimentally rather than assumed. If your paper's motivation was simply "Do organ definitions improve organ segmentation performance?" then the approach would be more acceptable.

• Regarding the results, shouldn't all the organs be reflected in Table 1? The current comparison seems limited to only a few organs, which can be confusing. A more comprehensive comparison would be helpful. Also, outperforming SAM-based models alone isn't sufficient since it's well-known that SAM models don't excel in medical imaging tasks.

Thank you for clarifying the other points. Including these explanations in the original paper would greatly enhance its clarity. Additionally, consider adding the contrastive alignment results to your ablation studies for a more thorough analysis.

Thank you for your constructive comments. We will address your concerns in the following parts.

W1: Details of Experiments

Sorry for the confusion caused by the omission of certain experimental details. In our experiments, we followed Universal's experimental settings [1]. We also trained the comparison methods that we implemented ourselves on the same 10 datasets. Specifically, we modified the number of output channels in the baseline models (e.g., nnUNet, Swin UNETR) to enable them to perform multi-organ and tumor segmentation.

W2: Details of ShareRefiner and PromptRefer

We hope the provided illustration will give you a clearer understanding of our ShareRefiner and PromptRefer. Both ShareRefiner and PromptRefer are built upon the attention mechanism [2]. Specifically, the ShareRefiner consists of a series of cross-attention blocks where each type of query (i.e., segmentation queries $\mathbf{Q}_S$ , anatomical prompt queries $\mathbf{Q}_A$ and textual prompts queries $\mathbf{Q}_T$) performs cross-attention with the extracted visual features. We use cross-attention to assign all possible relevant visual features to textual prompt queries, and use hard assignment for anatomical prompt queries. The reason for employing hard cross-attention is to ensure that each anatomical query gathers discriminative visual regions without overlaps. The effectiveness of the hard assignment is verified by the results in the following table.

In the PromptRefer, refined segmentation queries $\mathbf{Q'}_S$ engage in cross-attention with refined anatomical prompts queries $\mathbf{Q'}_A$ and textual prompts queries $\mathbf{Q'}_T$ to enhance segmentation. We employ a conventional attention mechanism, supplemented by carefully crafted attention masks, These masks force a group of prompt queries is employed to a specific segmentation query. This process aligns with empirical insights suggesting that accurately localizing typical tumors necessitates recognizing anomalous features within the pertinent organ. As can be seen from the table, this strategy helps to segment tumors that invade other organs (e.g., Stage-IV). We hope the above explanation and the provided code in Supplementary Material can address your confusion.

W3&W4: Clarification of Table 3 and Pre-trained models

1. Sorry for the confusion caused by the use of symbols (✓ and ✔️). As discussed in Section 4.2, the second-to-last row in Table 3, marked with a ✔️, indicates our use of hard assignment across all cross-attention layers. We conducted this experiment to validate our hypothesis that different types of prompts play distinct roles in aggregating visual features. We will clarify the symbol in the revised caption of Table 3.
2. The pre-trained model employed for anatomical prompts is the Swin UNETR. We utilize Clinical-Bert to encode textual prompts.

Questions:

We hope the following responses will address your questions:

Q1&Q2: We present examples of visual prompt in Figure (Figure 1 of the rebuttal PDF) and the details of textual prompt in Json. We have added more comparisons in the case of training models on the dataset and report the number of parameters of CAT and those comparison models. The results are shown in the following tables.

We trained the CAT, ZePT, and nnUNet models on four MSD datasets. The results demonstrate that CAT still achieves superior outcomes even when trained on a single dataset, further validating our approach of integrating textual and visual prompts for segmentation.

Q3: In Figure 4 of our paper, we illustrate how visual and textual prompts enhance segmentation. Unfortunately, the heatmap format may have caused some misunderstanding. To address this, we provide a comparison in Figure (Figure 2 of the rebuttal PDF). While our textual prompts cover most scenarios, using them alone fails to encompass all regions. This observation further supports the claim that aligning textual and visual representations is challenging. Conversely, relying solely on anatomical prompts results in a high false positive rate and overly sharp boundaries. We hope these visual examples provide clearer clarification for you.

Limitations

We appreciate your suggestions. To assess CAT’s efficacy in dealing with rare cases (i.e., the long-tailed problem), we introduce an in-house dataset where colon tumors have invaded adjacent organs. According to medical literature [3] [4], T4 colorectal tumors, which represent only about 5-8.8% of colon tumor cases, pose significant challenges in diagnosis and treatment. Our results demonstrate that CAT significantly outperforms other models in segmenting T4 colon tumors, underscoring the effectiveness of our design in handling complex medical scenarios. We hope our early exploration could bring new insights into the field of the community.

References

[1] Clip-driven universal model for organ segmentation and tumor detection.

[2] Attention is all you need.

[3] Results after multi-visceral resections of locally advanced colorectal cancers: an analysis on clinical and pathological t4 tumors.

[4] Identification of risk factors for lymph node metastasis of colorectal cancer.