Text-promptable Propagation for Referring Medical Image Sequence Segmentation

Thank you for your encouraging recognition of our work and constructive feedback. We have carefully addressed each of your points in detail and hope that these clarifications effectively resolve your concerns.

[iLhU-W5] Separate models for each of the 18 datasets or a single model with all datasets?

We train a single model across all datasets to achieve a universal solution for medical image sequence segmentation, in contrast to specialized methods that are trained for each dataset individually. Despite this challenging setup, our method achieves competitive performance.

[iLhU-Q1, Q3, W1, W3] Comparison with stronger baselines and 3D models.

Thank you very much for pointing out the area for improvement. We compared our method with other universal methods, such as CLIP-based segmentation [1], getting competitive results (73.70% vs. 76.48%).

Swin-UNet [2], a strong 3D model which is a UNet-like Transformer for medical image segmentation, was also evaluated. Using its official open-source code, we trained it on the BTCV dataset, achieving an average Dice score of 73.38%. Our model outperforms Swin-UNet by 3.1%, with an average Dice score of 76.48%.

Our Triple propagation leverages consistency in appearance and spatial relationships across frames or slices in the temporal order of medical image sequences. This approach exhibits strong tracking ability while maintaining a lower computational cost (130.77 GFLOPs vs. 142.78 GFLOPs) compared to 3D models.

• Table 1: Comparison results with stronger baselines and 3D models on abdominal organs.

  Method	Aorta	Kidney (L)	Kidney (R)	Liver	Spleen	Stomach	Pancreas	Gallbladder	Average
  CLIP-driven [1]	88.31	84.94	81.04	93.03	79.83	65.88	42.74	53.85	73.70
  Swin-UNet [2]	77.85	82.34	75.60	90.07	86.97	66.89	52.49	54.84	73.38
  Ours	86.14	87.53	84.16	90.32	88.41	67.35	47.61	60.29	76.48

• Table 2: Comparison of computational cost with 3D models.

  Method	FLOPs (G) $\downarrow$
  Swin-UNet [2]	142.78
  Ours	130.77

[iLhU-Q2, W2, W4] Value of adding medical text prompts.

The value of adding text prompts lies in clinical scenarios.

• When clinicians need to diagnose diseases that are challenging to detect, such as pneumothorax, they often need the help of radiologists to pinpoint the location. A text-promptable model can automate this process, streamlining workflows and improving efficiency.
• Another critical scenario is minimizing the risk of missed lesions. For example, missing polyps during endoscopy can have severe consequences, potentially endangering the patient's life. A text-promptable model can serve as a reminder, assisting clinicians in identifying such lesions more effectively.

[iLhU-Q4, W4] Personalization of descriptions.

We greatly value your suggestion and agree that personalized and context-specific text prompts can enhance the model's clinical applicability. As you mentioned, using text derived from actual clinical reports and tied to the same patient as the image would enable the model to understand personalized differences, thereby adding clinical value. We are committed to annotating detailed, patient-specific prompts for each sequence, even when they belong to the same category. For example, "The polyp in this image is tan located at the left." or "The polyp is pink and flat." We believe such efforts will better support the task of Referring Medical Image Sequence Segmentation.

[1] Liu, Jie, et al. "Clip-driven universal model for organ segmentation and tumor detection." ICCV 2023.

[2] Cao, Hu, et al. "Swin-unet: Unet-like pure transformer for medical image segmentation." ECCV 2022.

Thank you very much for your additional feedback. Below, we address your remaining concerns regarding our performance compared to state-of-the-art (SOTA) methods and the value of pseudo-text prompts.

1. Performance Compared to state-of-the-art and 3D Models

We respectfully note that our method achieves state-of-the-art performance when compared to strong baselines, including widely recognized 3D models such as Swin-UNet. On the BTCV dataset, our method achieves an average Dice score of 76.48%, which outperforms Swin-UNet (73.38%) by 3.1% points. Furthermore, our approach is computationally efficient, requiring fewer FLOPs (130.77 GFLOPs for our method vs. 142.78 GFLOPs for Swin-UNet).

In addition, when compared to other universal segmentation approaches such as CLIP-driven segmentation, our method consistently achieves superior results (73.70% for CLIP-driven vs. 76.48% for our method on average Dice score). These improvements highlight the efficacy of our proposed framework, particularly the advantages of the triple propagation mechanism in leveraging sequence consistency.

Our method also demonstrates robustness across various datasets (15 datasets for organs and 5 datasets for lesions), confirming its generalization ability. While 3D models may rely on volumetric context, our results show that the combination of temporal modeling with text prompts provides stronger performance, while reducing computational overhead, making it more suitable for large-scale clinical applications.

2. Impact of and Pseudo-Text Prompts

We appreciate your concerns regarding our explanation of text prompts. While pseudo-text lacks personalization, it offers distinct advantages:

• Performance Enhancement: In our experiments, pseudo-text consistently improved segmentation performance, even for challenging tasks. For example, when segmenting lesions like polyps, the inclusion of text prompts improved the average Dice score by 9.0% across five lesion types, including brain tumor, liver tumor, kidney tumor, breast mass, and polyp.

• Flexibility and Adaptability: Text prompts allow the model to identify target organ or lesion under their guidance, minimizing the risk of missed lesions. This capability supports radiologists and endoscopists in identifying nodules, polyps, and other critical abnormalities. The flexibility to focus on specific anatomical entities, such as the pancreas in multi-organ CT scans for pancreatic cancer diagnosis, makes our approach more practical for real-world deployment.

• Potential for Personalization: As noted previously, we are actively exploring the incorporation of patient-specific clinical reports to further enhance the utility and clinical relevance of the model.

We hope these clarifications address your remaining concerns. Your valuable suggestions are greatly appreciated.

[bfV2-W5] Exploration of text prompts.

We have expanded our experiments to evaluate the impact of various prompt types and designs:

1. different specificity for anatomical entities. We tested simplified prompts containing only class names, eg. "a MRI of the myocardium", "a CT of the liver tumor", "an ultrasound image of the prostate". These prompts yielded Dice scores of 75.65% for organs (-5.12%) and 67.61% for lesions (-5.08%), indicating that detailed and well-designed prompts are effective to enhance segmentation performance.

2. different attributes for anatomical entities. We introduced new prompts incorporating attributes like [position/location], [boundary] and [density]. For example, for myocardium:

• Position: "The myocardium is located between the endocardium and epicardium of the heart."
• Boundary: "The boundaries of the myocardium on MRI are well-defined, showing a clear demarcation."
• Density: "The myocardium typically exhibits low signal intensity on T2-weighted images."

These prompts achieved Dice scores of 80.84% for organs (+0.07%) and 73.91% for lesions (+0.22%), demonstrating that well-designed text prompts effectively enhance segmentation accuracy.

[bfV2-W6] Inconsistency of ambiguous expressions.

Thank you for pointing out the ambiguity in terminology. We will carefully replace "the referred object" at P2 L92, P3 L146 and L159, P5 L243 and P6 with consistent and precise expressions.

[1] Tian, Zhi, Chunhua Shen, and Hao Chen. "Conditional convolutions for instance segmentation." Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16. Springer International Publishing, 2020.

[2] Isensee, Fabian, et al. "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation." Nature methods 18.2 (2021): 203-211.

[3] Cao, Hu, et al. "Swin-unet: Unet-like pure transformer for medical image segmentation." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

Thank you for your encouraging recognition of our work and constructive feedback. We have carefully addressed each of your points in detail and hope that these clarifications effectively resolve your concerns.

[bfV2-Q1] What are the clinical scenarios and medical relevance of the proposed task?

In clinical scenarios, text-promptable method offers distinct advantages across various user groups by addressing key needs and enhancing workflows:

1. For Radiologists: The method serves as an efficient double-check mechanism, especially in high-stakes scenarios where subtle lesions might be missed due to fatigue or high workload. By automating the detection and segmentation of potential areas of concern, the model assists radiologists in validating their diagnoses and avoiding missed lesions.
2. For Clinicians with Limited Imaging Expertise: Many clinicians, such as general practitioners or specialists outside radiology, lack advanced training in interpreting complex imaging studies. Text-promptable segmentation bridges this expertise gap by providing clear visual cues and annotations to help identify critical findings. For example, the system can highlight the location of a pulmonary nodule based on a textual description, acting as a guide and saving valuable time during decision-making. This approach builds a stronger connection between radiologists and clinicians, fostering collaborative and effective patient care.
3. For Patients: Simple and intuitive visualizations of segmented anatomical structures or lesions can help patients to better understand their condition. For instance, finding out a lung nodule or an abnormality in an endoscopy image offers patients a clear representation of their medical situation, aiding in informed discussions with healthcare providers and improving their engagement in their treatment plans.

[bfV2-W2] Value of text prompts.

While fully supervised methods have demonstrated high accuracy in medical image segmentation, text prompts provide distinct advantages:

• Clinical Utility: Clinicians who may not have the radiological expertise to interpret complex imaging studies often need guidance on lesion locations. They may already have preliminary information about the presence of a lesion but unclear about its exact location (eg., pneumothorax, lump). Text prompts enable segmentation by helping clinicians locate lesions, e.g., identifying a lump in the lungs based on prior imaging.
• Multi-class Adaptability: Traditional methods are constrained by pre-defined categories, limiting adaptability. Text-promptable segmentation offers the flexibility to dynamically define and target specific categories, enhancing clinical relevance.
• Performance Impact: Our experiments demonstrate that text prompts contribute to segmentation accuracy. Without prompts, the average Dice score for five lesions decreases by 9.0% (from 72.69% to 63.69%), underscoring their importance.

[bfV2-W3] Explanation of the box head.

The mask head is implemented using dynamic convolution [1]. It takes multi-scale features from the feature pyramid network (FPN), concatenates them with relative coordinates, and uses a controller to generate convolutional parameters. The relative coordinates are generated by the box head, which provides a coarse location as an initial reference for mask prediction, as you correctly noted.

[bfV2-W4] Comparison with traditional supervised models.

We greatly value your suggestion that including comparisons with state-of-the-art models like nnU-Net strengthens the evaluation. Using the official nnU-Net code, we conducted experiments on abdominal organ segmentation (due to time constraints) and trained eight separate models, one for each organ, as nnU-Net specializes in per-organ segmentation. The average Dice score is 75.46%. Unlike nnU-Net, our model is universal, training on all organs and lesions simultaneously. Notably, our method outperforms Swin-UNet, achieving a 3.1% improvement in the average Dice score (76.48% vs. 73.38%).

[QiAh-Q4] Experiments benchmarking different versions of descriptions.

We conducted experiments with two types of prompt variations to evaluate their impact on segmentation performance:

1. different specificity for anatomical entities. Simplified prompts with only class names resulted in Dice scores of 75.65% for organs (-5.12%) and 67.61% for lesions (-5.08%). Examples of such prompts include: "an MRI of the myocardium", "a CT of the liver tumor", "an ultrasound image of the prostate". The results demonstrate that detailed, descriptive prompts significantly enhance segmentation performance compared to simplified ones.

2. different attributes for anatomical entities. We add new prompts describing anatomical entities with attributes such as [position/location], [boundary] and [density]. For example, for the myocardium:

• Position: "The myocardium is located between the endocardium and epicardium of the heart."
• Boundary: "The boundaries of the myocardium on MRI are well-defined, showing a clear demarcation."
• Density: "The myocardium typically exhibits low signal intensity on T2-weighted images."

These attribute-rich prompts yielded Dice scores of 80.84% for organs (+0.07%) and 73.91% for lesions (+0.22%), indicating that richer, attribute-based descriptions enhance segmentation accuracy when sufficient contextual information is provided.

[QiAh-Q5] Does your model generate binary segmentation using text to refer to class semantics?

Yes, we use text to refer to the class semantics. But in our model, the class label is used solely to indicate whether the current target is the referred object (binary 0/1) during training. It does not contain semantic information about the target anatomy itself. We sincerely hope this explanation addresses your concern.

[1] Ma, Jun, et al. "Segment anything in medical images." Nature Communications 15.1 (2024): 654.

[2] Cao, Hu, et al. "Swin-unet: Unet-like pure transformer for medical image segmentation." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[3] Isensee, Fabian, et al. "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation." Nature methods 18.2 (2021): 203-211.

[4] Zhao, Theodore, et al. "BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once." arXiv preprint arXiv:2405.12971 (2024).

Thank you for your encouraging recognition of our work and constructive feedback. We have carefully addressed each of your points in detail and hope these clarifications effectively resolve your concerns.

[QiAh-W1, Q1] What clinical scenario or problem is the paper targeting?

• Definition of medical image sequences. As you correctly pointed out, medical image sequences consist of consecutive slices of the same 3D image or video captured at one inspection. These include both temporally related frames from videos and spatially related slices within volumes. Such sequences differ from CT scans of the same patient taken at different time points. Modern medical imaging modalities are increasingly dominated by sequence-based data, such as CT or MRI slices, ultrasound scans, and endoscopy videos. Our work specifically targets medical image sequence segmentation—segmenting target objects within these sequences, guided by medical text prompts.

• Clinical scenarios. In clinical practice, pre-defined text prompts are utilized to identify and segment anatomical structures or lesions that are challenging to recognize, such as pneumothorax. This text-promptable method significantly aids clinicians who may lack radiological expertise to interpret complex imaging studies. Radiologists often provide guidance to clinicians, such as pointing out the exact location of a pulmonary nodule. Our method automates this process, thereby saving time and improving diagnostic efficiency. By embedding this contextual understanding into our model, we aim to bridge the gap between radiological expertise and broader clinical applications.

[QiAh-W2, Q3] Comparison with medical domain state-of-the-art.

We greatly value your suggestion to strengthen the comparative experiments. In response, we have incorporated evaluations against state-of-the-art methods, including the SAM-based MedSAM [1] and supervised models such as Swin-UNet [2] and nnUNet [3].

Due to time constraints, the experiments were conducted on the BTCV dataset, which includes 8 abdominal organs. We evaluate MedSAM under the inference tutorial with text-based prompts, which adopts the CLIP text model as the text encoder. Although MedSAM’s training dataset includes abdominal organs from the FLARE challenge, its average Dice score of 27.60% suggests that it struggles with zero-shot segmentation tasks. In contrast, our proposed method demonstrates superior generalization capabilities, as shown by the results presented in Table 5 of the main text.

Results for the 2D supervised models (Swin-UNet and nnUNet) are included in the table below. Notably, our method consistently outperforms state-of-the-art models in the medical domain.

[QiAh-Q2] Comparison with BiomedParse.

BiomedParse [4] is a groundbreaking work in the field, and we will appropriately cite it in the revised manuscript. While BiomedParse demonstrates strong general-purpose capabilities across multiple modalities, our focus is on clinically relevant segmentation tasks driven by domain-specific text prompts. Our approach prioritizes automation and accuracy to address clinical needs, particularly in challenging scenarios where text-promptable segmentation aligns with the requirements of clinicians.

Thank you for your encouraging recognition of our work and constructive feedback. We have carefully addressed each of your points in detail and hope that these clarifications effectively resolve your concerns.

[SfoW-W1(a)] Has any evaluation been conducted on 2D datasets?

Medical image sequences include both temporally related frames (e.g., in videos) and spatially related slices (e.g., in volumes). Our unified framework bridges 2D and 3D segmentation tasks, addressing diverse clinical needs. We classify datasets such as CT and MRI as 3D datasets, while ultrasound and endoscopy images are categorized as 2D datasets. Examples of 2D datasets include CAMUS, Micro-Ultrasound Prostate Segmentation Dataset, CVC-ClinicDB, CVC-ColonDB, ETIS, and ASU-Mayo.

[SfoW-W1(b)] Comparison against 3D segmentation algorithms.

The work CLIP-driven universal model for organ segmentation and tumor detection [1] is an excellent reference, which we will cite in the revised paper. Using the official code and the same data splits as ours, we trained this model on abdominal organs for 200 epochs. It achieved an average Dice score of 73.70%, while our method attained 76.48%, demonstrating superior performance. Our method benefits from customized prompts, enabling domain-specific adaptability. Notably, our model is significantly more computationally efficient, with 130.77 GFLOPs compared to over 300 GFLOPs for [1]. For comparison with nn-UNet [2], we trained the model using the official codes of [2] under default settings, resulting in an average Dice score of 75.46%. The corresponding results are presented below:

• Table 1: Comparison of computational cost.

  Method	FLOPs (G)
  Clip-driven [1]	>300
  Ours (TPP)	130

• Table 2: Comparison against 3D segmentation algorithms on abdominal organs.

  Method	Aorta	Kidney (L)	Kidney (R)	Liver	Spleen	Stomach	Pancreas	Gallbladder	Average
  CLIP-driven [1]	88.31	84.94	81.04	93.03	79.83	65.88	42.74	53.85	73.70
  nn-UNet [2]	92.17	79.59	78.42	87.56	81.18	68.07	56.84	59.87	75.46
  Ours	86.14	87.53	84.16	90.32	88.41	67.35	47.61	60.29	76.48

[SfoW-W2(a)] Ablation study on fusion designs.

Thank you for pointing out the need for ablation studies on fusion strategies. We have conducted additional experiments to evaluate the effectiveness of our proposed "Cross-modal Prompt Fusion" against simpler strategies. The results confirm that our "Cross-modal Prompt Fusion" significantly outperforms these alternatives, demonstrating its efficacy in leveraging image and text features for segmentation.

• Table 3: Ablation studies on fusion designs.

  Fusion design	Dice score (organ)	Dice score (lesion)
  Average	78.88	68.89
  Concatenation	77.50	69.64
  Ours	80.77	72.69

[SfoW-W2(b)] Analysis of N_q and computation cost.

As you correctly pointed out, the first image has N_q queries. Due to our propagation strategy, the best prediction of the first image is propagated to subsequent images, reducing the number of queries to just one for the rest of the images. This optimization significantly lowers the overall computational burden.

• Trainable params: 52.97M.
• FLOPs: 130.77 GFLOPs, considerably lighter than [1] (FLOPs > 300G).

The effect of N_q was studied under our propagation strategy. The results demonstrate that tracking the referred object becomes more robust when using a selection of queries (5->1->1), validating the design choice.

• Table 4: Analysis on query selection. The first column represents the number of queries from Slice 1 to Slice 3.

  The number of queries for slices	Dice score (organ)	Dice score (lesion)
  5 -> 5 -> 5	79.47	70.98
  5 -> 3 -> 1	78.47	71.67
  5 -> 1 -> 1	80.77	72.69

[1] Liu, Jie, et al. "Clip-driven universal model for organ segmentation and tumor detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Isensee, Fabian, et al. "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation." Nature methods 18.2 (2021): 203-211.