Organ-DETR: 3D Organ Detection Transfomer with Multiscale Attention and Dense Query Matching

We express our appreciation to the reviewer for providing the comments and suggestions. According to the comments, we plan to incorporate a paragraph to the Introduction that highlights the key contribution of this study. We also plan to edit the experimental section to highlight the improvements significant by Organ-DETR.

We share the belief that this study is well-suited for ICLR, given its emphasis on novel methods in deep learning. Throughout the manuscript, we treat organ detection as a 3D object detection problem, emphasizing efficient representation for detection tasks and a reliable matching strategy.

Once again, we appreciate all the comments, suggestions, and encouragement provided by the reviewer. We hope we have addressed every comment satisfactorily. Your thoughtful input is greatly appreciated and we kindly invite the reviewer to raise any additional questions or points during the forthcoming reviewer-author discussion period, set to take place from Nov 10 to Nov 22.

We appreciate the comments and feedback provided by the reviewer. We would like to address the concerns raised regarding our method.

Q1- Limited Novelty: The adoption of multi-scale attention in this paper is not novel, as it has been extensively explored in both medical imaging [1] and general visual recognition [2] contexts.

A: The phrase "multi-scale attention" is broadly descriptive and caused confusion by suggesting that Organ-DETR is similar to earlier techniques such as those in references [1] and [2]. It is crucial to emphasize that Organ-DETR is quite distinct from these previous methods in terms of its methodology, conceptual framework, and practical implementation.

In general, [1] and [2] use transformer-based encoders, while Organ-DETR uses a CNN-based encoder. MSA, as a module, refines the features for DETR. Both [1] and [2] lack cross-stage attention, while our MSA incorporates inner-stage self-attention and inter-stage cross-attention.

In detail, MERIT [1] introduces a cascaded backbone with two hierarchical transformers, generating multiscale features that might appear similar to our method. However, the objectives for utilizing these multiscale features differ significantly. In MERIT, the multiscale features contribute to generating multi-stage prediction maps, employed in a multi-stage feature mixing loss aggregation to enhance segmentation. In contrast, our MSA module focuses on cross-attention between multiscale features to capture long-range feature patterns. In MAFormer[2], MAF blocks are employed within a single stage, lacking cross-stage operations. In contrast, our MSA module incorporates inner-stage self-attention and inter-stage cross-attention, effectively capturing long/short-range information. Additionally, the means to capture global features differ. MAFormer uses fully connected layers in the proposed Global Learning with Down-sampling (GLD) module, while our MSA module leverages cross-attention between multiscale features.

Q2- One-to-Many Label Assignment: In the organ detection scenario, the use of a one-to-many label assignment strategy is not reasonable, considering that there is typically only one instance of each organ (e.g., liver, pancreas, spleen) in the human body.

A: We have addressed this concern meticulously earlier in Sec. 2 and subsection “Dense Query Matching” in Sec 3. Below we briefly summarize DQM from the manuscript: The proposed DQM, as a one-to-many matcher, improves the TRUE POSITIVE MATCHING RATE by increasing the ground truth instances to enhance RECALL. Increasing the number of true positive instances in DQM simultaneously decreases the number of negative queries, leading to lower false positives (and thus, higher precision) and boosting the training’s learning pace. In INFERENCE, DQM selects the BEST-PREDICTED query per class/organ whose prediction score is the HIGHEST among the others. Figure 6 provides an ablation study on DQM parameters. Based on the comment, there may have been confusion by the reviewer about the exact strategy of DQM and, therefore, also the confusion about its mechanism. To fix that confusion, we plan to mention different steps of DQM in Sec. 3 clearly to make the DQM’s methodology easy to follow.

Once again, we thank the reviewer for every comment. Our responses aim to address all concerns. If more questions or suggestions arise, we invite the reviewer to discuss them during the Nov 10-22 reviewer-author period. Your thoughtful input is highly valued, and we anticipate productive discussions during this time.

We appreciate the reviewer's comments and feedback on our study. We have taken into account all the comments and kindly request the reviewer to consider our responses, which we have addressed meticulously.

Q1: The experimental has no clinically practical use…The counterpart in medical imaging would be lesion/nodule detection.

A: As indicated by the title, our focus is on the detection of organs while lesions are not considered organs. Organ detection finds numerous applications for efficient data retrieval [1], robust quantification [2], improvement of downstream tasks (like semantic segmentation [3], etc), etc. We plan to allocate a section in the introduction to underscore the significance of organ detection in medical imaging.

[1] Xu et al. "Efficient multiple organ localization in CT image using 3D region proposal network." IEEE transactions on medical imaging. [2] Tong et al. Disease quantification on PET/CT images without explicit object delineation, MedIA. [3] Navarro et al. "A Unified 3D Framework for Organs-at-Risk Localization and Segmentation for Radiation Therapy Planning, EMBC2022.

Q2: Lack of comparison between medical detection pipelines such as nnDetection.

A: nnDetection is embedded in the RetinaNet, we borrowed this technique from Focused Decoder [draft: Wittmann et al., 2023] and reported the results of this well-known technique in the paper! As the authors of Focused Decoder mentioned: “We adopt Retina U-Net from nnDetection with minor modifications..., we extract nnDetection's generated hyperparameters and refine them via a brief additional hyperparameter search…, converting it to a RetinaNet variant.” We apologize for the confusion and we would be happy to clarify it in the revised version. It is also incorrect that we did not compare our method with medical detection pipelines. FocusedDec, RetinaNet, SwinFPN, and Transoar, highlighted in the manuscripts, represent state-of-the-art and contemporary techniques for organ detection.

Q3: RetinaNet performs similarly to our method on Verse despite slight differences in false positives.

A: We believe the reviewer meant "False Negative" instead of "False Positive" and RetinaNet has one false negative, failing to predict the top vertebrae. Detailed results in Supplementary Table 12 show Organ-DETR outperforming RetinaUnet with a 55.1 mAP score compared to 46.3. While we can't include all results in the paper, we've attached visualization snippets in Supplementary Figs. 10-12. As stated, code and pre-trained models will be released for researchers to run and visualize results on their own.

Q4: The authors only consider healthy organ detection.

A: We trained and evaluated the methods on five different CT datasets that included healthy and unhealthy organs. Hence, it is not correct that we only consider healthy organ detection. As discussed previously, this study is devoted to organ detection and does not delve into topics such as lesion detection or distinguishing between healthy and unhealthy states. However, the insights provided in this study can serve as a beneficial reference for researchers in alternative detection domains seeking to enhance their approaches.

Q5: For VerSe, many more landmark localization tools should have been evaluated in terms of localization error in mm.

A: We explicitly explained the preparation process of datasets including Verse in Supplementary: Section B-3. Given that the primary emphasis of this study is on organ/object detection, the metrics employed—mAP, mAR, and AP75—are well-suited for detection tasks. While additional metrics could offer further assessment, our study emphasizes organ detection as a general task, demonstrating its applicability across various CT datasets with distinct functions. As mentioned earlier, we will release the code and pre-trained models, allowing readers to conduct additional measurements as needed.

We trust that the provided responses address the reviewer’s concerns regarding Organ-DETR. We have taken diligent measures to ensure comprehensive coverage of Organ-DETR details within both the manuscript and Supplementary. As the reviewer-author discussion period is scheduled from Nov 10 to Nov 22, we eagerly await any forthcoming questions and will be delighted to offer further clarification during this time.