GraphMorph: Tubular Structure Extraction by Morphing Predicted Graphs

Thank you for your constructive suggestions! We will correct the typo error you raised in the revised version. Please find our reply to your questions below.

Q1: Some hyperparameters (e.g thresholds) have to be set. The methodology is not always clear to set these parameters. e.g. are experiments for setting $p_{thresh}$ (lines 560-563 ) done on test sets?

A1: Thanks for the suggestions. Parameters like $\lambda_{\text{class}}=0.2$, $\lambda_{\text{coord}}=0.5$, $\alpha=0.6$, and $\gamma=2$ in the loss function (lines 462) are adopted from established settings in previous works, such as Deformable DETR[1]. We adapted $\alpha=0.75$ specifically for the MassRoad dataset to better handle the sparse nature of road networks. Parameters like learning rate and weight decay were set in line with those used in Pointscatter[2] to ensure a fair comparison. For $p_{thresh}$, we empirically set it to 0.5 for all experiments on the four used datasets. In Table 9 of Appendix, we compared it to other thresholds on the test set of DRIVE dataset and found the values close to 0.5 yielded better results.

References:

[1] Xizhou Zhu, et al. "Deformable detr: Deformable transformers for end-to-end object detection." arXiv preprint arXiv:2010.04159, 2020.

[2] Dong Wang, et al. "Pointscatter: Point set representation for tubular structure extraction." European Conference on Computer Vision, pages 366–383. Springer, 2022.

Thank you for your constructive feedback! We will answer your questions below.

Q1: For the performance mentioned in the tables, the authors should provide standard deviation to understand whether their method is statistically significant or not. The authors should conduct t-test to confirm this.

A1: Thanks for your suggestion. We have provided error bars in our rebuttal pdf (detailed in Table 1). As can be seen from the results, most improvements of our method over baselines are significant. We will definitely include error bars for all experimental results in the revised version of the paper.

Regarding the t-test, we demonstrate the comparison of segmentation performance on the PARSE and ISBI12 datasets based on UNet, and the results are shown in the following table. PARSE is a 3D pulmonary artery dataset, and we show the results of GraphMorph's experiments on it in the rebuttal pdf.

The statistical results from our t-tests clearly demonstrate that GraphMorph significantly improves segmentation performance on both 2D and 3D datasets, as evidenced by the significant p-values across all metrics for the PARSE and ISBI12 datasets. Notably, while improvements in volumetric scores such as Dice and clDice are significant, the most striking results are observed in the topological metrics ($\beta_0$ error, $\beta_1$ error, and $\chi$ error), where our method consistently demonstrates substantial enhancements. These results underline the capability of GraphMorph to effectively leverage branch-level features to optimize the extraction of tubular structures' topology.

Q2: The authors should provide the computational complexity of their method with respect to the baselines. The complexity should be about the training and inference time of their method. I suspect the algorithm is slow due to iterating over each edge.

A2: We appreciate the request for detailed computational complexity comparisons. Below is a table that outlines the resource utilization during the training process on the DRIVE dataset with a UNet backbone.

The increase in parameters and FLOPs in our approach primarily stems from the integration of the Graph Decoder featuring a DETR module. This component is crucial for predicting accurate topological structures of the graphs. Advancements in transformer architectures that reduce computational overhead could potentially enhance the efficiency of our model during training. Implementations such as Lite-DETR [1], noted for their efficiency in handling transformers, could serve as alternatives to the current DETR module, potentially reducing training time and computational resource usage.

As detailed in Appendix F, the inference time analysis has been conducted in Table 5. The results show that the Morph Module is the most time-consuming part during inference process. However, the iterative execution over each edge using the SkeletonDijkstra algorithm is not the primary contributor to this time consumption. This is because the algorithm operates on much smaller patches. For example, a $384 \times 384$ image is divided into numerous $32 \times 32$ patches. Due to the limited size, running SkeletonDijkstra on one patch is highly efficient (about 0.01s per patch). The primary time expenditure currently arises from processing these patches sequentially. However, as each patch operates independently, there is significant potential to enhance efficiency by parallelizing the computations of all patches. Recognizing this opportunity, we plan to focus future work on optimizing the Morph Module by implementing parallel processing techniques to accelerate inference. Additionally, your question has inspired us to explore the parallelization of edge operations within each patch, which may further reduce time cost. We intend to detail these improvements in revised version of our paper and believe this will substantially elevate the operational efficiency of our model.

References:

[1] Li, Feng, et al. "Lite detr: An interleaved multi-scale encoder for efficient detr." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

Thank you for your detailed feedback! Please see our answers for your questions below.

Q1: Is the threshold in the SkeletonDijkstra algorithm dataset-dependent?

A1: $p_{thresh}$ is a hyper-parameter which is dataset-independent. It is set to 0.5 as the default setting across all experiments. We will add this setting to the "Implementation Details" in the revised version.

Q2: Is the segmentation network trained simultaneously with the graph decoder or one before the other?

A2: The segmentation network and the graph decoder are trained simultaneously to optimize both components effectively and enhance overall training efficiency. This joint training approach allows the segmentation network to leverage the branch-level features provided by the graph decoder, leading to more coherent and accurate predictions of tubular structures.

Q3: Are there any metrics computed on the extracted centerlines?

A3: In addressing the metrics for evaluating the quality of extracted centerlines, Table 1 in our main paper presents a detailed breakdown of the metrics. These metrics are calculated by comparing the centerline masks predicted by our model with the ground truth masks, detailed below:

1. Volumetric Scores: The first three columns of Table 1 focus on volumetric scores which assess different aspects of quality of extracted centerlines:

   • Dice evaluates the overlap between the predicted centerline and the ground truth. A higher Dice score indicates better alignment between the predicted centerline and its corresponding label.

   • AUC and ACC provide a comprehensive evaluation of the model’s ability to classify pixels correctly, indicating the model's performance in distinguishing between the centerline of tubular structures and the background.

2. Topological Metrics: The last three columns of Table 1 introduce topological metrics, which indicate accurate representation of topology:

   • $\beta_0$ error measures the error in the number of connected components of the extracted centerline. Predicted broken or redundant vessels will cause this metric to rise.
   • $\beta_1$ error assesses the accuracy in the number of loops or cycles within the centerline, which is effective for evaluating complex vascular topology.
   • Euler characteristic error ($\chi$​ error) combines the first two topological aspects, providing a holistic view of the topological accuracy.

Metrics like Dice, ACC, $\beta_0$ error, $\beta_1$ error, and $\chi$ error are also utilized in Table 2 and Table 3 of our main paper, which evaluate the performance of our model on the segmentation task. By using share metrics across different contexts, we establish a consistent framework for evaluating our approach against both tasks.

Q4: I was wondering whether the morph module could also be used during training to correct some segmentation errors.

A4: Currently, the morph module is designed to operate during the inference phase and does not propagate gradients, which precludes its direct use in training the segmentation network. However, the patterns of connectivity it identifies have significant potential to refine the training process.

Recognizing this potential, the possibility of modifying the morph module to enhance error correction during training is promising. One feasible approach could involve developing a differentiable version of the morph module that can be integrated into the back propagation process. Another one is to enhance the training process by assigning increased weights to path points identified by the morph module as crucial for connectivity. Such developments could enhance the robustness and accuracy of segmentation networks, and we plan to pursue this direction in our future work.

Thank you for your constructive feedback! We will respond to your concerns below.

Q1: The SkeletonDijkstra Algorithm is proposed by authors alone, any reference?

A1: The SkeletonDijkstra algorithm proposed by our paper represents an innovative adaptation of the classic Dijkstra's algorithm, specifically tailored to the unique challenges of tracing centerlines in tubular structures. This adaptation was necessary to handle the specific topology and morphology of tubular structures that are not addressed by traditional shortest path algorithms. While we have no external references for this particular adaptation, our extensive experiments in the paper confirm its efficacy and accuracy.

Q2: The datasets are pretty small and have been extensively tested in the past.

A2: Thanks for the comments. In medical imaging, especially for tubular structure analysis, large-scale public datasets are scarce due to privacy concerns and data collection complexities. The datasets we selected, while smaller relative to those used in broader computer vision tasks, are established benchmarks within the medical imaging community. Despite their frequent use in previous studies, these datasets still present unresolved challenges [1] [2], especially related to topological errors that happen regularly, which is derived from under-utilisation of branch-level features of tubular structures. Our approach mitigates this problem through the efficient exploitation of branch-level features.

To demonstrate the broader applicability of our approach, we have conducted experiments on the Massachusetts Roads dataset with 1171 aerial images in the paper. Moreover, we provided further validations with 3D CT scans in our rebuttal pdf. These results not only demonstrate the method's effectiveness but also its potential in more clinically relevant, real-world settings.

Q3: All examples are primarily 2D patches. The effectiveness of the method on 3D data needs to be validated.

A3: We recognize the need to validate our method, GraphMorph, on 3D datasets for its clinical relevance. Thus, we have extended GraphMorph to use the 3D UNet architecture and tested it on the pulmonary arterial vascular segmentation dataset from the PARSE challenge [3], which includes 100 annotated 3D CT scans. These cases were divided in a 7:1:2 ratio for training, validation, and testing.

The preliminary results, as detailed in Table 2 of our rebuttal pdf, show that our method consistently outperforms existing baselines across all metrics, mirroring the success we observed with 2D data. This alignment between 2D and 3D results not only underlines the effectiveness of our method but also its adaptability to 3D vessel segmentation task. We plan to apply our method to more diverse 3D medical datasets to further validate the effetiveness of our method.

Q4: The numerical improvements reported in the paper, overall are marginal.

A4: Thanks for the comments. Our method, GraphMorph, is specifically designed to enhance the accuracy of topological features in tubular structure extraction, which is a critical aspect often overlooked in most traditional pixel-level frameworks. The focus of our method on topological accuracy is reflected in the significant improvements in topological metrics, such as the reduction of $\beta_0$ error by 25% to 45% as reported in Table 2 of our paper.

While the improvements in volumetric metrics like Dice and clDice are relatively modest (approximately 1%), the advancements in topological integrity are significant. Accurately restoring topology is both challenging and crucial in tubular structure extraction tasks. The greatly reduced topology errors are derived from our direct exploitation of branch-level features. This can also be illustrated by our experiments on the 3D pulmonary artery dataset PARSE.

Q5: On smaller datasets for evaluation, error bars or std should also be reported.

A5: Thanks for the suggestion. We have provided error bars in our rebuttal pdf (detailed in Table 1). As can be seen from the results, most improvements of our method over baselines are significant. We will definitely include error bars for all experimental results in the revised version of the paper.

Q6: "Ultimately, our research not only advances technical knowledge in medical image analysis, but also holds potential for significant positive impacts in healthcare efficiency and patient care." These claims are significantly over-claimed.

A6: We appreciate your feedback on our claims regarding the broader impacts of GraphMorph. Our results on 2D datasets demonstrate the clinical potential of our method for 2D tubular structure extraction. However, we acknowledge that its performance in complex 3D vascular tasks was not fully explored in the initial submission. In our rebuttal pdf, we have included preliminary results from applying GraphMorph to 3D pulmonary artery segmentation, where we observed obvious improvements.

Moving forward, we plan to extend our evaluations to more diverse 3D datasets to further validate and demonstrate the clinical applicability of our method. We will also ensure that our claims will be carefully measured to reflect the current state and future potential of our research accurately in the revised version.

References:

[1] Hu, Xiaoling, et al. "Topology-preserving deep image segmentation." Advances in neural information processing systems 32 (2019).

[2] Dong Wang, et al. "Pointscatter: Point set representation for tubular structure extraction." European Conference on Computer Vision, pages 366–383. Springer, 2022.

[3] Luo, et al. "Efficient automatic segmentation for multi-level pulmonary arteries: The PARSE challenge." arXiv preprint arXiv:2304.03708, 2023.

Dear Reviewer Ysem,

Thank you for the time and effort you've invested in reviewing our paper. We have carefully responded to each of your comments and questions. As the deadline for the author-reviewer discussion is nearing, we would greatly appreciate it if you could kindly take a look at our responses and provide your valuable feedback. We are fully prepared to engage in further discussions should you have any additional concerns or suggestions.

Thank you once again and we greatly appreciate your contributions to refining our work and eagerly await your re-evaluation.

Paper 13902 Authors

This paper provided some clever technical novelty to improve topology constrained tubular structure extraction. From a paper perspective, this is a reasonably all-round paper that has an interesting idea and consistent empirical performance improvement. For a NeurIPS paper, this is probably ok.

However authors should not over claim too much on the clinical impact side of business. From what I can see, this paper has very limited utility in real clinical applications. From the submitted version, it has not been rigorously tested. With rebuttal, it is on some quite limited public datasets which say very little on real clinical application complexity. You would need at least participate in some well known medical imaging public challenges and achieve some encouraging results to start with.

https://atm22.grand-challenge.org/