Towards Generalizable Retina Vessel Segmentation with Deformable Graph Priors

Thank you for your constructive comments and kind support! All your concerns have been carefully addressed as below. The manuscript will be revised accordingly. We sincerely hope our responses fully address your questions.

W1(a) Preprocessing Dependency: … the extent of manual intervention in these steps is not clearly quantified…

A1(a): The graph atlas construction is only performed once on the CHASE training set before training. To evaluate the robustness, we use the graph prior from the CHASE to train GraphSeg on the DRIVE dataset. The results show: 1) GraphSeg(trained) outperforms UNet(trained) due to the graph prior, which demonstrates that the graph prior extracted from one dataset can be transferred to the other dataset. 2) GraphSeg outperforms UNet(trained) on most metrics, which confirms the generalizability of GraphSeg and demonstrates the robustness of GraphSeg to graph priors

W1(bc) Cross-Dataset Evaluation. … additional datasets and Comparison Methods…

A1(bc): We provide additional results by including more datasets and additional seven methods, i.e., AttUNet[3], AGNet[4], ConvUNeXt[5], DCSAU-Net[6], R2UNet[7], SAUNet[8], CDARL[9]. FRSGNet[1] is current SOTA for retinal vessel segmentation

GraphSeg outperforms other methods in generalization studies (on DRIVE and HRF), especially for F1 and Soft Dice. For other methods, since the F1 scores and soft Dice scores are significantly lower, the acc, sensitivity and specificity are not very meaningful due to the significant class imbalance between the foreground (vessels) and background

The test results on the new STARE dataset show that GraphSeg outperforms other methods in soft Dice, F1, and Sensitivity, further confirming the superior generalizability of GraphSeg

[1][ArXiv’25] Full-scale Representation Guided Network for Retinal Vessel Segmentation

[2][BOE’24] a full-resolution dilated convolution network for retinal vessel segmentation

[3][MIDL’18] Learning where to look for the pancreas

[4][ISAIMS’20] Retinal Blood Vessel Segmentation via Attention Gate Network

[5][KBS’22] An efficient convolution neural network for medical image segmentation

[6][CBM’23] A deeper and more compact split-attention U-Net for medical image segmentation

[7][JCAA’24] Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation

[8][MICCAI’20] Shape attentive u-net for interpretable medical image segmentation

[9][MIA’24] Contrastive diffusion adversarial representation learning for label-free blood vessel segmentation

W2(a) Definition of the variable $d$ in Eq (1).

A2(a): The variable $d$ represents the landmark detection variable that identifies the points where vascular structures exhibit significant geometric dissimilarity. These are typically the branching points or bifurcations of blood vessels, which is firstly introduced in line 153

The variable $d$ follows a Gamma prior distribution, and its value is defined within the context of the model’s optimization to help guide the detection of landmarks by enforcing structural constraints during the vascular structure matching

W2(b) Comparison with BayeSeg: A more detailed comparison would strengthen the contribution’s novelty

A2(b): The key technical differences between GraphSeg and BayeSeg as follows:

• Graph-based Prior: Comparing to BayeSeg, GraphSeg introduces a deformable retinal graph prior that allows for better modeling of vascular structures with topological flexibility. The deformable graph prior, adapting to different vascular morphologies, is a key distinction that enhances the robustness and generalizability of GraphSeg across domains, as demonstrated in the ablation study
• Anatomical Consistency: BayeSeg focuses on shape-based decomposition without explicit structural alignment to a prior, while GraphSeg integrates anatomical constraints through the deformable graph prior, leading to better vessel structure continuity
• Domain Generalization: By jointly modeling anatomical topology and image structure, GraphSeg performs better in unseen domains, as evidenced by experiments on multiple retinal datasets

Notably, the deformable graph priors are derived from a shared retinal atlas, enabling the modeling of dependencies between vascular structures across different retinal images. This fundamentally distinguishes GraphSeg from conventional models (e.g., BayeSeg), which operate independently on each image under the i.i.d. assumption, thus neglecting cross-sample anatomical dependencies.

W2(c) Lack of Code Availability.

A2(c): We are committed to making the entire code publicly available upon acceptance of the paper. The pre-processing strictly follows [1] by performing their code on the CHASE dataset.

[1] PAMI’24. Statistical analysis of complex shape graphs.

Q3. Is GraphSeg the first method to use a retinal atlas or Bayesian framework for retinal vessel segmentation? how does GraphSeg differ from others?

A3. To our knowledge, GraphSeg is the first method to combine a deformable graph prior with a Bayesian framework for retinal vessel segmentation. While prior works have explored Bayesian frameworks and anatomical priors separately, GraphSeg introduces a novel approach by using deformable retinal graph priors that adapt to various vascular structures across datasets. This is a unique contribution that distinguishes GraphSeg from previous methods.

Previous works like BayeSeg have employed Bayesian frameworks for segmentation but only focused on compact anatomies such as heart and prostate. Similarly, graph-based approaches have been used in CT vessel segmentation [1], but they tried to construct image-specific graphs from the first-stage segmentations to further refine segmentations by graph convolutional networks, which is different with our image-agnostic deformable graph priors constructed from retinal Atlas for informing vessel topology.

[1] Graph Convolution Based Cross-Network Multi-Scale Feature Fusion for Deep Vessel Segmentation

Q4. ...how consistent is the distribution of the constructed retinal atlas... the generalization of the constructed retinal atlas...

A4. The distribution of vessel graphs are different as shown in Figure 3 in appendix. However, since our graphs are deformable with $\delta$ for capturing displacement, the graph atlas extracted from CHASE can be transferred to other datasets. To evaluate the generalizability, we use the graph prior from the CHASE to train GraphSeg on the DRIVE dataset. The results and explanation are shown in the response A1(a) to W1(a). The results show that the graph prior extracted from one dataset can actually be used for the other dataset.

Q5. ...computational complexity...

A5. We provide the computational cost comparison as follows. The computational costs are comparable for GraphSeg to other methods. In the inference stage, we only need to run the Decomposition (green in Figure 2) and Segmentation (blue in Figure 2) parts. Therefore, the computational costs are comparable. Besides, without the Deformable graph prior, GraphSeg is downgraded to BayeSeg, thus the computational costs are almost the same for GraphSeg and BayeSeg for inference. We trained GraphSeg for 2.8 hours on GTX 4090. Since each method uses different training settings, it is not directly comparable. Therefore, we focus on inference metrics which are more relevant for practice.

Thank you for your comments! All your concerns have been carefully addressed as below. The manuscript will be carefully revised accordingly. We sincerely hope our responses fully address your questions.

W1.1 The experiments may not fully support the claimed superiority of the method. There is a lack of repeated trials to assess consistency and robustness.

A1.1. Thanks for your comment. The improvement of GraphSeg in generalizability study is significant. We also perform a statistical test as follows. GraphSeg outperforms BayeSeg significantly as shown in the table

• CHASE DB

• DRIVE

• HRF

W.1.2 Scale of Data: The dataset used is relatively small, which may limit the generalizability of the findings. The cross-dataset generalization experiments are limited and not comprehensive enough to confirm the method’s robustness.

A1.2. Thank you for your comment. The three datasets used in this work (DRIVE, HRF, and CHASE) are widely recognized as standard benchmarks for retinal vessel segmentation [1]. To further evaluate the generalizability of GraphSeg, we have also conducted experiments on the STARE dataset and compared with seven additional methods, including AttUNet [6], AGNet [7], ConvUNeXt [2], DCSAU-Net [3], R2UNet [4], SAUNet [8], and CDARL [9]. All models were trained on the CHASE dataset, and evaluated by ACC, Soft Dice, F1 score, Sensitivity, and Specificity. Note that Soft Dice, F1 score, Sensitivity are more appropriate to evaluate vessel segmentation performance due to the significant class imbalance between the foreground (vessels) and background. From the test results on STARE one can see that GraphSeg outperforms other methods in soft Dice, F1 score, and Sensitivity, further confirming the superior generalizability of GraphSeg.

• CHASE (train)

• DRIVE

• HRF

• STARE

[1] [ArXiv’25] FSG-Net: Full-scale Representation Guided Network for Retinal Vessel Segmentation

[2] [Knowledge-based systems’22] ConvUNeXt: An efficient convolution neural network for medical image segmentation

[3] [Computers in Biology and Medicine’23] DCSAU-Net: A deeper and more compact split-attention U-Net for medical image segmentation

[4] [Journal of Computational Analysis & Applications’24] Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation

[5] [Biomedical Optics Express’24] FRD-Net: a full-resolution dilated convolution network for retinal vessel segmentation

[6] [Medical Imaging with Deep Learning’18] Attention u-net: Learning where to look for the pancreas

[7] [ISAIMS’20] AGNet：Retinal Blood Vessel Segmentation via Attention Gate Network

[8] [MICCAI’20] Saunet: Shape attentive u-net for interpretable medical image segmentation

[9] [MedIA'24] C-DARL: Contrastive diffusion adversarial representation learning for label-free blood vessel segmentation

W2. Originality… The limitation in experiments makes it difficult to judge the reliability and practical value of the method…

A2. Thanks for your concern. We provide additional results as shown in the table above. Experiments show the superiority of our GraphSeg. We would like to clarify the originality of our contributions. Specifically, our work introduces: (1) a novel Bayesian retinal vessel segmentation framework that goes beyond traditional methods focusing on compact shapes; (2) the development of deformable retinal graph priors for aligning heterogeneous vascular structures in retinal images, along with a statistical model to drive the alignment between image vascular structures and these graph priors; and (3) extensive validation of our framework across multiple widely used datasets, demonstrating its superior generalizability in unseen scenarios.

Q3. Could you please clarify how the retinal vascular graph atlas is initially obtained?

A3. Thanks for your concern. We strictly follow [1] to construct the graph Atlas as we described in Appendix.A.1

[1] PAMI’24. Statistical analysis of complex shape graphs.

Q4. …whether pre-processing and decomposition were applied consistently across all compared methods…

A4. Thank you for your valuable comment. In our cross-dataset evaluation, we ensured consistency in the pre-processing steps across all compared methods to provide a fair comparison. Specifically, the same pre-processing pipeline (including image resizing, normalization, and augmentation) was applied uniformly to all methods, including the proposed GraphSeg approach.

For the decomposition step, which is specific to GraphSeg and BayeSeg, we only applied it to these two methods and not to the baseline approaches. We will clarify this point in the revised manuscript and emphasize that while pre-processing was consistent, the combination of decomposition and deformable graph priors were unique to GraphSeg. It’s worth to mention that the decomposition and deformable graph priors in GraphSeg are trained end-to-end instead of stage-by-stage.

Q5. … To better validate the performance and robustness of the proposed model, it would be important to include more recent and larger-scale datasets….

A5. Please refer to A1.2.

Q6. In Table 3, sensitivity increases significantly with the addition of more components. Could you explain the reason behind this?

A6. Thank you for your insightful question. The significant increase in sensitivity observed with the addition of more components in Table 3 is attributed to the model’s ability to capture finer details of the vascular structures, particularly in challenging cases where small or faint vessel segments are present.

As more components are introduced, the model becomes better at distinguishing between foreground vascular structures and the background, improving its sensitivity to vessels that might otherwise be overlooked. This is especially beneficial in scenarios with complex or noisy images, where the model needs to identify subtle vascular features.

Thank you for your reply! I recognize the authors fully address my concerns on Originality and partially address my concerns on Quality. I will raise my score, and I would appreciate the authors considering the following suggestions.

By pointing out Q1 (Q3 in rebuttal with authors' answer A.3), I was expecting there would be a specific design for retina images, as the retina is of a different imaging domain from conventional graphics, and the authors claim this as their first contribution. I understand the authors did make some adjustments to the Graph Atlas, but these parameter-level adjustments are still under the original scope of the existing Graph Atlas method. However, I still appreciate this effort and encourage the authors to include the corresponding paragraph in the main paper, as it is an important component.

For the Q2 (Q4 in rebuttal with answers A4), I have queried from the "evaluation" perspective, to make sure pre-processing will help evaluate generalisability rather than hinder it. Specifically, if the preprocessing aligns the input distribution more closely with the domain assumptions of the proposed model, it can lead to artificially improved generalization performance compared to other methods. This could misrepresent the true robustness of the approach in out-of-domain settings. I strongly encourage the authors to describe their preprocessing pipeline in greater detail. It is important, as the authors claimed generalization in the title and the third contribution.

For Q3 (Q5 in rebuttal with answers A5 and A1.2), STARE is also a widely used dataset. It initially had very few images, eventually reaching 400 images. The authors should clearly specify which version or how much data they used in their experiments. I understand the data scarcity and will not insist that authors include more datasets. However, I still encourage the authors to test their results on additional datasets to evaluate generalization beyond segmentation. If the authors' experiments cover a wider dataset for generalization, I may further raise the score.

For Q4 (Q6 in rebuttal with A6), I recommend further discussion in the paper, as the analysis of experimental results is necessary.

Thank you for your constructive comments and kind support! All your concerns have been carefully addressed as below. The manuscript will be carefully revised accordingly. We sincerely hope our responses fully address your questions.

W1. Does not distinguish veins and arteries

A1. We acknowledge the importance of distinguishing between veins and arteries in retinal vessel segmentation, especially for clinical applications. While our approach focuses on general vascular segmentation, we recognize that specialized methods differentiating veins from arteries, such as those proposed by Zhou et al., may offer additional clinical value. We appreciate the reviewer pointing out this gap, and we will include a discussion on this in the revised manuscript, highlighting the potential for extending our method to vein/artery differentiation. We will also reference Zhou et al.’s work in the related work section to clarify how their method differs from ours and discuss the trade-offs between our generalizable approach and more specialized methods.

W2&Q3. Does not consider the effect of pathology in the experimental evaluation. Does the presence of pathology affect performance? An algorithm that works on normal cases (the majority) but fails in the presence of pathology has little clinical utility.

A2. Thank you for your insightful comment. The HRF dataset includes both healthy and pathological images (e.g., diabetic retinopathy), and the DRIVE and CHASE datasets predominantly contain healthy images, with only minor pathological cases. GraphSeg performs well across all three datasets, particularly with respect to generalizability. Although our current experiments focus on generalization across different retinal datasets, we agree that pathology could significantly impact segmentation performance.

Thank you for your comments! All your concerns have been carefully addressed as below. The manuscript will be carefully revised accordingly. We sincerely hope our responses fully address your questions.

Q1. It is uncertain whether the proposed method shows state-of-the-art performance when comparing the model to recent vessel segmentation methods.

A1. Thanks for your suggestion. Following your suggestion, we compare GraphSeg with additional seven methods, including AttUNet [6], AGNet [7], ConvUNeXt [2], DCSAU-Net [3], R2UNet [4], SAUNet [8], and CDARL [9]. FRSGNet[1] is current SOTA for retinal vessel segmentation. We tried to compare with the suggested methods. However, the code for HRD-Net is not available, and the code for DPL-GTF-EFA cannot be executed properly. Finally, we provide two experimental results of CDARL. One was tested using the official checkpoint, the other was reproduced on CHASE.

GraphSeg outperforms other methods in model generalization studies (on DRIVE and HRF), especially for F1 and Soft Dice. For other methods, since the F1 scores and soft Dice scores are significantly lower on DRIVE and HRF, the acc, sensitivity and specificity are not very meaningful due to the significant class imbalance between the foreground (vessels) and background.

• CHASE (train)

• DRIVE

• HRF

[1] [ArXiv’25] FSG-Net: Full-scale Representation Guided Network for Retinal Vessel Segmentation

[2] [Knowledge-based systems’22] ConvUNeXt: An efficient convolution neural network for medical image segmentation

[3] [Computers in Biology and Medicine’23] DCSAU-Net: A deeper and more compact split-attention U-Net for medical image segmentation

[4] [Journal of Computational Analysis & Applications’24] Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation

[5] [Biomedical Optics Express’24] FRD-Net: a full-resolution dilated convolution network for retinal vessel segmentation

[6] [Medical Imaging with Deep Learning’18] Attention u-net: Learning where to look for the pancreas

[7] [ISAIMS’20] AGNet：Retinal Blood Vessel Segmentation via Attention Gate Network

[8] [MICCAI’20] Saunet: Shape attentive u-net for interpretable medical image segmentation

[9] [MedIA'24] C-DARL: Contrastive diffusion adversarial representation learning for label-free blood vessel segmentation

Q2. … requires a higher computational cost to perform vascular structure matching in the inference stage… Please provide the computational cost comparison.

A2. Thanks for your suggestion, we provide the computational cost comparison as follows. The computational costs are comparable for GraphSeg to other methods. In the inference stage, we only need to run the Decomposition (green in Figure 2) and Segmentation (blue in Figure 2) parts. Therefore, the computational costs are comparable. Besides, without the Deformable graph prior, GraphSeg is downgraded to BayeSeg, thus the computational costs are almost the same for GraphSeg and BayeSeg for inference. We trained GraphSeg for 2.8 hours on GTX 4090. Since each method uses different training settings, it is not directly comparable. Therefore, we focus on inference metrics which are more relevant for practice.

Q3. The performance improvement of the proposed method in the generalizability study is quite marginal. Please perform a statistical test to see whether the performance of the proposed method is statistically different from the baseline models.

A3. Thanks for your comment. Following your suggestions, we perform a statistical test between GraphSeg and BayeSeg as follows. GraphSeg outperforms BayeSeg significantly, as shown in the table. For other methods, their F1 scores and soft Dice scores in the model generalization studies are lower than that of BayeSeg, and thus they also have significant difference with GraphSeg in generalizability.

• CHASE DB

• DRIVE

• HRF

Thank you for your constructive comments and kind support! All your concerns have been carefully addressed as below. The manuscript will be carefully revised accordingly. We sincerely hope our responses fully address your questions.

Q1. The paper lacks a comprehensive analysis of the computational overhead introduced by the variational Bayesian framework and deformable graph alignment.

A1. Thanks for your suggestion, we provide the computational cost comparison as follows. The computational costs are comparable for GraphSeg to other methods. In the inference stage, we only need to run the Decomposition (green in Figure 2) and Segmentation (blue in Figure 2) parts. Therefore, the computational costs are comparable. Besides, without the Deformable graph prior, GraphSeg is downgraded to BayeSeg, thus the computational costs are almost the same for GraphSeg and BayeSeg for inference. We trained GraphSeg for 2.8 hours on GTX 4090. Since each method uses different training settings, it is not directly comparable. Therefore, we focus on inference metrics, which are more relevant for practice.

Q2. How sensitive is the method to the quality and representativeness of the statistical retinal atlas used for graph prior construction?

A2. Thanks for your insightful question. In this work, the deformable graph prior provides a guidance for the Decomposition part to extract the graph-like structures for subsequent segmentation. As shown in Figure 4-6, the vascular shape $s$ extracted from the Decomposition is graph-like and the other parts, like color and shadow, are decomposed into the $m$. This is the source of our GraphSeg’s generalizability.

The graph construction follows [1]. Graph Atlas is obtained by performing their code on the training set of CHASE. Following your suggestion, we also use the graph prior from the CHASE to train GraphSeg on the DRIVE dataset. The results are shown as follows: (1) GraphSeg(trained) outperforms UNet(trained) due to the graph prior, which demonstrates that the graph prior extracted from one dataset can be transferred to the other dataset. (2) GraphSeg outperforms UNet(trained) on most metrics, which confirms the generalizability of GraphSeg and demonstrates the robustness of our framework with respect to graph priors.

• DRIVE (train)

• CHASE

• HRF

[1] PAMI’24. Statistical analysis of complex shape graphs.

Q3. The paper appears to lack a comprehensive comparison with recent end-to-end deep learning approaches for vessel segmentation (e.g., U-Net variants, Transformer-based methods, recent CNN architectures).

A3. Thanks for your suggestion. Following your suggestion, we compare GraphSeg with seven additional methods, including AttUNet [6], AGNet [7], ConvUNeXt [2], DCSAU-Net [3], R2UNet [4], SAUNet [8], and CDARL [9]. FRSGNet[1] is the current SOTA for retinal vessel segmentation.

GraphSeg outperforms other methods in model generalization studies (on DRIVE and HRF), especially for F1 and Soft Dice. For other methods, since the F1 scores and soft Dice scores are significantly lower on DRIVE and HRF, the acc, sensitivity, and specificity are not very meaningful due to the significant class imbalance between the foreground (vessels) and background.

• CHASE (train)

• DRIVE

• HRF

[1] [ArXiv’25] FSG-Net: Full-scale Representation Guided Network for Retinal Vessel Segmentation

[2] [Knowledge-based systems’22] ConvUNeXt: An efficient convolution neural network for medical image segmentation

[3] [Computers in Biology and Medicine’23] DCSAU-Net: A deeper and more compact split-attention U-Net for medical image segmentation

[4] [Journal of Computational Analysis & Applications’24] Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation

[5] [Biomedical Optics Express’24] FRD-Net: a full-resolution dilated convolution network for retinal vessel segmentation

[6] [Medical Imaging with Deep Learning’18] Attention u-net: Learning where to look for the pancreas

[7] [ISAIMS’20] AGNet：Retinal Blood Vessel Segmentation via Attention Gate Network

[8] [MICCAI’20] Saunet: Shape attentive u-net for interpretable medical image segmentation

[9] [MedIA'24] C-DARL: Contrastive diffusion adversarial representation learning for label-free blood vessel segmentation