Pg-GAT: A Complete Graph Model for Cancer Detection and Subtyping in Whole Slide Images Analysis

My major concerns are about the experiments:

• The proposed method cannot achieve SOTA performance on Camelyon 16 and 17 benchmarks. Why not listing the challenge winners in table 1? For example, in Camelyon 16 challenge, the winner (Harvard & MIT) already achieves 99.4% AUC. Also, the SOTA of Camelyon 17 is from DeepBio Inc. For more recent results on the two benchmarks, you can check table 1 in PFA-Scannet [MICCAI 2019]. It is suggested to include a comparison to these top-performing methods and explain how your approach compares in terms of performance and computational efficiency.

• Why not using the challenge metrics for evaluation? For Camelyon16, FROC is a more challenging metric compared with AUC reported in this paper. Also, kappa score should be compared for Camelyon17 benchmark. Please explain why you chose AUC over FROC for Camelyon16 and to provide results using both metrics if possible. Similarly, for Camelyon17, results using the kappa score are requested, which would allow for a more direct comparison to other methods evaluated on this benchmark.

• The tumor localization performance is not satisfying. Even though the proposed method is not prioritized for this task, the gap of CAMIL, around 3% dice, is too large. Please discuss potential reasons for this performance gap in tumor localization, and suggest ways that might improve this aspect of proposed method, even if it's not the primary focus.

• Novelty limitations: The proposed use of Graph Attention Networks (GAT) for WSI-based cancer analysis has precedent in prior studies, reducing the novelty of Pg-GAT in this context[1,4]. DASMIL [1], for example, already uses a GAT layer and message passing to allow interaction along patches on different resolutions before attention pooling.
• Performance Concerns: Results in Tables 1 and 2 do not consistently outperform baselines across all metrics. Specifically, Pg-GAT's tumor localization scores, evaluated by the Dice score, and classification accuracy fall short of fully distinguishing it from competing methods. More relevant confidence-based metrics, such as FROC, are related to localization and Camelyon. Since it is based on confidence, it measures the capabilities to distinguish diseases without any manual threshold.
• Limited Scope in Model Efficiency Analysis: Figure 4, presenting model size versus performance, is restricted to a single dataset, limiting insights into Pg-GAT’s comparative efficiency across different dataset complexities.
• Different backbone: Attempts to reproduce baselines such as GTP and CAMIL resulted in out-of-memory (OOM) issues, so results were drawn from prior publications. However, this approach introduces limitations, as these reported baselines utilize a different backbone (SIMCRL), which could affect comparability and undermine the experimental rigor. That comparison is vital to quantifying how the solution is better than other GNN-based solutions.
• Interpretability Limitations: The interpretability section would benefit from more generalized tumor examples to substantiate that Pg-GAT can adapt to varied tumor complexities.

Minor Comments

• Clarify abbreviations as Acc and Minor typographical errors in section headers and figure captions should be corrected (e.g., "TansMIL" to "TransMIL").
• Several mentioned methods do not appear in the comparison tables as ABMIL, DSMIL, STEMIL, EGT (graph-based)
• Limited Focus in Related Work: The Related Work section centers primarily on GNN-based approaches for WSI analysis rather than providing a balanced discussion that includes broader Multiple Instance Learning (MIL) solutions for WSIs. A more comprehensive review of MIL approaches used in generic WSI settings would better contextualize Pg-GAT within the landscape of WSI analysis techniques. Additionally, CAMIL and GTP are presented as primary graph-based baselines, but they are not the only approaches that employ graphs for WSI representation( EGT, DASMIL[1], H2MIL[2], GDSMIL[3]). Including a discussion on other graph-based MIL solutions would provide a clearer picture of how Pg-GAT fits among similar models and address potential limitations in novelty.
• This manuscript lacks implementation details (lr, scheduler, epochs, batch_size, etc.), which doesn't allow for reproducibility

[1]: A Graph-Based Multi-Scale Approach With Knowledge Distillation for WSI Classification, TMI [2] H^2-MIL: Exploring Hierarchical Representation with Heterogeneous Multiple Instance Learning for Whole Slide Image Analysis [3] Enhancing PFI Prediction with GDS-MIL: A Graph-Based Dual Stream MIL Approach. [4] Whole Slide Cervical Cancer Screening Using Graph Attention Network and Supervised Contrastive Learning

1. The use of spatial graphs and graph attention networks for WSI analysis can't be viewed as contribution in 2024. Several prior graph-based methods for WSIs, such as PatchGCN [1], TEA-Graph [2], and SlideGraph+ [3], already incorporate spatial encoding and graph neural networks for WSI analysis. This prior work is notably missing from the paper’s introduction, overlooking important context for the field.

2. The paper’s critique of earlier methods lacks specificity. It is unclear which prior methods employ spectral graphs in WSI analysis, or why spatial graphs, as implemented here, would not require a large adjacency matrix. Additionally, the claim of “unleashing the potential of attention mechanisms” is vague; graph attention networks (introduced in 2017) [4] have long supported attention mechanisms, raising questions about Pg-GAT’s unique contributions beyond the projection-gated pooling.

3. Evaluating Pg-GAT on a binary cancer classification task does not fully leverage the model’s contextual learning capabilities, as the task could be solved with only one positive patch. A more suitable evaluation might involve context-dependent tasks, such as survival prediction, where broader spatial relationships are essential.

4. The baselines in Table 1 are insufficient, especially with two methods omitted due to out-of-memory errors, leaving only three comparison models (two of which are variants of CLAM). This limited comparison fails to provide a comprehensive benchmark against relevant methods.

5. The unexpectedly strong performance of Mean-GAT in TCGA-NSCLC compared to more established methods like TransMIL, CLAM, and GTP suggests potential issues in evaluation or methodology. This inconsistency should be clarified to ensure the robustness of Pg-GAT’s reported results.

6. Pg-GAT’s low detection performance in Table 2 contradicts its claim of effective tumor localization. This discrepancy undermines the model’s stated contribution of providing interpretability and detection capabilities.

[1]. Chen, R. J., Lu, M. Y., Shaban, M., Chen, C., Chen, T. Y., Williamson, D. F., & Mahmood, F. (2021). Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24 (pp. 339-349). Springer International Publishing.

[2]. Lee, Y., Park, J. H., Oh, S., Shin, K., Sun, J., Jung, M., ... & Kwon, S. (2022). Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning. Nature Biomedical Engineering, 1-15.

[3]. Lu, W., Toss, M., Dawood, M., Rakha, E., Rajpoot, N., & Minhas, F. (2022). SlideGraph+: Whole slide image level graphs to predict HER2 status in breast cancer. Medical Image Analysis, 80, 102486.

[4]. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y. (2017). Graph attention networks. arXiv preprint arXiv:1710.10903.

This work replicates the pipeline established in MUSTANG [1] published at BMVC 2023, without proper acknowledgment or citation. The fundamental proposed architecture and application area - hierarchical GAT layers combined with pooling for WSI analysis - is identical to MUSTANG's published approach.

The only differences are the authors:

• Use a "grid-graph" defined on spatial nearest neighbors, rather than a k-NN graph defined in feature space. The use of spatial nearest neighbor graphs was established in [2], which is not cited either.
• Use of Projection-gated topK pooling [3] instead of SAGPooling [4]. Ablation on this was also carried out in MUSTANG.
• Apply this approach to benchmark CAMELYON and TCGA datasets, rather than the multi-stain dataset used in MUSTANG.
• Apply pos-thoc GNNExplainer[5]

Neither of these differences represents a significant technical contribution for a conference such as ICLR. I therefore recommend rejection based on lack of novelty and proper attribution.

I have raised further concerns regarding this submission with the AC, SAC and PCs.

[1] Gallagher-Syed et al. Multi-Stain Self-Attention Graph Multiple Instance Learning Pipeline for Histopathology Whole Slide Images. BMVC 2023. https://papers.bmvc2023.org/0789.pdf

[2] Chen et al. Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks. MICCAI 2021. https://arxiv.org/abs/2107.13048

[3] Cangea et al. Towards Sparse Hierarchical Graph Classifiers. NeurIPS 2018 Relational Representation Learning Workshop. https://arxiv.org/abs/1811.01287

[4] Lee et al. Self-Attention Graph Pooling. ICML 2019. https://arxiv.org/abs/1904.08082

[5] Ying et al. GNNExplainer: Generating Explanations for Graph Neural Networks. NeurIPS 2019. https://arxiv.org/abs/1903.03894

• Lack of Novelty in Graph Construction: The authors appear to lack familiarity with existing work in WSI analysis. Their proposal of leveraging spatial correlations in WSI graphs is not novel at all. For example, H2MIL[1] constructs a hierarchical graph based on spatial relationships, TEA-Graph[2] establishes spatial graphs while incorporating node similarity for graph construction, and HEAT[3] uses heterogeneous graphs based on node relations. The proposed Pg-GAT method does not clearly demonstrate novelty over these established approaches.

• Insufficient Comparison: Although Pg-GAT is evaluated on benchmark datasets, there is a lack of direct comparison with recent methods in WSI analysis, like DTFD-MIL[4], making it difficult to assess its competitive performance in this context.

• Invalid Code Link: The authors put an invalid code link in their abstract, I think they should upload a valid code link.

• Overclaim of Contribution: The authors argue they offer model interpretability, however, they just directly use well-established tools proposed by other papers.

• Insufficient Evaluation: The authors claim their model is computationally lightweight, however, they did not compare the FLOPs in their experiments.

[1] Hou W, Yu L, Lin C, et al. H^ 2-MIL: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis[C]//Proceedings of the AAAI conference on artificial intelligence. 2022, 36(1): 933-941.

[2] Lee Y, Park J H, Oh S, et al. Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning[J]. Nature Biomedical Engineering, 2022: 1-15.

[3] Chan T H, Cendra F J, Ma L, et al. Histopathology whole slide image analysis with heterogeneous graph representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15661-15670.

[4] Zhang H, Meng Y, Zhao Y, et al. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 18802-18812.