Robust Graph Attention for Graph Adversarial Attacks: An Information Bottleneck Inspired Approach

1. Limited Novelty: The model architecture is inspired by the Information Bottleneck, which I believe is similar in motivation to other IB-based models and not sufficiently novel. The only contribution of this method is actually the gradient decent of attention matrix B in Theorem 3.1. Although the authors highlight differences between RGA-IB and other IB-based models, the distinction between local and global dependencies contributing to improved robustness is not entirely convincing. The robustness improvement might result from implementation details and hyperparameter tuning.

2. Evaluation: The experiments are not comprehensive enough.

• Baselines: How does the model compare to RUNG [1], ElasticGNN [2], SoftMedian [3], and TWIRLS [4]?
• Datasets: How does the model perform on heterophilic datasets?
• Attacks: How does the model perform under gradient-based local attacks and injection attacks?

[1] Robust Graph Neural Networks via Unbiased Aggregation

[2] Elastic Graph Neural Networks

[3] Robustness of Graph Neural Networks at Scale

[4] Graph Neural Networks Inspired by Classical Iterative Algorithms

3. The evidence of the connection between lower IB loss and improved robustness needs further development. Since GIB, UGRL, and RG-GIB are all IB-based models, why is RGA-IB more effective in reducing IB loss? Additionally, in Table 5, the authors should include these baselines to demonstrate that RGA-IB is more effective in reducing IB loss.

4. Minor suggestions: The presentation could be further enhanced. The method section appears somewhat disorganized, and the detailed computation of IB loss should be the main focus of this section. Equation (1) represents the gradient descent of matrix B and should not be formatted as a theorem. Additionally, Algorithm 1 lacks informativeness yet occupies excessive space. The experiment table's font is too small, making it inconsistent with the main text. In the experiment section, the authors should use a dedicated subsection for experimental settings to clearly present datasets, baselines, implementation details, and attacks, rather than introducing these aspects randomly within the main discussion.

1. The paper lacks details on computational resources and does not include comparisons related to memory usage or runtime, which are essential metrics for a comprehensive evaluation.

2. From an attacker’s perspective, the attacker may propose an adaptive attack against your proposed method. But this paper lacks exploration in this area.

3. This paper lacks comprehensive ablation studies. For example, Section 4.1 describes experimental settings such as the proportion of nodes used for training and the number of training epochs, which could also benefit from detailed ablation studies to better understand their impact on the model's performance.

4. The method introduces additional complexity in calculating attention weights and might cause higher computational costs, especially on large graphs. So perhaps this method is not feasible in real world applications.

This paper has the following Weaknesses:

• This research focuses on attention-based graph neural networks which might be a little narrow.
• The attack methods used in this paper are old. For the untargeted attack, the authors only used meta attack which was published in 2019. For the targeted attack, the authors only used a method that was published in 2018. I strongly suggest the authors add more and also some newer attack methods. For example, for untargeted attacks, the authors can also use this method [1]. For untargeted attacks, the authors can also use these methods [2,3]. Only one old attack method for each category is not convincing enough to prove the robustness is related to IB loss.
• The scalability of the proposed method is not clear. What about the running time of RGA-IB and GPU memory needed compared with other robust GNNs used in the paper such as Pro-GNN and Difformer? It is recommended to have a part to discuss this. The datasets used are relatively small. The authors can consider conducting some experiments on large-scale datasets such as OGB-arxiv [4].

References
[1] Scalable attack on graph data by injecting vicious nodes. Data Min Knowl Disc 34, 1363–1389 (2020).
[2] Adversarial Attacks on Graph Neural Networks via Node Injections: A Hierarchical Reinforcement Learning Approach. In WWW (2020).
[3] Robustness of Graph Neural Networks at Scale. In NeurIPS (2021).
[4] Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS.

1. The major concern with this framework is its time complexity; however, there is no time complexity analysis throughout the paper.
2. The experiments mainly focus on small graph datasets. I would suggest the authors conduct more experiments on larger datasets.
3. Intuitively, lower IB loss would lead to lower accuracy on a clean graph dataset. However, as I noticed in Tables 1, 2, and 3, RGA-IB consistently achieves the best result in the clean dataset setting. Could the authors provide an explanation for this phenomenon?
4. I would suggest the authors add more detailed descriptions of how RGA-IB differs from previous work to further highlight its contributions, especially in relation to studies that discuss Information Bottleneck theory [1].

[1] A Unified Framework of Graph Information Bottleneck for Robustness and Membership Privacy. KDD 2023.

1. Scalability and Computational Complexity: The proposed method introduces an attention matrix BBB of size N×NN \times NN×N, which leads to significant computational and memory demands, especially during gradient computation and matrix updates. This could limit the scalability of the method, particularly on large-scale graphs, and a complexity analysis is needed to better understand these demands.
2. Evaluation on Small to Medium Datasets: The datasets used for evaluation, such as Polblogs and Pubmed, are not large-scale. It is unclear how well the proposed method would generalize to or perform on larger graphs, raising questions about its applicability to real-world scenarios involving massive networks.
3. Limited Novelty: While the method applies the Information Bottleneck (IB) principle to improve robustness, this concept has already been explored in other works for enhancing robustness. Consequently, the contribution and novelty of this specific architecture may be limited compared to existing research.
4. Presentation and Organization Issues: Although the paper provides a detailed explanation of the method, there is considerable room for improvement in presentation, particularly in the methods section. The implementation details should not dominate the main part of the methodology, and the theoretical justification is lacking. The theorem and lemma only explain how to derive the gradient descent formulation.