InfoIGL: Invariant Graph Learning Driven by Information Theory

• The optimization of the proposed InfoIGL may be difficult due to the significant impact of Task 1 on Task 2 and the extra constraints between the contrastive learning designs. The authors should clarify the training strategy, especially for parameter initialization and the meaning of attention scores. Additionally, the relationship between redundant information and overfitting should be discussed more clearly. The paper could benefit from more detailed implementation details to facilitate reproducibility.

• Limited discussions compared with other baselines, such as CAL[1], which also adopts an attention mechanism to reduce redundant information. The authors should provide a more detailed theoretical analysis to highlight the merits of InfoIGL. The paper could include more ablation studies to further validate the significance of each component in the InfoIGL framework. The authors could provide more comprehensive comparisons with widely-known baselines in the field.

• Evaluations are only conducted on the graph classification task in limited scenarios. It would be beneficial to see experiments on more diverse scenarios (transductive and inductive settings) and tasks, such as node classification.

[1] Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, and Tat-Seng Chua. Causal attention for interpretable and generalizable graph classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1696–1705, 2022.

1. Although this paper converts the goal of identifying invariance on graphs into reducing redundant information and maximizing mutual understanding by means of approximate substitution and mathematical reduction, etc. However, these derivation methods are the application of existing works, mostly. Second, the degree of relaxation is kind of large, and thus the invariable learning is transformed into contrastive learning in general. There are questions whether the so-called invariance learned in this way is valid and should be further explained.

2. Weak contribution in methods: The core methods of InfoIGL are attention mechanism and graph contrastive learning, which are direct application of existing works and lack of necessary innovation. Apart from the theoretical derivation of the first half, all that is left of this paper is a graph contrastive learning framework, which means that the authors believe invariant identification can be achieved through contrastive learning. I think it needs further investigation.

3. Confusing experiment setups: In terms of data sets, although a benchmark dataset of graph OOD is adopted (GOOD), the selected data sets are kind of small and have less value in the real world (like CMNIST). In addition, the visualization experiment only demonstrated the embeddings of InfoIGL variants and its own, not compared to baselines. And there is a lack of visual experiments on the interpretability, i.e., which parts of the graph are more important to label predictions? This is also the problem proposed in this paper, but there is no relevant experiment.

4. Mutual information is viewed as measurement of invariance directly and heuristically. Though the theoretical proof seems solid and sufficient, I would like to see more clear and easy-to-follow examples.

1. The proposed method is simple, based on two contrastive learning to preserve the invariant features to improve the out-of-distribution generalization. However, the paper doesn't sufficiently discuss how the contrastive learning losses are derived from Theorem 2.
2. The paper could provide more discussion on how the attention mechanism can help to reduce the entropy of graph representations.
3. The L_cau in Figure 3 is not defined.
4. The technique contribution is limited. The attention mechanism and designed contrastive learning objectives are nothing new.

W1: The paper's presentation requires further improvement. For instance, in Section 3.2.4, consider changing "is not uncommon" to "is common."

W2: The method's description is unclear, and some details raise questions:

(i) In Equation (7), the authors mention initializing semantic $w_{c}$ based on the average semantic representation for examples in class $c$." It's unclear how class information is obtained in a self-supervised contrastive learning setup.

(ii) Equation (9) includes the symbol ${\tau}'$, which lacks a specific definition in the paper.

(iii) The authors don't incorporate any graph features into InfoIGL. Does this mean that InfoIGL is applicable to other data domains like images and natural language?

W3: The paper uses limited datasets, and some common datasets are omitted. CAIL and GREA, two important baselines, use datasets such as TUDataset and ogbg datasets. It would be beneficial if the authors could conduct experiments on a wider range of datasets to validate their proposed method.