Graph Neural Networks on Symmetric Positive Definite Manifold

Thank you for the thorough evaluation and constructive feedback on our paper. We appreciate the careful consideration of our proposed SPDGNN architecture. We have taken note of your comments and suggestions, and we are committed to addressing them in the revision. Below are our responses.

Q1: "Missing at least one recent paper on the same topic...."

Thanks for sharing with us this up-to-date work. There are three main reasons for the slight discrepancies between its results and the ones we reported: 1) Variations in experimental settings, as it employs higher-dimensional representations (8, 64, 256 dimensions), whereas we chose matrix features of 5 and 10 dimensions, corresponding to node representations of 15 and 55 dimensions, respectively. 2) Differences in the reported metrics, as it presents accuracy for Cora, whereas our four datasets are uniformly reported in terms of F1-score. 3) Differences in datasets like Citeseer and PubMed . Furthermore, to our knowledge, its source code is currently not publicly available, making it challenging for us to consider it as our comparative method. Nonetheless, we will cite it and add relevant discussion in the revised version.

Q2: "..., it would be good to discuss why geometric learning seems to underperform on PubMed and Cora and gives scores for other types of GNN on the same experiments..."

Thank you for your thoughtful comments. We understand your concern regarding the seemingly lower performance of geometric learning on PubMed and Cora. Allow us to provide a more detailed explanation:

• Problem Setting Clarification: It's crucial to emphasize that the default development of GNNs is rooted in Euclidean space assumptions, and successful methods within this paradigm often presume settings in Euclidean space. However, these assumptions may not universally hold, prompting our exploration of alternative geometric frameworks, specifically hyperbolic and SPD geometries. Our endeavor surpasses the constraints of Euclidean geometry, delving into more intricate geometric attributes.

• Addressing Fundamental Problems: Our approach is distinct in that it tackles a more fundamental problem within geometric learning. While state-of-the-art GNN methods developed in Euclidean space can be extended to hyperbolic or SPD manifolds, our work focuses on exploring the foundational geometric embedding space upon which GNNs rely. This emphasis on fundamental geometric properties allows us to understand the limitations and potentials of these spaces in the context of graph representation learning.

• Representative Baselines: We exclusively compare our approach against representative methods endowed with geometric properties, specifically Euclidean and hyperbolic geometries. The comparison is intentionally limited to methods that share a similar architectural foundation, allowing for a more focused assessment of the inherent advantages and challenges associated with our proposed SPD geometry.

In essence, our work extends beyond the conventional Euclidean assumptions of GNNs, exploring alternative geometric spaces to address fundamental challenges in graph representation learning. We acknowledge the potential superiority of state-of-the-art methods in Euclidean space, and we appreciate your suggestion to include additional context in our discussion. We will ensure that the revised manuscript provides a more comprehensive explanation of the motivation behind our geometric choices and their implications on the reported performance.

Q3: "The case of edge-features is not addressed"

Thank you for your nice suggestion. Indeed, edge features constitute a pivotal concern. However, this manuscript concentrates on the exploration of latent geometric spaces, with further investigations into its higher echelons earmarked for our subsequent endeavors.

Q4: "Could a scaling factor be used in conjunction with the Stiefel parameter to give more freedom to the model?"

Thank you for your suggestion, but preliminary analysis suggests that the inclusion of a scaling factor is unavailing for degrees of freedom. Stiefel parameters are acquired through Riemannian gradient descent, ensuring the positive definiteness of the transformed matrix. However, the utilizing of a scaling factor may potentially disrupt this assurance.

Q5: "Is the initial mapping limited to a linear map, what is its influence?"

There is no absolute constraint. In truth, the objective of the initial mapping is to alter the dimensionality $n$ of input features to accommodate the matrix dimension $m$ (ensuring $n = m(m+1)/2$). The utilization of a linear mapping is the most direct and efficient means for this purpose.

Thank you for your constructive feedback and valuable suggestions. In response, we have endeavored to address your concerns and provide clarifications.

Q1: "Many of the contributions in this work are, in fact, already present in the literature..."

Thank you for providing insightful comments and pointing out areas for improvement in our manuscript. While it is true that some aspects of our work share common ground with prior research, we contend that our study brings a unique contribution through generalizing the fundamental components of GNNs onto SPD manifold for learning node embeddings. Nevertheless, have carefully addressed each of your concerns in the revised manuscript.

• Feature Transformation: The feature transformation utilizing Stiefiel bilinear mapping [3] is indeed a classical methodology, as described in our related work. However, it is crucial to note that, within the domain of graph neural networks and graph embedding [1,4], Isometry maps for SPD matrices have been the prevailing methodology. Our contribution lies in introducing dimensionality transformations atop Isometry maps for crafting our feature transformation layer. We have conducted a comprehensive ablation study, including relevant comparisons, to substantiate the superiority of our approach over IsometryQR. We appreciate your suggestion to enhance references in the subsection on feature transformation, particularly by incorporating [3].

• Non-linearity: Regarding non-linearity and the choice of elevating eigenvalues, our contribution lies in propounding a non-linear methodology grounded in the log-Cholesky metric. In contrast to methods based on eigenvalue decomposition [3], our approach demonstrates efficiency by seamlessly integrating into the neighborhood aggregation layer (Eq.17) within the SPDGNN framework. This integration avoids the computational overhead associated with standalone eigenvalue decomposition.

• Riemannian/Cholesky Approach: We respectfully disagree with the interpretation that our Riemannian/Cholesky approach is borrowed from [2]. The Log-Cholesky metric, as a Riemannian metric on the SPD manifold, has found widespread application in various domains. [2] focuses on a distinct task related to Skeleton-Based Interaction Recognition and employs Cholesky space differently. In contrast, our methodology extends the operations of GNN onto the SPD manifold based on the log-Cholesky metric. Additionally, we have duly cited [2] in the related work pertaining to SPD neural networks.

• Comparison with [1]: We have addressed your suggestion to compare our method with [1] in the experiments. In the revised version, we have included SPD4GCN [1] as a baseline and reported the results of performance and efficiency.

We can observe that the performance of our SPDGNN surpasses that of [1] significantly, owing to our more comprehensive architecture.

We can observe a significant enhancement in the efficiency of SPDGNN compared to [1], attributed to our metric selection and component optimizations as discussed in Section 2. While current SPD methods incur a certain increase in time overhead compared to hyperbolic methods due to various factors, including Cholesky decomposition, Riemannian gradient descent, and hyperplane classifier, the further exploration of its efficiency is warranted by the rich geometric properties.

Thank you for taking time to review our paper and for providing valuable feedback. We appreciate your acknowledgement of our robust theoretical results and the improvements in experimental performance. We have thoroughly considered your suggestions and have addressed the raised points.

Q1: "Although the paper reads well, there are some areas of lower clarity, I recommended proofreading to improve the writing a bit."

Thanks for your suggestions. We have conducted a thorough proofreading to enhance the overall writing quality of the paper.

Q2: "The paper does not cite previous work on SPD neural networks,..."

Thank you for sharing with us these related works. [1] devises a densely connected feed-forward network explicitly tailored for the SPD manifold, incorporating a bi-linear mapping layer and a non-linear activation function. Meanwhile, [2] applies a bi-linear mapping layer and a non-linear activation function for image set classification. [3] focuses on coping with manifold-value images and presents the analogue of convolution operations for manifold-value data. While our work is related, the key differences are three-fold: 1) We concentrate on graph neural networks for non-structural graph data. 2) We investigate the generalization of GNN components to SPD manifold with the Log-Cholesky metric and propose a framework named SPDGNN, encompassing linear, non-linear, neighborhood aggregation, and MLR layers. 3) We conduct experiments on four real-world graphs and validate our method against various GNN models under both Euclidean and hyperbolic geometries. Nonetheless, we will cite these works and provide relevant discussion in the revised version.

Q3: "..., a better choice would be to use some of the more recent OGB benchmarks..."

Thank you for your valuable suggestions. Currently, the benchmarks commonly used in hyperbolic graph neural networks are still Disease, Airport, PubMed, and Cora [4,5]. Moreover, addressing OGB benchmarks poses additional resource demands for hyperbolic models due to the intricate mapping between the Riemannian manifold and the tangent space. We may consider expanding our experiments to OGB benchmarks after enhancing the efficiency of hyperbolic and other Riemannian manifold models.

Q4: "I might be missing something but why does the choice of n >= p ensure positive-definiteness of the transformed matrices? (page 5)"

There might be a misunderstanding. It is not the case that $n \geq p$ ensures the positive definiteness of the transformed matrix, but rather it is guaranteed by a full-rank row matrix. The terms $p$ and $n$ merely denote the numbers of rows and columns, respectively.

Q5: "Can the constraint of orthogonal matrices be relaxed?"

Certainly. Given that our transformation matrix $\mathbf{M}$ has $p$ rows and $n$ columns, where $p \leq n$, the orthogonality constraint solely limits that the rows to be orthogonal vectors. Hence, in essence, it is a semi-orthogonal constraint.

Q6: "... In the ablation study, can the authors clarify what alternatives were used..."

Thank you for your insightful feedback. In response to your suggestions, we have adjusted the settings for the ablation study, conducting new experiments on variants involving linear layer ablation to provide a more comprehensive understanding of our components. Specifically, we replaced the Stiefel Linear layer with IsometryQR, an isometric mapping approach [5,6]. Our Stiefel linear layer, built upon this method, incorporates functionality for dimension transformation, allowing us to obtain more compact node embeddings.The ablation results also demonstrate the efficacy of our method. Furthermore, in the context of non-linear ablation, we chose to directly omit the non-linear module. This decision stems from the fact that the non-linear module proposed in our approach, based on the Log-Cholesky metric, can be seamlessly integrated into the neighborhood aggregation module. This integration eliminates the separate overhead incurred by Cholesky decomposition, which is a fundamental reason for not directly introducing non-linear methods based on eigenvalue decomposition [1,2].

Thank you for the time and valuable comments. We are glad that you liked our methodology and experimental endeavors. We notice that your concerns lie in two major points: the evaluation compared to (Zhao et.al 2023) and the clarity of exposition. Below, we offer detailed responses to these points and have made corresponding modifications to enhance the revised version in accordance with your concerns.

Q1: "The exposition of this paper can be significantly improved. ... Figures 2 and 3 could be annotated and captioned more clearly..."

We regret any inconvenience caused to the readers and have meticulously refined our manuscript, particularly focusing on the logic and expression in the abstract, as outlined in our General Response. Additionally, we appreciate your suggestions regarding the clarity of Figures 2 and 3.

• In Figure 2, we present a 3D illustration of the SPD hyperplane, parameterized by the normal vector $\mathbf{V}$ and a bias point $\mathbf{P}$. The gray convex conical point cloud outlines the SPD space, while the green points, sampled within SPD space, delineate the SPD hyperplane.
• In Figure 3, we present a comparative analysis of 3D illustrations, depicting hyperplanes and node embeddings learned through Euclidean and SPD MLR, respectively. Unlike the flat Euclidean hyperplanes, the SPD hyperplanes encompass both flat and curved hyperplanes, corresponding to the geometric structures of Euclidean and hyperbolic spaces, respectively. These comprehensive hyperplanes effectively discriminate node embeddings from distinct catalogs (colors), demonstrating the efficacy of SPD MLR in concurrently incorporating both Euclidean and hyperbolic geometries for classification.

Q2: "Some important baselines appear to be lacking... & I miss any comparisons on runtime..."

Thank you for your valuable suggestions. Considering the scope of [1] does not align with the domain of graph neural networks, while [2] represents a direct extension for it, we exclusively choose SPD4GCN [2] as our comparative method. Noteworthy distinctions between our approach and [2] include:

• Metric: Our proposed operations are grounded in the Log-Cholesky metric, whereas [2] relies on the Gyrocalculus from [1].

• Architecture: In addition to the fundamental neighborhood aggregation layer in GNNs, we introduce dimension-varying linear transformation and SPD MLR layers, setting our approach apart from [2].

We have meticulously reproduced the experimental results of [2] using their source code, adhering to consistent experimental settings and data partitioning akin to other baselines. Due to variations in experimental settings, there are certain discrepancies compared to the original results; however, the overall trends remain consistent, particularly in the case of suboptimal results observed on the Airport dataset. These results have been integrated into the revised version of the manuscript.

We can observe that the performance of our SPDGNN surpasses that of [2] significantly, owing to our more comprehensive architecture.