Riemannian Transformation Layers for General Geometries

We thank reviewer $\textcolor{red}{piKA}$ for the careful review. Below is our detailed response. 😄

---

1. Thm. 4.1 --> Def. 4.1 & Thm. 4.2

Thank you for the suggestive comments. We have revised Sec. 4.1 in our updated manuscript. In the revised paper, we first extend the reformulation of the Euclidean FC layer [a, Sec. 3.2] to general manifolds and provide the general definition in Def. 4.1. Subsequently, we demonstrate that Def. 4.1 can be solved using our proposed method, which is presented in Thm. 4.2 (corresponding to the previous Thm. 4.1).

---

2. Flexibility: Comparison against Modified Previous Grassmannian Layers

As shown in Tab. 2, previous Grassmannian layers inherently allow changes at most to the ambient dimension. We further conduct additional ablations on modifying previous layers, including channel aggregation and truncation & lifting for the subspace dimension. However, these modifications do not result in significant improvements for the previous layers and, in most cases, lead to even worse performance. The primary reason is that these modifications fail to respect the latent geometry, potentially leading to unsatisfactory results.

Modification 1: Aggregation of the Channel Dimension

To enable the previous layers to reduce the channel dimension, we implemented an aggregation layer after these layers. The aggregation was performed using the Fréchet mean or the extrinsic mean [b] along the channel dimension.

The 5-fold average results are presented in Tab. A. We observe that none of the three previous networks—GrNet, GyroGr, and GyroGr-Scaling—benefit from aggregation. The aggregation often degrades overall performance. Furthermore, we find that using the Fréchet mean for aggregation causes GrNet to produce NaN values. Consequently, we report N/A for this outcome.

Table A: Comparison against previous layers using channel aggregation on the Radar dataset. Here, $C_{\text{in}}$ and $C_{\text{out}}$ represent the input and output channel dimensions, respectively.

(To be continued)

We thank Reviewer $\textcolor{purple}{qFyX}$ for valuable comments. Below, we address the comments in detail. 😄

---

1. Riemannian Convolution and GPUs Acceleration

Matrix Manifolds: GPUs & Dimensions

As the Riemannian computations on matrix manifolds require SVD-based matrix functions, CPUs could normally outperform GPUs. However, we found that our convolution benefits significantly from GPUs acceleration when the input matrices have relatively small dimensions ($\leq 30$), which is the case in all our implementations.

SPDConvNets: The input dimensions of the SPD matrices for our SPDConvNets are [20,20], [28,28], and [28,28] on the Radar, HDM05, and FPHA datasets, respectively. Under these small dimensions, we observed that GPUs can greatly accelerate training, reducing the average training time by up to four times.

GrConvNet: The input dimensions of the Grassmannian matrices for our GrConvNet are [20,p] on the Radar dataset, with $p = 4, 8, 6$. Similarly, we found that GPU acceleration significantly reduces training time.

However, for relatively large dimensions, CPUs are generally faster than GPUs. For example, previous SPD networks modeled each raw signal into a relatively large global covariance matrix ([93,93] for the HDM05 and [63,63] for the FPHA datasets). In such cases, Riemannian computations are more efficient on CPUs.

Traning Efficiency

Tab. A summarizes the average training time for each SPD network on the most efficient device. Due to the fast and simple computation under LEM, PEM, and LCM, our SPDConvNet models based on these three metrics are significantly more efficient than those based on AIM and BWM, whose computations are more complex. Particularly, SPDConvNet-LCM is the most efficient one among the five SPDConvNets. It outperforms several prior SPD baselines in terms of training time and even achieves comparable efficiency against the simplest SPDNet. However, SPDConvNet delivers noticeable performance improvements, surpassing SPDNet by up to 5.02%, 16.59%, and 6.24% on the Radar, HDM05, and FPHA datasets, respectively.

Table A: Training efficiency (seconds/epoch) of different SPD networks.

Vector Manifolds and GPU Acceleration

When it comes to vector manifolds, such as the hyperbolic manifold, their Riemannian computations are decomposition-free, allowing them to benefit significantly from GPU acceleration. During the rebuttal, we further implemented our Riemannian transformation under two hyperbolic geometries and found that GPUs are significantly faster than CPUs. For more details, please refer to CQ#1 in the common response.

---

2. Riemannian FC Layers on Manifolds with Implicit Structures

Thank you for the insightful comments! Our FC layer in Thm. 4.2 requires the operators such as Riemannian log and exp. If a Riemannian metric has explicit expressions for these operators, the final expression can often be further simplified, as demonstrated in App. F for the SPD manifolds, Thms. 6.1 and 6.2 for the Grassmannian, and Thms. J.1–J.2 for hyperbolic spaces. On the other hand, if closed-form expressions are unavailable, we can rely on approximation algorithms for computation, such as [a] for Lie groups. This could bring extra computational burden, as it is not easy to do further simplification.

However, most matrix or vector manifolds in machine learning typically have closed-form formulations for Riemannian computations. A plethora of examples can be found in [b].

---

3. Advantages: Intrinsicness, Generality, and Flexibility

Compared with previous transformation layers, our Riemannian transformation offers three major advantages: it more faithfully respects the latent geometry, provides generality in handling different geometries, and offers flexibility in changing dimensionality. Please refer to CQ#2 in the common response for more details.

---

4. Numerical Tricks for Incompleteness

In this paper, incompleteness (locally defined exponential map) occurs only for PEM and BWM on the SPD manifold. This issue can be resolved using numerical tricks, such as regularizing the eigenvalues of the input tangent vector, as discussed in Rmk. B.1 in the appendix.

For the general case, we can regularize the norm of the tangent vector to ensure it lies within the domain of the exponential map. Specifically, for a given manifold $\mathcal{M}$ and $p \in \mathcal{M}$, we assume that the exponential map is well-defined over the ball $B_p(r)$ in the tangent space $T_p \mathcal{M}$:

$

B_p(r) = \lbrace v \in T_p \mathcal{M} \mid \Vert v \Vert _p < r \rbrace,

$

where $\Vert \cdot \Vert _p$ denotes the norm over $T_p \mathcal{M}$. To regularize a tangent vector $v$ for $\mathrm{Exp} _p(v)$, we define:

$

v = \begin{cases} v & \text{if } v \in B_p(r), \\ \epsilon \frac{v}{\Vert v \Vert _p} & \text{otherwise,} \end{cases}

$

where $\epsilon$ is a small positive value.

---

References

[a] A reduced parallel transport equation on Lie groups with a left-invariant metric

[b] https://www.manopt.org/manifolds.html

---

6. Sec. 7: Manifold Embedding as the Riemannian FC Layer

In several applications [e,i-k], a common approach for embedding Euclidean features into a manifold involves mapping the Euclidean vector to the tangent space at the origin via a linear layer, followed by the exponential map at the origin:

$

\operatorname{Exp}_E(Ax + b): \mathbb{R}^n \rightarrow \mathcal{M}, \quad Eq.(A)

$

where $Ax + b$ denotes the linear function mapping $\mathbb{R}^n$ into $\mathbb{R}^m \cong T_E\mathcal{M}$.

However, our framework provides a novel intrinsic insight. Eq. (A) is, in fact, a special case of our Riemannian FC layer (Thm. 4.2) between $\mathbb{R}^n$ and $\mathcal{M}$. By setting $\mathcal{N} = \mathbb{R}^n$ in Thm. 4.2, the Riemannian FC layer in Eq. (12) reduces precisely to Eq. (A). Therefore, beyond the existing tangent-space interpretation, Eq. (A) serves as an FC layer from the Euclidean space $\mathbb{R}^n$ to the manifold $\mathcal{M}$. This could be a natural choice when mapping Euclidean features into the manifold.

---

7. Numerical Tricks for Ill-defined Riemannian Logarithm

In this paper, the ill-definedness of the Riemannian logarithm arises only on the Grassmannian. As discussed Rmk. B. 2, the Grassmannian logarithm $\mathrm{Log}_P(Q)$ exists only if $P$ and $Q$ are not in each other's cut locus. However, this can be numerically solved, such as the extended Grassmannian logarithm [l, Alg. 1] or replaces the inverse with the Moore–Penrose inverse $(\cdot) ^\dagger$ [h]: \begin{gathered} \mathrm{Log} _ P (Q) = O \arctan (\Sigma) R^{\top} \\ \left(I_n-P P^{\top}\right) Q\left(P^{\top} Q\right)^\dagger \stackrel{\text { SVD }}{:=} O \Sigma R^{\top}. \end{gathered} In our implementation, we use the latter one, as it is already successfully implemented in several applications [h,i].

---

8. On the Linear Isometry

Denote the $m$-dimensional manifold as $\mathcal{M}$, the Riemannian metric at the origin $e$ as $g _{e}$, and the standard orthonormal basis as $\lbrace e _i = (\delta _{1i}, \cdots, \delta _{mi}) ^\top \in \mathbb{R} ^m \rbrace _{i=1} ^m$. If $g _{e}$ is the standard Euclidean inner product, then $\lbrace e _i \rbrace _{i=1} ^m$ forms an orthonormal basis. Otherwise, an orthonormal basis can be obtained via a linear isometry.

Concrete examples: Below is a summary of nine geometries across three manifolds:

• SPD manifolds: Lines 1618–1624 in App. K.3 provides the linear isometries for all five metrics.
• Grassmannian manifolds: As the Riemannian metric at the origin is the standard inner product, the standard orthonormal basis can be directly obtained, as presented in Eq. (136).
• Hyperbolic manifolds: In App. J of the revised manuscript, we further manifest our FC layer on two hyperbolic models. As the Riemannian metrics at the origin are relatively simple, their orthonormal bases can be directly obtained, as discussed in Apps. K.8 and K.9, respectively.

Canonical method: The general case can be solved by performing a congruence decomposition of an SPD matrix. Since the inner product $g _e$ can be identified as an SPD matrix, we can perform congruence decomposition on the SPD matrix to transform it into the identity matrix (corresponding to the standard inner product). For any tangent vectors $v, u \in T _e \mathcal{M} \cong \mathbb{R} ^m$, we have:

$

g _e(v, u) = v^\top S u = (Cv) ^\top (Cu),

$

where $S = C ^\top C$ is the congruence decomposition of the SPD matrix $S$ representing $g _e$. Here, $C$ is ensured to be invertible. Then, an orthonormal basis can be obtained as $\lbrace C ^{-1} e _i \rbrace _{i=1} ^m$, with $\lbrace e _i \rbrace _{i=1} ^m$ as the familiar standard orthonormal basis.

Remark: All nine geometries, including five for SPD manifolds, two for the Grassmannian, and two for hyperbolic spaces, discussed in this paper does not need the canonical method, as each of them has a much simpler way to identify the orthonormal basis.

---

References

[a] Building neural networks on matrix manifolds: A Gyrovector space approach

[b] RMLR: Extending multinomial logistic regression into general geometries

[c] A Lie group approach to Riemannian batch normalization

[d] Riemannian residual neural networks

[e] Vector-valued distance and gyrocalculus on the space of symmetric positive definite matrices

[f] Hyperbolic neural networks

[g] Hyperbolic neural networks++

[h] The Gyro-structure of some matrix manifolds

[i] Matrix Manifold Neural Networks++

[j] Hyperbolic graph convolutional neural networks

[k] Modeling graphs beyond hyperbolic: Graph neural networks in symmetric positive definite matrices

[l] A Grassmann manifold handbook: basic geometry and computational aspects

We thank Reviewer $\textcolor{green}{48Wz}$ for the careful review and suggestive comments. Below, we address the comments in detail. 😄

---

1. Optimal Metric Could Vary across Datasets and Network Structures

Matrix classification: Although LEM performs better on all three datasets under the current method, it is difficult to conclude that LEM is the universally optimal metric for SPD matrix learning. The optimal metric could vary across datasets and tasks. Supporting evidence can be found in [a, Tabs. 1-2], [b, Tabs. 3-9], [c, Tabs. 4-5], and [d, Tab. 2].

Hyperbolic manifold: Unlike matrix manifolds, the hyperbolic space has the concept of hyperbolicity to describe the dataset. In CQ#1 in the common response, we observe an interesting phenomenon in hyperbolic geometries: When the hyperbolicity is high, the Poincaré ball model tends to be more effective, while the hyperboloid model is more effective in lower-hyperbolicity cases.

Metric as a hyperparameter: The above discussion indicates that metric is a crucial hyperparameter for Riemannian computations. Therefore, this requires the ability of the proposed methods to handle different geometries, enabling comparisons across geometries within a consistent architecture. Fortunately, our methods possess this flexibility and generality.

---

2. Advantages over the Gyrovector Calculus

Broader applicability than the gyro framework: Although gyro calculus has demonstrated success in various applications [e-i], it does not generally exist on all manifolds. In contrast, our Riemannian FC layer can be implemented across diverse manifolds, extending beyond those that admit gyro structures. Furthermore, as shown in Tab. 1, the three previous gyro SPD FC layers [i] are incorporated as special cases within our framework.

---

3. Theoretical Advantages over Previous Layers: Intrinsicness, Generality, and Flexibility

Compared with previous transformation layers, our Riemannian transformation offers three major advantages: it more faithfully respects the latent geometry, provides generality in handling different geometries, and offers flexibility in changing dimensionality. Please refer to CQ#2 in the common response for more details.

---

4. Experiments on the Hyperbolic Space

We have further implemented our Riemannian FC layer over the Poincaré ball and hyperboloid models. Please refer to CQ#1 in the common response. We also thank the reviewer for mentioning horocycles. However, it is likely to be non-trivial to generalize horocycles from Poincaré ball into general manifolds. In contrast, our methods enjoy great applicability.

---

5. Training Efficiency

Tab. A summarizes the average training time for each SPD network. Our SPDConvNet models based on LEM, PEM, and LCM are significantly more efficient than those based on AIM and BWM. This efficiency stems from the fast and simple Riemannian computations under LEM, PEM, and LCM, in contrast to the more burdensome computations involved in AIM and BWM. Additionally, our parameter trivialization, as discussed in Sec. 4.3, not only eliminates the need for complex Riemannian optimization but also simplifies the final FC layer expression (see App. F). As a result, SPDConvNets based on LCM, LEM, and PEM achieve training efficiency even comparable to the vanilla SPDNet while outperforming several prior SPD baselines in terms of training time.

Moreover, SPDConvNet delivers noticeable performance improvements, surpassing SPDNet by up to 5.02%, 16.59%, and 6.24% on the Radar, HDM05, and FPHA datasets, respectively. This highlights that SPDConvNet not only retains computational efficiency but also achieves a noticeable improvement in accuracy, offering a compelling balance between efficiency and performance.

Table A: Training efficiency (seconds/epoch) of different SPD networks.

Remark: The above discussion has been added into App. I. 4.

Thank you for your response. The architecture design seems well-motivated, and addresses short-falls of HNN such as how it reduces to an Exp-Log chain. However, I am afraid that the experiments on hyperbolic space do not seem complete. The table should include experiments on the PubMed dataset for completeness, and a comparison to Riemannian ResNets [a], one of the few prior works that also propose a "general Riemannian" neural network architecture.

[a] also allows one to use different metrics, is "intrinsic" to the underlying manifold, and can theoretically work for general manifolds. To me, it seems that the major difference between [a] and the proposed work is that [a] only uses Exp maps, while the paper proposes to use both Exp and Log maps. It would strengthen the paper to add this comparison for hyperbolic space, and justify the benefits of using an additional Log map.

[a] Katsman, et al., 2023. Riemannian Residual Neural Networks

2. Comparison with RResNet [A]

Difference and Relation

$

\text{Residual Block: } f: \mathcal{M} \ni x \mapsto \mathrm{Exp} _{x ^{(i-1)}}\left(\ell _i\left(x ^{(i-1)}\right)\right) \in \mathcal{M}, \quad \text{Eq. (A)}

$

where $\ell _i: \mathcal{M} \rightarrow T \mathcal{M}$ generates the vector field.

RResNet can be combined with our layers. Our proposed approach focuses on FC layers, while RResNet introduces ResBlocks. These two methods address different aspects of the problem and are not mutually exclusive; they can complement each other. Specifically, as illustrated by Eq. (A), RResNet can not perform dimensionality reduction. A transformation layer is normally required to reduce the input dimensionality before applying ResBlocks. In the original implementation [B], the hyperbolic RResNet (HResNet) uses a linear layer (torch.nn.linear) for dimensionality reduction, while the SPD RResNet employs BiMap. This transformation layer can be naturally replaced with our proposed FC layers.

SPD Manifolds

Comparison on the SPD manifold: In Tab. 3 of our main paper, we have directly compared RResNets with our SPDConvNets on the SPD manifold. Tab. C recalls the results from Tab. 3. Our SPDConvNets outperform RResNets across all three datasets. Specifically, SPDConvNets achieve improvements of 2.54, 11.04, and 5.2 on the Radar, HDM05, and FPHA datasets, respectively, when evaluated under their respective optimal metrics.

Table C: Comparison of SPDConvNets against RResNets on the Radar, HDM05, and FPHA datasets.

Additional Experiments on the Hyperbolic RResNet (HResNet)

As HResNet relies on horosphere projection based on the Poincaré ball model, we focus on the Poincaré ball model and integrate our RiemFC-P (based on the Poincaré ball model) into the HResNet backbone with 3 residual blocks. We evaluate RiemFC-P on HResNet under different configurations, including different numbers of horospheres and dimensions of the Poincaré ball. These experiments are conducted on the Disease and Airport datasets for the link prediction task, both of which exhibit high hyperbolicity.

The results, as shown in Tabs. D and E, demonstrate that integrating RiemFC-P into HResNet consistently improves its performance. This highlights the advantage of using RiemFC-P to better align with the underlying hyperbolic geometry.

Table D: Comparison of HResNet with or without RieFC-P under different settings on the Disease dataset.

Table E: Comparison of HResNet with or without RieFC-P under different settings on the Airport dataset.

References

[A] Riemannian Residual Neural Networks

[B] https://github.com/KaimingHe/deep-residual-networks

We thank the reviewer for the suggestive comments. Below is our detailed response. 😄

1. Expanded Experiments on the Pubmed Dataset

Following your suggestion, we conducted additional experiments to compare the previously proposed transformation layers with our Riemannian FC (RieFC) layers on the Pubmed dataset. Tab. B presents the results across all four datasets.

Pubmed exhibits moderate hyperbolicity ($\delta = 3.5$), where neither the Poincaré ball nor hyperboloid-based RiemFC demonstrates a significant improvement， and the hyperboloid-based RiemFC performs poorly. This aligns with our previous findings: Poincaré ball-based methods are effective in highly hyperbolic settings, while hyperboloid-based or tangent space-based methods perform better in low-hyperbolicity scenarios.

Table B: Comparison of different transformation layers on the link prediction task. The hyperbolicity is denoted as $\delta$ (lower is more hyperbolic). The best two results are hightlighted in $\textcolor{red}{**red**}$ and $\textcolor{blue}{**blue**}$.

5. Activation on the Manifold: from Feature Activation into Geometry Activation

Matrix power as the non-linear activation function: In our implementation, we use the matrix power function as the activate function in SPDConvNets, as discussed in App. I. 2. Matrix power has shown success in different applications [e,f]. As shown in [g, Sec. 3.1], the matrix power can deform the SPD Riemannian metric. An illustration can be found in [h, Fig. 1]. Intuitively speaking, matrix power can interpolate between a given metric to a metric of Log-Euclidean Metric (LEM) families. We expect this deforming effect of matrix power to activate the latent geometries, as our network is completely induced from the Riemannian geometry. Therefore, at the beginning of the network, we apply the matrix power to first activate the geometry.

Ablation on matrix power: We further conduct ablations on different powers on the SPD manifold. We focus on SPDConvNet-LEM and SPDConvNet-AIM on the Radar dataset. The 5-fold average results are presented in Tab. B. We observe that the network benefits from a proper matrix power.

Table B: Ablations on power activation.

Non-linearity of our Riemannian transformation: We argue that our Riemannian transformation layer is already highly non-linear. Recalling Thms. 5.1, 6.1, and 6.2, our SPD and Grassmannian FC layers involve various kinds of non-linear matrix functions. Therefore, we did not use an additional activation function after our transformation layer. While activations, such as the matrix power, can be applied following each transformation layer, they will introduce additional computational costs. Given the inherent non-linearity of the transformation, we only employed such activations at the beginning of the network, as described in the main paper.

---

5. Our Riemannian FC Layer Does not Generally Reduce to an Exp-Log Chain.

Since the Möbius transformation is defined via the tangent space, it is natural that stacking Möbius transformations reduces to an exp-log chain. This is why gyro addition was introduced for biasing. However, this is not the case for our Riemannian FC layer. Referring to Eq. (A), stacking multiple Riemannian FC layers generally does not reduce to an exp-log chain.

The primary reason lies in the presence of $\mathrm{Log} _{P _k}(X)$ instead of $\mathrm{Log} _{E}(X)$. Since $\mathrm{Log} _{P _k}$ does not generally cancel out with $\mathrm{Exp} _{E}$, the exp-log chain reduction generally does not occur in our framework.

---

6. PP --> PT.

Thanks for the careful review. We have revised this typo. In this paper, PT denotes the parallel transport, while PP denotes the projector perspective of the Grassmannian.

References

[a] Hyperbolic neural networks

[b] A Riemannian network for SPD matrix learning

[c] Hyperbolic neural networks++

[d] Matrix Manifold Neural Networks++

[e] Deep CNNs meet global covariance pooling: Better representation and generalization

[f] Fast differentiable matrix square root and inverse square root

[g] The geometry of mixed-Euclidean metrics on symmetric positive definite matrices

[h] RMLR: Extending Multinomial Logistic Regression into General Geometries

---

3. Convolution within Each Receptive Field Is an FC Transformation

From FC layer to convolution: In each receptive field, the Euclidean convolution takes the form of the FC transformation ($Ax+b$). Let us focus on a single receptive field. Given a $c$-channel vector in a receptive field $\mathbf{x} = \text{concat}(x_1, \cdots, x_c) \in (\mathbb{R}^n)^c$ with $x_i \in \mathbb{R}^n$ as the feature vector in the $i$-th channel, the Euclidean convolution within this receptive field can be expressed as:

$$\text{Conv}(\mathbf{x}) = \text{concat}\left(f^1(\mathbf{x}), \cdots, f^k(\mathbf{x})\right),$$

where $f^i(\cdot) : (\mathbb{R}^n)^c \to \mathbb{R}^m$ represents the FC transformation parameterized by the $i$-th convolutional kernel. Therefore, the overall process of convolution can be characterized as: 1) splitting input features into multiple receptive fields; 2) applying the FC transformation within each receptive field; and 3) concatenating the resulting features. The above process can be easily adapted to a manifold, as long as the Riemannian FC layer is proposed. This is basically how HNN++ constructs the convolutional layer based on their proposed Poincaré FC layer [c, Sec. 3.4].

Revised Sec. 4.2: We acknowledge that Sec. 4.2 in our previous manuscript did not clearly convey this idea, and we have revised this section in the updated paper. Additionally, we have included a figure for better illustration. Thank you for your careful review!

---

4. Biases Are Already Contained in Our Riemannian FC Layer.

In our Riemannian FC layer (Eq. (A) in this rebuttal), $\lbrace P _i \rbrace _{i=1} ^m$ corresponds to the bias. An intuitive understanding can be gained by considering the special Euclidean case where $\mathcal{M}=\mathbb{R} ^m$ and $\mathcal{N}=\mathbb{R} ^n$. This makes our FC layer reduce to the familiar Euclidean FC layer, as shown in Prop. 4.4 and proven in App. K.2.

In this case, Eq. (A) becomes:

$

\mathcal{F}: \mathbb{R} ^n \ni x \mapsto \sum _{i=1} ^m\left(\left\langle x-p _i, a _i\right\rangle e _i\right) \in \mathbb{R} ^m,

$

where $\lbrace e _i \rbrace _{i=1} ^m$ is the familiar standard orthonormal basis. Further considering our parameter manipulation discussed in Sec. 4.3:

$

p _i &= \mathrm{Exp} _{\mathbf{0}} (\gamma _i [z _i]) = \gamma _i [z _i], \\\\
a _i &= \Gamma _{\mathbf{0} \rightarrow p _i} (z _i) = z _i,

$

where each $\gamma _i \in \mathbb{R}$ and $z _i \in \mathbb{R} ^n$ are parameters, and $[z _i]$ are the unit vector of $Z _i$. Combining the above, we have:

$

\mathcal{F}: \mathbb{R} ^n \ni x \mapsto \sum _{i=1} ^m\left( \left(\left\langle x, z _i\right\rangle  + b _i\right) e _i\right) \in \mathbb{R} ^m,

$

where $b _i = -\gamma _i \Vert z _i \Vert$ comes from $p _i$. The right-hand side is the element-wise reformulation of $Zx+b$. Here, we can see that $p _i$ corresponds to the bias.

For an in-depth understanding, please refer to our revised Sec. 4.1, where we technically extend the Euclidean FC layer into manifolds via the hyperplane projection.

We thank Reviewer $\textcolor{blue}{gKrX}$ for the valuable comments. Below, we address the comments in detail. 😄

---

1. Validated Effectiveness and Flexibility on the Hyperbolic Space

We manifest our Riemannian FC layer on the Poincaré ball and hyperboloid models, respectively. The experiments validate the effectiveness and flexibility of our Riemannian FC layer. For further details, please refer to CQ#1 in the common response.

---

2. Fundamental Differences with HNN [a, Sec. 3.2] and SPDNet [b, Eq. (1)]; Generalization of HNN++ [c, Sec. 3.2]

Key Difference

$

\text{Möbius Transformation in HNN: }& \mathrm{Exp} _0\left(f\left(\mathrm{Log} _0(x)\right)\right), \text{ with } f \text{ as a linear map}, \\ \text{BiMap Transformation in SPDNet: }& WSW^\top, \text{ with } W \text{ row-wisely orthogonal }.

$

• Möbius & BiMap: The Möbius transformation in HNN is defined by tangent space, while the BiMap in SPDNet takes a special form of linear map. The former largely resorts to the local linearization by a single tangent space (typically at the identity element), while the latter has little connection to the latent Riemannian geometry, although it can preserve the SPDness. As a result, both methods fail to faithfully respect the complex latent geometry.
• Riemannian FC layer: In contrast, our Riemannian FC layer (Thm. 4.2) is independent of the above two methods and is derived from the reformulation of the Euclidean FC layer. It is directly induced by the Riemannian geometry and does not resort to local linearization by a single tangent space. Its expression also greatly differs from the previous ones:

$

\mathcal{F}: \mathcal{N} \rightarrow \mathcal{M}: \operatorname{Exp} _E ^{\mathcal{M}}\left(\sum _{i=1} ^m\left(\left\langle\log _{P _i} ^{\mathcal{N}}(X), A _i\right\rangle _{P _i} ^{\mathcal{N}} B _i\right)\right), \quad Eq. (A)

$

where each $P _i \in \mathcal{N}$ and $A _i \in T _{P _i} \mathcal{N}$ are parameters. Here $\lbrace B _i \rbrace _{i=1} ^m$ is an orthonormal basis over the tangent space at the origin $E$ of $\mathcal{M}$.

Connection: Möbius Transformation as a Trivial Riemannian Transformation

The tangent transformation, such as the Möbius one, is a special case of our Riemannian FC layer under trivial parameters $\lbrace P _i = E \rbrace _{i=1} ^m$.

For simplicity, let's take the Poincaré ball $\mathbb{P} _{K} ^n$ as an example. Setting all $P _k$ equal to zero vector $\mathbf{0}$ and $\lbrace B _i \rbrace _{i=1} ^m$ as the standard basis $\lbrace e _i = (\delta _{i1}, \delta _{i2}, \dots, \delta _{im}) ^\top \rbrace _{i=1} ^m$, our Riemannian FC layer are reduced to

$

\mathcal{F}: \mathbb{P} _{K} ^n \rightarrow \mathbb{P} _{K} ^m: &\operatorname{Exp} _{\mathbf{0}}\left(\sum _{i=1} ^m\left(\left\langle\log _{\mathbf{0}}(x), a _i\right\rangle _{e_i} e _i\right)\right) \\ &\stackrel{(1)}{=} \operatorname{Exp} _{\mathbf{0}}\left(\sum _{i=1} ^m \left(\left\langle\log _{\mathbf{0}}(x), \widetilde{a} _i\right\rangle e _i\right)\right) \\ &\stackrel{(2)}{=} \operatorname{Exp} _{\mathbf{0}}\left( f(\mathrm{Log} _{\mathbf{0}}(x)) \right).

$

The last equation is exactly the Möbius transformation. Here, $\langle \cdot,\cdot \rangle$ is the standard Euclidean inner product and $f$ takes the form of $Ax$. The above comes from the following:

(1) $\langle u,v \rangle _{\mathbf{0}}=4 \langle u,v \rangle$ and let $\widetilde{a} _i = 4 a _i$;

(2) $Ax$ can be written element-wisely as $\langle x, \widetilde{a}_i \rangle$.

Related Work & Revised Sec. 4.1

The most closely related work is the Poincaré FC layer in HNN++ [c, Sec. 3.2] and the three gyro SPD FC layers in [d, Sec. 3.2]. However, both approaches rely on specific properties, such as gyro structures, as discussed in Sec. 3.2. Our contributions lie in generalizing the Poincaré FC layer to general Riemannian geometries and providing a general solution. Additionally, as shown in Tab. 1, the previous three gyro SPD FC layers are incorporated into our framework as special cases.

Remark: To improve clarity, we have revised Sec. 4.1 to better present how the ideas in HNN++ are extended to the general case.

---

CQ#2. Advantages: Intrinsicness, Generality, and Flexibility (Reviewers $\textcolor{green}{48Wz}$ and $\textcolor{purple}{qFyX}$)

• Typical previous layers:

$

\text{BiMap Transformation in SPDNet [d, Eq. (1)]: }& WSW^\top, \\ \text{FRMap + ReOrth in GrNet [e, Eq. (2-3)]: }& \mathcal{Q}(WX),

$

where $W$ is a row-wisely orthogonal parameter, and $\mathcal{Q}(\cdot)$ returns the orthogonal matrix from the QR decomposition.

Compared with previous transformation layers, our Riemannian transformation offers three major advantages: it more faithfully respects the latent geometry, provides generality in handling different geometries, and offers flexibility in changing dimensionality.

• Intrinsicness: Our Riemannian FC layer faithfully respects the underlying complex geometries.

  • The widely used BiMap has little connection to the underlying geometry. While it numerically preserves SPDness, it is not associated with any of the five SPD Riemannian metrics (as shown in Tabs. 6-7). Similar issues exist with the FRMap + ReOrth layers for the Grassmannian, which lack connection to Grassmannian geometry (as shown in Tab. 8), apart from the preservation of orthogonality.
  • In contrast, our SPD and Grassmannian transformation layers (Thms. 5.1, 6.1, 6.2) are fully derived from Riemannian geometries, faithfully respecting the underlying latent structures. Notably, for the SPD manifold, the concrete expressions of FC layers differ across various geometries. This distinction emphasizes a critical insight: Riemannian geometry is essential for computations on the manifold. Ignoring these vibrant Riemannian geometries might compromise the overall representation power and fidelity of the transformations.

• Generality in applicable metrics: Our Riemannian FC layer in Thm. 4.2 can be implemented across different geometries, as it only requires explicit expressions of the associated Riemannian operators. Therefore, our method allows for direct comparison of different geometries within the same network architecture. Tabs. 3-4 indicate that the optimal metric may vary. We also observed similar findings on hyperbolic spaces, which are discussed in CQ#1. These indicate that the Riemannian metric should serve as a crucial hyperparameter in Riemannian networks. Supporting evidence can be also found in [f, Tabs. 1-2], [g, Tabs. 3-9], [h, Tabs. 4-5], and [i, Tab. 2].

• Flexibility in dimensionality: Taking the Grassmannian $\mathrm{Gr}(p, n)$ as an example, our transformations, as summarized in Tab. 2, naturally support changes in the subspace dimension $p$, ambient dimension $n$, and channel dimension $c$. In contrast, prior layers are restricted to modifying at most the ambient dimension $n$. The experiments on the Grassmannian, shown in Tab. 4, demonstrate that this flexibility can be advantageous for network training. While previous layers can be numerically modified to allow for more flexible dimensionality, such tricks often fail to respect the latent geometry and may lead to unsatisfactory results. The associated ablation studies are the 2nd response to Reviewer $\textcolor{red}{piKA}$.

---

References

[a] Hyperbolic neural networks

[b] Hyperbolic neural networks++

[c] Hyperbolic graph convolutional neural networks

[d] A Riemannian network for SPD matrix learning

[e] Building deep networks on Grassmann manifolds

[f] Building neural networks on matrix manifolds: A Gyrovector space approach

[g] RMLR: Extending multinomial logistic regression into general geometries

[h] A Lie group approach to Riemannian batch normalization

[i] Riemannian residual neural networks