We thank Reviewer $\textcolor{red}{Yeux}$​ for the constructive suggestions and insightful comments! In the following, we respond to the concerns in detail. 😄

1. Summary tables of Grassmannian and hyperbolic GyroBNs.

The specific implementation of Alg. 1 on a gyrogroup can be carried out in a plug-in manner by substituting the required operators from Tab. 2 and Tab. 8 into Alg. 1. This approach is how we construct the specific Grassmannian and hyperbolic GyroBNs detailed in Sec. 6. To improve readability and accessibility, we have added a summary table in App. G, which comprehensively lists all the required operators for the Grassmannian and hyperbolic GyroBNs.

2. Computational efficiency: Our GyroBN is more efficient than previous Riemannian BN methods.

We compared the training efficiency of our GyroBN with previous BN methods on both Grassmannian and hyperbolic spaces. On the most representative dataset for the Grassmannian, the training time per epoch for GyroBN is approximately one-tenth of that required by previous BN methods. For further details, please refer to CQ#1 in the common response.

3. Initialization in Alg. 1.

The initialization generally aligns with the standard Euclidean BN [a]. However, on the hyperbolic space, we follow [b, Alg. 1] to initialize the running mean and variance for a fair comparison.

• Running mean and variance: Similar to the standard Euclidean BN, they are initialized as the identity element ($I_{p,n}$) and 1 on the Grassmannian. On the hyperbolic space, following [b, Alg. 1], we initialize the running mean and variance as the first batch mean and variance.
• Biasing parameter: Similar to the standard Euclidean BN, it is initialized as the identity element: $I_{p,n}$ for the Grassmannian and the zero vector for the hyperbolic space.
• Scaling parameter: Similar to the standard Euclidean BN, it is initialized as 1.

4. Discussion on gyrogroups with non-isometric gyrations.

Hypothesis A: Any gyration over the gyrogroup $G$ is a gyroisometry.

If Hyp. A does not hold, GyroBN can still be applied numerically; however, its ability to control sample statistics is not guaranteed.

Recalling Eq. (59) in the proof of Thm. 3.5, we observe that the gyroisometry of gyrations is the necessary and sufficient condition for the gyro left translation to be a gyroisometry (by Thm. 3.3). Combined with the proof of Thm. 4.1, this implies that GyroBN can normalize the sample statistics only if all gyrations in the gyrogroup are gyroisometries.

That said, as demonstrated in Prop. 3.6, Hyp. A holds for all gyrogroups currently used in machine learning, ensuring that GyroBN maintains its theoretical guarantees in these cases. However, we acknowledge that in the future, gyrogroups with non-isometric gyrations might be introduced. This remains an open question and could present interesting challenges for extending the current framework of GyroBN.

References

[a] Batch normalization: Accelerating deep network training by reducing internal covariate shift

[b] Differentiating through the fréchet mean

We thank Reviewer $\textcolor{blue}{A3ej}$ for the thoughtful comments. We respond to the concerns as follows.😄

1. The Applicability of GyroBN.

We have discussed the limitation of GyroBN regarding its inapplicability to non-gyrogroup manifolds at the beginning of the appendix. However, GyroBN already supports seven non-trivial gyrogroups, as shown in Tab. 2, as well as all trivial gyrogroups, including various matrix Lie groups and Euclidean space. Furthermore, several previous Riemannian batch normalization methods are special cases of GyroBN, as discussed in Sec. 5. We hope that our work will advance the development of deep networks for data with non-trivial geometries.

2. Additional experiments: HNN++ [a] and RResNet [b] could benefit from our GyroBN.

Following the reviewer's constructive suggestion, we further conduct experiments on the two mentioned hyperbolic baselines, HNN++ and RResNet. We use the official code to reimplement the experiment.

Experiments on HNN++

We evaluate HNN++ with and without RBN-H [c] and our GyroBN-H on the Cora, Disease, and Airport datasets for the link prediction task. Like the HNN experiments, we used a two-layer HNN++ backbone as the encoder, with hyperbolic hidden feature dimensions set to 128. Tab. A reports the five-fold average results, leading to the following findings:

• GyroBN can improve and stabilize HNN++. While RBN-H could degrade HNN++’s performance, particularly on the Disease dataset, GyroBN consistently improves it. Additionally, we observed that HNN++ could be sensitive to weight decay. For instance, on the Cora dataset, setting the weight decay to $1e^{-3}$ reduces the performance to 62.55, prompting us to use zero weight decay for HNN++ on this dataset. In contrast, HNN++ with GyroBN is robust to weight decay. Furthermore, HNN++-GyroBN achieves the smallest performance variance across all three datasets, indicating that GyroBN facilitates more robust learning.
• GyroBN Accelerates Convergence. On the Cora dataset, HNN++ requires over 400 epochs to converge. In contrast, HNN++-GyroBN converges in approximately 150 epochs, significantly improving convergence.

Table A: Comparison of HNN++ with or without RBN-H and GyroBN-H on the Cora, Disease, and Airport datasets.

Experiments on RResNet

We focus on the RResNet Horo [b], which leverages a vector field induced by the horosphere projection feature map. Following the settings of HNN and HNN++, we use 2 residual blocks as the backbone. For a more comprehensive comparison, we evaluate hidden feature dimensions ranging from 8 to 128. Tab. B reports the five-fold average results on the Cora dataset, leading to the following findings:

GyroBN consistently improves performance. While RBN performs well only under the 128-dimensional setting, it fails under other dimensions, performing worse than the baseline RResNet. In contrast, GyroBN consistently improves RResNet’s performance across all hidden feature dimensions.

GyroBN stabilizes performance and preserves representation power. Both RResNet and RResNet-RBN exhibit relatively big fluctuations in performance across different hidden dimensions. However, RResNet-GyroBN shows a more stable performance. This indicates that our GyroBN could maintain the representation power of the backbone network.

Table A: Comparison of RResNet with or without RBN-H and GyroBN-H under different hidden feature dimensions on the Cora dataset.

Remark: The above discussion has been added to the revised submission (App. F. 6.)

References

[a] Hyperbolic neural networks++

[b] Riemannian residual neural networks

[c] Differentiating through the fréchet mean

Thank you for your response. The experimental results seem rather promising.

Overall, I'll raise my score to an 8. I would be more comfortable with a 7 (a weak accept versus 8 accept, 6 marginal accept), but that does not seem to be an option.

We thank Reviewer $\textcolor{green}{MrZQ}$ for the careful review and the suggestive comments. Below, we address the comments in detail.😄

1. GyroBN on other hyperbolic baselines.

We have conducted additional experiments during the rebuttal, including evaluations of GyroBN on two hyperbolic baselines: HNN++ [a] and Riemannian ResNet (RResNet) [b]. The results demonstrate that GyroBN improves and stabilizes the backbone networks while accelerating convergence. For more details, please refer to the 2nd response to $\textcolor{blue}{A3ej}$ or App. F.6.

2. Expanded preliminaries.

We thank the reviewer for their constructive comments. In the revised submission, we have expanded the preliminaries by adding a subsection to introduce the basics of Riemannian geometry. To avoid the burden of symbols, we explain Riemannian operators—such as geodesics, the Riemannian log, the Riemannian exp, and parallel transport—through their geometric reinterpretations of Euclidean vector operations. Additionally, we also recap Lie groups and include several simple examples for better understanding.

3. Adding high-level ideas into the introduction.

The key operations in Euclidean BN consist of centering (vector subtraction), biasing (vector addition), and scaling (vector scalar product). Since gyro spaces are natural extensions of Euclidean spaces, we use gyro subtraction, addition, and scalar product to construct GyroBN. To conduct an in-depth theoretical analysis, we introduce the relaxed pseudo-reductive gyrogroups. To reflect this high-level idea, we have revised the introduction and added the following into L 53 in the revised version:

• We use the gyro addition, subtraction, and scalar product to extend the centering (vector subtraction), biasing (vector addition), and scaling (vector scalar product) in the Euclidean BN into manifolds in a principled manner. For broader applicability and in-depth theoretical analysis, we adapt the existing gyrogroup into a more relaxed structure, termed pseudo-reductive gyrogroup.

4. Intuitive explanations of abstract concepts.

We thank the reviewer for their constructive suggestion. We agree that some concepts in gyrogroups can appear abstract. The best way to understand them is to contrast them with the trivial case of $\mathbb{R}$. In the revised submission, we have added intuitive explanations for several key concepts, including gyrations, the left gyroassociative, the left reduction, and pseudo-reduction, into App. C.4.

Following your suggestion, we also use the Grassmannian as an example to illustrate some concepts. However, due to the cumbersome concrete expressions of the Grassmannian gyro operations, we use notations, such as $\oplus^{\mathrm{Gr}}$, to maintain clarity while still providing the necessary intuition. Additionally, all the involved Riemannian and gyro structures across different manifolds are summarized in Tabs. 2 and 8, offering a comprehensive overview. We hope these make the abstract concepts more accessible to readers.

References

[a] Hyperbolic neural networks++

[b] Riemannian residual neural networks

4. GyroBN effectively reduces condition numbers of weight and network Jacobian matrices

In the GyroGr network, the transformation layer outputs $n \times p$ Grassmannian matrices, whose condition numbers are trivial since all singular values of a Grassmannian matrix are 1. Similarly, for the HNN model, the transformation layer outputs a vector, leading to a trivial condition number as well. Thus, we focus on evaluating the condition numbers of the weight matrices in the hidden transformation layer and the overall network Jacobian matrix. Without loss of generality, we conduct evaluations on Grassmannian GyroGr networks under one block architecture, both with and without different BN layers, using each trained model. The results demonstrate that GyroBN consistently reduces the condition numbers of the weight matrices and the network Jacobian.

Condition numbers of weight matrices in the GyroGr transformation layer

Table B: Statistics of condition numbers for the weight matrices in the GyroGr transformation layer across different normalization methods. As weight matrices have 8 channels, we report their mean, standard deviation (std), minimum, and maximum values. The dimensions of the 8-channel transformation matrices are $83 \times 10$ on the HDM05 dataset and $140 \times 10$ on the other datasets.

As shown in Tab. B, GyroBN consistently reduces the condition numbers of the weight matrices across datasets. Interestingly, ManifoldNorm achieves the lowest condition numbers. However, its performance is even worse than the GyroGr baseline, suggesting that excessively reducing the condition number might limit the model’s capacity to capture sufficient expressive features.

Condition number of network Jacobian (output w.r.t. input)

Table C: Condition numbers of network jacobian (output w.r.t. input) across different normalization methods. The statistics come from 100 random samples. The dimensions of Jacobian matrices are $117 \times 930$ on the HDM05 dataset and $11 \times 9600$ on the NTU120 dataset.

We further analyze the condition number of the network Jacobian on the HDM05 and NTU120 datasets. We randomly select 100 samples to calculate the statistics of the associated network condition numbers, as shown in Tab. C. Our GyroBN consistently lowers condition numbers compared to other methods across both datasets, indicating that GyroBN stabilizes network training. Additionally, the reduced maximum condition numbers suggest that GyroBN avoids extreme outliers in conditioning, further highlighting its advantage over prior methods. Interestingly, on the HDM05 dataset, both ManifoldNorm and RBN increase the network conditioning.

We thank Reviewer $\textcolor{purple}{oCvr}$ for the encouraging feedback and the constructive comments! In the following, we respond to the concerns point by point. 😄

1. The empirical superiority of GyroBN stems from its unique ability to control sample statistics.

Controlling sample statistics is a crucial factor behind the success of Euclidean batch normalization, a capability absent in several previous Riemannian batch normalization methods. In contrast, our proposed GyroBN effectively extends this functionality to the gyro space, as established in Thm. 4.1.

2. Gyrodistance is identical to the geodesic distance within all gyrogroups in Tab. 2, commonly used in machine learning.

While there is no general guarantee that gyrodistance is identical to geodesic distance across all gyrogroups, our Prop. 3.6 demonstrates that the two distances are identical for the gyrogroups listed in Tab. 2. These include three SPD, two Grassmannian, and two constant curvature space gyrogroups, all of which are frequently encountered in machine learning applications. For these gyrogroups, the Fréchet mean, Fréchet variance, and running mean update under the gyrodistance are equivalent to their geodesic counterparts.

Furthermore, if there will be a gyrogroup in machine learning, whose gyrodistance is different from its geodesic ones, we could also expect the rationality of gyrodistance. As the data distribution in the GyroBN (Fréchet mean and variance) are defined by the gyrodistance, our GyroBN can still normalize the data distribution in the sense of gyro calculus. As gyro operations intuitively help gyro networks (mentioned by reviewer A3ej), we also expect our GyroBN can be a compelling algorithm. However, at present, we have not encountered such gyrogroups in machine learning.

3. Numerical experiments show that GyroGr and HNN could bring covariate shifts.

Due to the complexity of the manifold, Riemannian networks usually do not maintain the data distribution, highlighting the utility of our GyroBN. To better analyze covariate shifts, we conduct numerical experiments over the Grassmannian and hyperbolic networks.

• Grassmannian: The transformation and pooling layers can be expressed as $f _{trans}: \mathrm{Gr}(p,n) \rightarrow \mathrm{Gr}(p,n),$ $f _{pooling}: \mathrm{Gr}(p,n) \rightarrow \mathrm{Gr}(p,n/2),$ (assuming $n$ is even).

  Since the ONB Grassmannian $\mathrm{Gr}(p, n)$ is a quotient manifold and pooling changes dimensions, we use the geodesic distance between the batch mean and the identity element $I _{p, n} = (I _p, \mathbf{0})^\top \in \mathbb{R}^{n \times p}$ as a consistent measure. Note that the geodesic distance on $\mathrm{Gr}(p, n)$ is bounded by $\sqrt{p}\frac{\pi}{2}$ [a, Thm. 8].

  We generated 30 random Grassmannian matrices of size $100 \times 10$ with an initial batch mean as the identity element $I _{p, n}$. We denote the resulting batch mean after transformation or pooling as $M _{out}$. If the distribution were preserved, the geodesic distance $d(M _{out}, I _{p, n})$ (or $d(M _{out}, I _{p, n/2})$ for pooling) would be zero. Tab. A shows that $M _{out}$ significantly deviates from $I _{p, n}$, indicating a covariate shift.

Table A: Ten-fold results for geodesic distance $d(M _{out}, I _{p, n})$ and shift $\Delta = \frac{d(M _{out}, I _{p, n})}{b} \times 100$, where $b = \sqrt{p} \frac{\pi}{2}$.

• Hyperbolic Space: We examine covariate shift in the transformation layer (Möbius matrix-vector multiplication [b, Lem. 6]) in HNN. We focus on the canonical Poincaré ball (curvature $K=-1$). We randomly generate 30 5-dimensional Poincaré vectors. The batch mean vectors of input and output in the transformation layer are:

The geodesic distance between $M_{in}$ and $M_{out}$ is $0.55$, indicating the covariate shift in HNN. We agree that analyzing the covariate shift is helpful to understand the impact of BN. We will add the above discussion to the supplementary.

Thanks for the instant reply and thoughtful reviews. We have added the discussion on covariate shift and condition number in App. F.7 and F. 8, respectively. These are included in the revised version of the paper. 😊