A Lie Group Approach to Riemannian Batch Normalization

We thank Reviewer $\textcolor{blue}{R6Kt}$ ($\textcolor{blue}{R4}$) for the thoughtful comments. We respond to the concerns as follows.

1. Difference with the prior work [Kobler+22a] and [Chakraborty+20].

The theory and implementation of our LieBN are different from [Kobler+22a] and more general than [Chakraborty+20]. Please refer to CQ#1 in the common response.

2. Computational details on back-props

As summarized in the following table, two types of matrix functions are involved in our LieBN on SPD manifolds: one is based on Eigendecomposition, and the other is the Cholesky decomposition. The Eigen-based matrix functions include matrix logarithm, exponential, and power. Their BP can be calculated by the Daleckii-Krein formula [2 Thm. V.3.3]. The BP of the Cholesky decomposition has been well established in torch.linalg.cholesky. Therefore, we use torch.linalg.cholesky for the Cholesky decomposition. We have added a detailed discussion on matrix BP in the App. F in our revised manuscript.

Table A: Matrix functions involved in LieBN on SPD manifolds

3. Large std on EEG

Recalling Tab. 4-5, it is only on the EEG datasets that the performance shows relatively large std, which should be attributed to the characteristics of the EEG dataset. EEG signals exhibit low signal-to-noise ratio (SNR), domain shifts, and low specificity. These challenges are more obvious in the inter-session and inter-subject scenarios.

Despite the relatively large std, our LieBN outperforms the SPDDSMBN on the inter-subject classification by 3.87% (50.10 v.s. 53.97). Besides, our LieBN can show less std than the SPDDSMBN baseline on EEG datasets. For instance, for the inter-subject task (Tab. 5 (b)), the std of LCM-based LieBN is clearly less than SPDDSMBN ( 51.53±4.96 v.s. 50.10±8.08).

4. Discussion and analysis about the choice of metrics.

The foremost challenge when building Riemannian learning algorithms is choosing an appropriate Riemannian metric. If the framework is general, there would be many metrics to select from, and the model would be more likely to perform better. This is why we view the generality of our framework as an advantage.

Generally speaking, AIM is the best candidate metric on SPD manifolds. The most significant reason is the property of affine invariance, which is a natural characteristic of describing covariance matrices. In our experiments, the LieBN-AIM generally achieves the best performance. In some scenarios, there are better choices than AIM. As shown in Tab. 4 (b), the best result on the HDM05 dataset is achieved by LCM-based LieBN, which improves the vanilla SPDNet by 11.71%. Therefore, when choosing Riemannian metrics on SPD manifolds, a safe choice would start with AIM and extend to other metrics.

We thank reviewer $\textcolor{green}{LaGM}$ ($\textcolor{green}{R3}$) for the valuable comment. In the following, we respond to the concerns in detail.

1. Difference from the previous RBN methods.

The RBNs in these three works are either designed for a specific manifold or metric, or fail to control mean and variance, while our LieBN fulfills normalization on general Lie groups. We have summarized their limitations in Tab. 2 and discussed the difference in Sec. 3.2. Here we clarify the difference between our LieBN and their RBN in details.

- Difference with the RBN proposed in [1]

The RBN in [1] is confined within the standard Affine-Invariant Metric (AIM). In contrast, we implement our LieBN under three invariant metrics: Log-Euclidean Metric (LEM), Log-Cholesky Metric (LCM), and AIM. Moreover, our LieBN further covers the deformed families of LEM, LCM, and AIM.

For the specific AIM, our LieBN is also better than the RBN [1] in terms of computational efficiency. The RBN in [1] is based on parallel transportation, Riemannian exp and log. The specific equation is calculated by the matrix power function (Eq. (10) in [1]). In contrast, our LieBN is based on Lie group left translation, which is fulfilled by Cholesky decomposition (Tab. 3). Note that Cholesky decomposition is more efficient than the matrix power function. As shown in Tab. 5 (b), DSMLieBN-AIM-(1) achieves similar performance to SPDDSMBN with less training time (6.94 v.s. 7.74).

Besides, the RBN in [1] is problematic when extending to other manifolds or metrics. As shown in Eq. (9), the core formulation of the RBN in [1] can be described as

Since $\Gamma_{P \rightarrow Q}$ becomes affine action under AIM, this formulation can coincidentally control mean under AIM on SPD manifolds. However, the resultant mean and variance are generally agnostic under general manifolds. Therefore, the RBN based on $\Gamma _{P \rightarrow Q}$ could not be extended into other metrics or manifolds. On the contrary, our Props. 4.1-4.2 guarantee the rationality of our LieBN on other Lie groups, such as SO(n). For experiments on SO(n), please refer to the 3rd response to $\textcolor{red}{R1}$.

- Difference with the RBN proposed by [3]

Similar to [1], the RBN in [3] generally fails to normalize mean or variance either, as this RBN is a variant of $\Gamma_{P \rightarrow Q}$. In contrast, our LieBN can always control both mean and variance on Lie groups.

- Difference with the RBN proposed by [4]

The RBN on matrix Lie groups in [4] is also confined within a specific metric and group structures (Sec. 3.2 in [4]). On the contrary, our LieBN is designed for general Lie groups. In this sense, the RBN on matrix Lie groups in [4] is a special case of our LieBN. Furthermore, as stated at the end of Sec. 4, our LieBN framework can be extended into right-invariant metrics.

Besides, as stated in Rmk. C.5 in the appendix, the proof presented in [4] is a bit problematic, as the author fails to consider Riemannian volume when dealing with the probability density function. In contrast, our theoretical results in Props. 4.1-4.2 are more solid and general.

- Summary

In summary, the RBNs in [1-3] are either confined within a specific manifold or metric, or fail to control mean and variance. In contrast, we propose a framework of batch normalization over general Lie groups and showcase its generality over three families of Lie groups on SPD manifolds.

[1] Kobler R, Hirayama J, Zhao Q, et al. SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG.

[2] Brooks D, Schwander O, Barbaresco F, et al. Riemannian batch normalization for SPD neural networks.

[3] Lou A, Katsman I, Jiang Q, et al. Differentiating through the fréchet mean.

[4] Chakraborty R. Manifoldnorm: Extending normalizations on Riemannian manifolds.

We thank reviewer $\textcolor{red}{ZRPd}$ ($\textcolor{red}{R1}$) for the valuable feedbacks. Below is our detailed response.

1. The proposed technique is not a simple tweak of those from Kobler et al. (2022b), Lou et al. (2020), and Chakraborty (2020).

As stated in CQ#1 of the common response, our work is entirely theoretically different from Kobler et al. (2022b) and Lou et al. (2020), and more general than Chakraborty (2020). Please refer to the common response for details.

2. The experimental results are convincing, given the only modification to SPDNet is the BN layer.

This paper mainly focuses on building LieBN on general Lie groups. Since we only change the model by adding a single BN layer, this slight modification is not expected to outperform every baseline. Nevertheless, our experiments on the SPD manifolds indicate that our LieBN can improve the performance of SPD neural networks. On the Radar, HDM05, FPHA, and EEG datasets, our LieBN improves the SPD baselines by 2.22%, 11.71%, 4.8% and 3.87%, respectively. This is convincing to claim that SPD neural networks can benefit from our LieBN.

Besides, SPDDSMBN is one of the SOTA methods in EEG classification. Tab. 5 indicates the superiority of our work against SPDDSMBN. Moreover, Tab. 5 shows that LCM- or LEM-based LieBN can achieve comparable performance against SPDDSMBN with less training time. For instance, for inter-subject classification, LCM-based LieBN reaches similar results to SPDDSMBN while costing only half of the training time. Besides, LCM-based LieBN tends to show smaller std than SPDDSMBN (4.96 v.s. 8.08 for the inter-subject task)

3 LieBN on LieNet (Huang et al., 2017)

The Lie group in LieNet [1] is SO(3). Our LieBN can also be implemented in this Lie group. However, we only implement centering and biasing operations for now due to limited time.

Theoretical ingredient

Since we only implement the centering and biasing of LieBN, the core operation of LieBN becomes

where $\{P _i\}$ is a batch of SO(3), $M$ is the SO(3) batch mean，and $B$ is an SO(3) biasing parameters. The batch mean can be obtained by [2, Alg. 1], while the running mean can be updated by the geodesic connecting the running mean and batch mean. The calculation of batch mean [2, Alg. 1] requires Lie group exp & log on SO(3). We present all the necessary ingredients in Tab. 1. For more detail, please refer to [2-3].

Table 1: Riemannian operators required in LieBN on SO(3)

Implementation details and dataset

As LieNet was originally implemented by Matlab, we use the open-sourced Pytorch code [4] to reimplement LieNet by torch. We found that the matrix logarithm in the Pytorch code [4] fails to deal with singular cases well. Hence, we use Pytorch3D [5] to calculate the matrix logarithm, i.e., pytorch3d.transforms.matrix_to_axis_angle. Besides, we implement matrix exponentiation by Rodrigues' formula [6, Eq. (2)].

Following LieNet, we adopt the G3D-Gaming dataset [7], and follow all the learning settings as in the original paper of LieNet. We apply the LieBN layer before each pooling layer. As the RotMap layer in the original LieNet is actually based on left translation as well, we use the LieBN layer to substitute the RotMap layer. Following LieNet, we adopt the architecture of three pooling layers, which can be illustrated as

where $f _{\log}$ and $f _{\mathrm{FC}}$ denote the matrix logarithm and FC layer. Similar to LieNet, we train the network by the standard cross-entropy loss.

Results

The results are presented in the following table. Due to different software, our reimplemented LieNet is slightly worse than the performance reported in [1]. However, we still can observe a clear improvement of LieNetLieBN over LieNet.

Table 2: Results on LieNet with or without LieBN

Thank the authors for the response. I still keep my original rating since I fail to see the significance of this work with respect to existing works.

We thank reviewer $\textcolor{brown}{B3vJ}$ ($\textcolor{brown}{R2}$) for the encouraging feedback and constructive comments! The following is our detailed response.

1. Gaussian distribution by Lie groups thermodynamics.

Thanks for the valuable comment. The Gaussian distribution you mentioned on the SPD manifold is very inspiring. We have added this reference in Sec. 4, where we introduce Gaussian distribution on manifolds. In our future research, we will explore this distribution and attempt to establish statistics learning methods based on this Gaussian distribution.

2. Adding some reference on Riemannian batch normalization.

We have added the reference you mentioned in the main paper. These papers are very interesting, especially HPDNet on Hermitant Positive Definite (HPD) manifolds proposed in [1]. As HPD manifolds are the counterparts of SPD manifolds in the complex domain, HPD manifolds also have Lie group structures. Theoretically, our LieBN framework could also be implemented on HPD neural networks. We will explore these possibilities in the future.

[1] Brooks, D., Schwander, O., Barbaresco, F., Schneider, J. Y., & Cord, M. A Hermitian Positive Definite neural network for micro-Doppler complex covariance processing.

5. Effect of the reduced amount of data.

Following [2], we use 10% of training data to train SPDNet with and without LieBN on the Radar dataset. Following the main paper, we set $\theta=1, 1, -0.5$ and $(\alpha,\beta)=(1,0)$ for $(\theta,\alpha,\beta)$-AIM, $(\theta,\alpha,\beta)$-LEM, and $(\theta)$-LCM. The 10-fold average results are presented in the table below. We could observe a clear improvement of our LieBN to SPDNet under the limited data availability scenarios.

Table 2: Robustness to lack of data of SPDNet with and without LieBN on the Radar dataset.

6. Links between the Fréchet variance and the variance of the Gaussian.

There are some theoretical links between the Fréchet variance and the variance in the Gaussian distribution in Eq. (12).

Recalling the Gaussian distribution (Eq. (12)) on manifold $\mathcal{M}$

As shown in [3, Props. 4.7], $\mathrm{Var}(X)=\mathbb{E} _{\mathbf{X}}\left[\mathrm{d}^2(X, M)\right]=\sigma^2$. Therefore, $\sigma^2$ is the population variance. The empirical counterpart of population variance is the Fréchet variance [4, P12]. As shown by our Props. 4.2, our LieBN can control the sample variance.

The Maximum Likelihood Estimation (MLE) of $\sigma^2$ does not have a general solution, as $k(\sigma)$ depends on the specific metrics. However, the relation is very clear for the invariant metrics on the Lie groups of SPD manifolds. As shown in [5, Props. 7], under AIM, the Fréchet variance $v ^2$ satisfying $\sigma ^2=\phi(v ^2)$ with $\phi$ strictly increasing. For AIM, our LieBN implicitly controls the population variance through the explicit control of sample variance. For the LEM and LCM, our LieBN can directly transfer the latent Gaussian distribution:

This result is obtained by Cor. C.4 in App. C and has been discussed in detail in App. C (Page 17).

7. It should be "neutral element" instead of identity in Eq. (13).

Thanks for pointing this out! All the "identity" or "identity element" in the paper means neutral element. We agree that the current usage of "identity element" and "identity matrix" might lead to potential confusion. In the original submission, we clarified at the beginning of Sec. 4 that an identity element may not necessarily be an identity matrix. Now we have changed the "identity" element to the "neutral" element throughout the manuscript for better clarity.

[1] Thanwerdas Y, Pennec X. Is affine-invariance well defined on SPD matrices? A principled continuum of metrics.

[2] Brooks D, Schwander O, Barbaresco F, et al. Riemannian batch normalization for SPD neural networks.

[3] Chakraborty R, Vemuri B C. Statistics on the Stiefel manifold: theory and applications.

[4] Pennec X. Probabilities and statistics on Riemannian manifolds: A geometric approach.

[5] Said S, Bombrun L, Berthoumieu Y, et al. Riemannian Gaussian distributions on the space of symmetric positive definite matrices.

We thank reviewer $\textcolor{purple}{Urfr}$ ($\textcolor{purple}{R5}$) for the encouraging feedback and the constructive comments!😄😄 In the following, we respond to the concerns point by point.

1. Table readability.

Thanks for the suggestive comments. Due to the page limits, we presented the tables concisely. After our work gets accepted, we will move some experimental settings into the appendix and allocate more space for the tables (possibly allowing for two lines or a large figure).

2. The novelty of the work should emphasized more.

Thanks for your constructive suggestion! We emphasize our novelty in the introduction in the revised paper (Page 2), highlighting our contributions that our work can theoretically normalize mean and variance and apply to diverse Lie groups.

3. How to choose hyper-parameters in LieBN.

Our SPD LieBN has at most three types of hyper-parameters: Riemannian metric, deformation factor $\theta$, and $\mathrm{O}(n)$-invariance parameters $(\alpha,\beta)$. The general order of importance should be Riemannian metric $>$ $\theta$ $>$ $(\alpha,\beta)$.

The most significant parameter is the choice of the Riemannian metric, as all the geometric properties are sourced from the metric. A safe choice would start with AIM, and then decide whether to explore other metrics further. The most important reason is the property of affine invariance of AIM, which is a natural characteristic of covariance matrices. In our experiments, the LieBN-AIM generally achieves the best performance. However, AIM is not always the best metric. As shown in Tab. 4 (b), the best result on the HDM05 dataset is achieved by LCM-based LieBN, which improves the vanilla SPDNet by 11.71%. Therefore, when choosing Riemannian metrics on SPD manifolds, an appropriate choice would start with AIM and extend to other metrics. Besides, if efficiency is an important factor, one should first consider LCM as it is the most efficient one.

The second one is the deformation factor $\theta$. As we discussed in Sec. 5.1, $\theta$ interpolates between different types of metrics ($\theta=1$ and $\theta \rightarrow0$). Inspired by this, we select $\theta$ around its deformation boundaries (1 and 0). In this paper, we roughly select $\theta$ from $\{ \pm 0.5, \pm 1,\pm 1.5 \}$

The less important parameters are $(\alpha,\beta)$. Recalling Tab. 1, $(\alpha, \beta)$ only affects the Riemannian metric tensor and geodesic distance. For our specific SPD LieBN, they only affect the calculation of variance, which should have fewer effects than the above two parameters. Therefore, we simply set $(\alpha,\beta)=(1,0)$ during experiments.

We have added the above discussion to the App. G.

4. $\mathrm{O}(n)$-invariance hyper-parameters $(\alpha, \beta)$

As we stated in the last question above, $(\alpha, \beta)$ is less important than Riemannian metrics and deformation factor $\theta$. Nevertheless, we also conduct experiments on the effect of $(\alpha, \beta)$ here. As $\alpha$ is less important as a scaling factor [1], we set $\alpha=1$ and change the value of $\beta$.

Recalling Eq. (3), $\beta$ controls the relative importance of the trace part against the inner product. Therefore, we set the candidate values of $\beta$ as $\{1,\frac{1}{n},\frac{1}{n^2}, 0, -\frac{1}{n} + \epsilon,-\frac{1}{n^2}\}$, where $n$ is the input dimension of LieBN, and $\epsilon$ is a small positive scalar to ensure $\mathrm{O}(n)$-invariance, i.e., $\min (\alpha, \alpha+n \beta)>0$. $\frac{1}{n^2}$ and $\frac{1}{n}$ means averaging the trace in Eq. (3), while the sign of $\beta$ denotes suppressing (-), enhancing (+), or neutralizing (0) the trace.

We focus on AIM-based LieBN on the HDM05 dataset. We set $\theta=1.5$, as it is the best deformation factor under this scenario. Other network settings remain the same as the main paper. The 10-fold average results are presented in the following table. Note that the dimension $n$ of the input feature of the LieBN layer is 30 in this setting. As expected, $\beta$ has minor effects on our LieBN. We also add this discussion in App. G.1 in our revised paper.

Table 1: The effect of different $\beta$ for AIM-based LieBN on the HDM05 dataset.

1. I am deeply disapointed by this answer: it's the job of authors to deal with the page limit. Promising to (re)move useful parts to the appendix if the paper is accepted is not acceptable.

2. Nice, it's much better by now.

3 and 4) Interesting and worth reading, thanks !