Intrinsic Riemannian Classifiers on the Deformed SPD Manifolds: A Unified Framework

We thank $\textcolor{blue}{ur4M}$ （$\textcolor{blue}{R4}$）for the careful review and the suggestive comments. Below, we address the comments in detail. Notice that as we have revised our manuscript, references to our paper here always denote its revised version, unless otherwise stated.

---

1. Results of Riemannian multinomial logistic regression (RMLR) on other manifolds.

Recalling our RMLR in Eq. (14), three operators are involved: Riemannian metric, logarithm, and parallel transportation (or left translation). For specific manifolds, one only needs to put these three operators into Eq. (14). We further implement our RMLR on $\mathrm{SO}(n)$. Please refer to CQ#2 in the common response.

2. The distance is defined twice: in Eq (8) and Eq (11).

The distance in Eq. (11) is different from Eq. (8). The distance in Eq. (11) is the one used in this paper. For better clarity, we have change the $d(\cdot)$ in Eq. (8) as $\tilde{d}(\cdot)$.

3. It can be directly derived from Eq (3) by parametrizing $b _k$ as $\langle p_k, x \rangle$ and then interpreting the subtraction as a Riemannian logarithm.

Thanks for your insightful comment. We can indeed directly interpret subtraction as the Riemannian logarithm and obtain the RMLR in Eq. (14). However, this direct re-interpretation is only an intuitive re-interpretation and has little theoretical support, as it involves no margin distance or margin hyperplane. However, the margin distance and hyperplane play an important role in designing RMLR, such as the ones on hyperbolic[1] and spherical manifolds [2].

Our insight is that RMLR in Eq. (14) is derived from the Riemannian margin distance (Def. 3.3) and hyperplane (Eq. (7)). In this way, it is precise to call Eq. (14) as RMLR.

We assume your concerns might come from our original structure of Sec. 3.2. To obtain RMLR, the margin distance in Def. 3.3 should be first solved. In the original submission, we put the results on margin distance in the appendix and directly presented our RMLR in the main paper. We have added a theorem about the solution of margin distance (Thm. 3.4) between Def. 3.3 and RMLR (Thm. 3.5) in the revised paper.

4. RMLR has already done in [3].

Our work differs with [3] both in theory and practice, and further covers all the results in [3].

Theory: Although our RMLR (Eq. (14)) has a similar expression with [3, Eq. (18)], the underlying theories are significantly different. All the results in [3], including Eq. (18) in [3], are confined within SPD manifolds under the metric pulled back from the Euclidean space. However, numerous metrics are not pullback metrics from the Euclidean space, such as BWM, AIM on SPD manifolds [4], the invariant metric on matrix Lie groups [5], and the metrics on the Grassmann and Stiefel manifolds [6]. [3] fails to construct MLR under these metrics. In contrast, our RMLR is more general and can deal with non-flat metrics. Throughout the proof of Thms. 3.4- 3.5, our RMLR only requires geodesic connectedness. Diverse metrics or manifolds in machine learning satisfy this property. Therefore, our RMLR has broader application scenarios.

Pratice: We cover all the results in [3] and extensively study diverse SPD MLRs under deformed metrics. In detail, [3] only implements SPD MLR under $(\alpha,\beta)$-LEM, while our work implements five families of deformed metrics. Besides, as we discussed in Rmk 4.4, when $\theta=1$, our SPD MLR under $(1,\alpha,\beta)$-LEM covers the SPD MLR in [3].

5. More precise contributions of the paper.

This work presents a general framework for building Riemannian classifiers on manifolds and specifically showcases our framework on the deformed SPD manifolds. Please refer to CQ#1 in the common response for a detailed explanation.

I thank the authors for all the answers. I read the new version of the paper and it is now much clearer. I am still a little bit disappointed by the numerical experiments where the effects of the metric could have been shown with figures (heat maps, line plots, ...) instead of tables.

Overall, the paper is original and can bring value to the community of machine learning on Riemannian manifolds. Therefore, I raised my rating to a 6.

We thank $\textcolor{blue}{ur4M}$ ($\textcolor{blue}{R4}$) for raising the score, and we are delighted that you appreciate the value of our work.😄

1. Visualization of experimental results.

Thanks for your suggestive comments. Following your suggestion, we visualize histograms of our experimental results on the HDM05 and Radar datasets. Please check G.4 in the appendix. If our paper gets accepted, we will move some experimental settings into the appendix and put these histograms into the main paper.

We thank $\textcolor{green}{Reviwer Qd7L}$ ($\textcolor{green}{R3}$) for the instant reply and thoughtful comments. We respond to the questions point by point as follows. As we have revised our manuscript, references to our paper here always denote its revised version.

---

1. “unified framework” --> "general framework".

Thanks for your suggestive comments. Our RMLR in Eq. (14) is a general framework, as it only requires the existence of Riemannian logarithm $\mathrm{Log} _P Q$ for any pair $P, Q$ in the manifold $\mathcal{M}$. We call this property geodesic connectedness, and diverse manifolds in machine learning satisfy this property. Therefore, our framework can be implemented beyond SPD manifolds. We have extended our framework into the Lie group. Please refer to CQ#2 in the common response for more details.

Although most of the existing manifolds in machine learning are geodesic connectedness, we do not know if, in the future, there would be a powerful metric with the unknown Riemannian logarithm. Therefore, as you said, "unified" might be a bit overly absolute. We have changed the word "unified" to "general".

2. Experimental analyses should be extended with additional datasets and backbones.

Our RMLR in Eq. (14) can readily apply to other manifolds or metrics. Please refer to CQ#2 in the common response for the experiments on $\mathrm{SO(n)}$.

3. Comparative analysis of complexity (memory and running time).

Memory cost: Recalling Eq. (14) and Tab. 2, each class $k$ requires an SPD parameter $P _k$ and Euclidean parameter $A _k$ in SPD MLR. Therefore, the memory cost of the SPD MLR under each metric is the same.

Computational cost: The key factor of the computational complexity of SPD MLRs under different metrics lies in the number of matrix functions, such as matrix power, logarithm, Lyapunov operator, and Cholesky decomposition. These matrix functions are divided into two categories: one is based on eigendecomposition, and the other is the Cholesky decomposition. The following table summarizes the number of matrix functions in SPD MLR. Therefore, the general efficiency of SPD MLR should be EM>LEM>LCM>AIM>BWM. Note that Cholesky decomposition is more efficient than eigendecomposition. Without deformation, the order should be EM>LCM>LEM>AIM>BWM. We report the average training time per epoch in Tab. 10 in App. G.3. As shown in Tab. 10, EM-, LEM-and LCM-based SPD MLR is more efficient than AIM- and BWM-based MLR.

Table A: Number of matrix functions for each class $k$ in different SPD MLRs. a (b) means the number of matrix functions in the SPD MLR under the deformed (standard) metric.

We thank $\textcolor{brown}{Reviwer fSLb}$ ($\textcolor{brown}{R2}$) for the constructive suggestions and insightful comments! In the following, we respond to the concerns in detail. Notice that as we have revised our manuscript, references to our paper here always denote its revised version.

1. Our work and contributions are significantly different from Nguyen & Yang (2023) [1] and Thanwerdas & Pennec (2019a; 2022a) [2-3].

We have briefly discussed the difference in Rmks. 3.6 and 4.4. Here, we present a more detailed explanation.

Difference with gyro SPD MLR [1]

Our work differs from gyro SPD MLR [1] both in theory and practice.

Theory: Our framework uses no gyro structure and is more general than gyro SPD MLR.

Firstly, the gyro MLR is based on gyro structures. However, not all metrics can induce gyro structures. In contrast, our method only requires Riemannian logarithm and parallel transportation (or group translation), which usually exists in manifolds in machine learning. Our method is thus more general and can be applied to more metrics.

Secondly, even if the gyro structure exists, the margin distance to the hyperplane in Gyro MLR is based on gyro distance and gyro trigonometry [1, Def. 2.22]. Note that gyro distance is defined by the tangent space at the identity [1, Def. 2.15]. However, the most natural counterparts on manifolds are geodesic distance and Riemannian trigonometry (Eq. 12). Therefore, we directly use geodesic distance and Riemannian trigonometry to obtain the margin distance Eq. (13).

Thirdly, Gyro MLR is obtained case-by-case for a specific gyro space, since the pseudo-gyrodistance [1, Def. 2.22] should be specifically solved for each gyro space. On the contrary, our work solves the margin distance in a general form (Thm. 3.4), and presents a general framework in Thm. 3.5. For a specific metric, one only needs to put the Riemannian logarithm and parallel transport into Eqs. (14-15). These operators exist in most of the manifolds in machine learning.

Practice: Our work implements five families of deformed SPD MLRs (Tab. 2), while gyro SPD MLR only implements gyro SPD MLRs under the standard AIM, LEM, and LCM. Besides, as we discussed in Rmk. 4.4, our SPD MLRs incorporate the results presented in [1]. Moreover, the BWM and EM are geodesically incomplete, while gyro operations require geodesic completeness [1, Eqs. (1-2)]. Therefore, EM and BWM could be problematic to induce gyro structures, as could gyro SPD MLRs. On the contrary, SPD MLRs can be easily obtained by our Thm. 3.5 for these two metrics. The following table briefly summarizes the difference between our SPD MLRs and gyro SPD MLRs. In addition, our RMLR in Thm. 3.5 can also be implemented into other manifolds, such as $\mathrm{SO}(n)$. Please refer to CQ#2 in the common response for our MLR in Lie groups.

Tab. 1 Difference between our SPD MLRs and the gyro SPD MLRs.

Difference with [2-3]

Although power deformation is discussed in [2-3], they only cover the deformation of $(\alpha,\beta)$-AIM, BWM, and the standard EM. Our paper further generalized $(\alpha,\beta)$-LEM, LCM, and $(\alpha,\beta)$-EM into deformed metrics (see Sec. 4.1). Besides, we not only discuss the properties of all the deformed metrics in Fig. 1 and Tab. 1, but also validate the efficacy of deformation. Experimental results shown in Tabs. 3-6 indicates the effectiveness of the deformed metrics.

2. Tabs. 3-6 is too small to read

Thanks for the suggestion. After our work gets accepted, we will divide each table into two lines and move some stuff to supplementary.

3. Limitations are not discussed.

The main limitation of our work is the over-parameterized $P _k$ in our RMLR. Recalling our RMLR in Eq. (14), each class would require an SPD parameter $P _k$ and a Euclidean parameter $A _k$. Consequently, as the number of classes grows, the classification layer would become burdened with excessive parameters. We have added a discussion on this limitation in App. A and will address this problem in our future work.

Thanks for the instant reply and insightful comments! According to your comments, we have revised our paper to ensure our claims are more precise. We address your concerns in the following.

1. Geodesic connectedness --> The existence of the Riemannian logarithm

Thanks for your insightful comments. We carefully read the literature you present. In the first literature [1], geodesic connectedness refers to whether each two points in the manifold can be joined by a geodesic, which is a bit different from our previous definition (our previous definition is defined by a unique geodesic).

Nevertheless, the ultimate reason for presenting this definition is that we want a short name to describe the existence of the Riemannian logarithm. We have changed the "geodesic connectedness" into "the existence of the Riemannian logarithm" throughout the paper for better clarity.

• Why we require the existence of the Riemannian logarithm.

Recall the reformulation of MLR Eq. (6-7):

$$p(y=k \mid S) \propto \exp (\operatorname{sign}(\langle \tilde{A} _k, \mathrm{Log} _{P _k}(S) \rangle _{P _k})\|\tilde{A} _k\| _{P _k} \tilde{d} (S, \tilde{H} _{\tilde{A} _k, P _k})),$$ $$\tilde{H} _{\tilde{A} _k, P _k} = \{S \in \mathcal{M}: g _{P _k}( \mathrm{Log} _{P _k} S, \tilde{A} _k) = \langle \mathrm{Log} _{P _k} S, \tilde{A} _k \rangle _{P _k}=0\}.$$

The minimal requirement is the existence of Riemannian logarithm $\mathrm{Log} _{P _k}(S)$ for each $k$. Without the existence of the Riemannian logarithm, even the hyperplane is ill-defined. However, previous work further relies on additional properties to realize MLR in manifolds, such as the generalized law of sines and gyro structures. In contrast, throughout the proof of Thm. 3.3-3.4, we only require the existence of the Riemannian logarithm.

2. Applicability

As we clarified, the only requirement is the existence of the Riemannian logarithm. Many metrics or manifolds on machine learning indeed satisfy this property, including the five families of metrics discussed in this paper on SPD manifolds, the Lorentz model on hyperbolic manifolds [2, Eq. (11)], spherical manifolds [3, Sec. 3.1.1], and Lie groups [4, Tab. 1].

In our main paper, we have implemented our framework under five families of metrics on SPD manifolds. Besides, we further implement our framework on $\mathrm{SO}(n)$. These applications should justify the applicability of our approach.

3. The Grassmann

Thanks for your insightful comments on the cut locus of the Grassmann. On the Grassmann $\mathrm{Gr} _{n,p}$, for $P, Q \in \mathrm{Gr} _{n,p}$, the Riemannian logarithm $\mathrm{Log} _{P} (Q)$ exists for $Q \in \mathrm{Gr} _{n,p} \backslash \mathrm{Cut}_P$ [5, Props. 5.6], where $\mathrm{Cut}_P$ is the cut locus of $P$.

As we mentioned above, if $S$ is in $\mathrm{Cut} _{P _k}$, the hyperplane and MLR do not exist. In contrast, assuming $S$ is not in $\mathrm{Cut} _{P _k}$ for each $k$, our Thm3. 3.3-3.4 still holds. In [6], the author also makes a similar assumption when dealing with the logarithm on the Grassmann [6, Sec. 3.2, P4]. However, how to ensure or transform $S$ outside $\mathrm{Cut} _{P _k}$ is out of the scope of this work. We will explore this possibility in the future.

[1] https://arxiv.org/pdf/math/0005039.pdf

[2] Liu Q, Nickel M, Kiela D. Hyperbolic graph neural networks.

[3] Chakraborty R. Manifoldnorm: Extending normalizations on Riemannian manifolds.

[4] Boumal N, Absil P A. A discrete regression method on manifolds and its application to data on SO (n).

[5] Bendokat T, Zimmermann R, Absil P A. A Grassmann manifold handbook: Basic geometry and computational aspects.

[6] Nguyen X S. The Gyro-Structure of Some Matrix Manifolds.

We thank $\textcolor{red}{Reviwer tGbz}$ ($\textcolor{red}{R1}$) for the valuable comments. Below is our detailed response.

---

1. Novelty and Contributions

This paper presents a general framework for building Riemannian Multinomial Logistic Regression (RMLR) on manifolds, and specifically showcases five implementations on SPD manifolds. Please refer to CQ#1 in the common response for detailed discussions about our contributions and novelty.

2. Using J. Cavazza, A. Zunino, M. San Biagio, and V. Murino, “Kernelized covariance for action recognition,” in Pattern Recognition (ICPR), 2016 23rd International Conference on. IEEE, 2016, pp. 408–413. Eman A. Abdel-Ghaffar, Yujin Wu, Mohamed Daoudi, Subject-Dependent Emotion Recognition System Based on Multidimensional Electroencephalographic Signals: A Riemannian Geometry Approach. IEEE Access 10: 14993-15006 (2022)

We have added these two papers in the reference of our revised version. Since our SPD MLRs are independent of network structures, our MLRs should be able to be applied to classify the SPD features in these two papers without any problems. We will explore this possibility in the future.

CQ#2: RMLR on $\mathrm{SO}(n)$ ($\textcolor{brown}{R2}$, $\textcolor{green}{R3}$, and $\textcolor{blue}{R4}$)

Our RMLR (Eq. (14)) can readily apply to other manifolds. We focus on LieNet [8], where the latent space is $\mathrm{SO}(3)$. We call RMLR in $\mathrm{SO}(3)$ as LieMLR.

Theoretical ingredient

To implement RMLR (Eq. (14)+Eq. (15).), we need the Riemannian metric, Riemannian logarithm, and parallel transportation, which can be found in [9, Tab. 1] and [10, Sec 3.2.1]. We denote $\log$ as the matrix logarithm and skew-symmetric matrices as $\mathfrak{s o}(n)$, which is the Lie algebra of $\mathrm{SO}(n)$. Supposing $S,P \in \mathrm{SO}(n)$, $V _1,V _2, \in T _P \mathrm{SO}(n)$ and $V \in \mathfrak{s o}(n)$, we present all the necessary ingredients in Tab. 1.

Table 1: Riemannian operators required in LieMLR on SO(n)

According to Thm. 3.5, the RMLR on $\mathrm{SO}(n)$ is given as

$$p(y=k \mid S ) \propto \exp ( \langle \mathrm{Log} _{P_k} S, \Gamma _{I \rightarrow P _k} ({A}_k) \rangle _{P_k})= \exp (\langle \log(P ^\top _k S), A _k \rangle)$$

where $k$ denotes $k$-th class, $S \in \mathrm{SO}(n)$ is an input $\mathrm{SO}(n)$ feature, and $P _k \in \mathrm{SO}(n)$ and $A_k \in \mathfrak{so}(n)$

Implementation details and dataset

As LieNet was originally implemented by Matlab, we use the open-sourced Pytorch code [11] to reimplement LieNet by torch. We found that the matrix logarithm in the Pytorch code [11] fails to deal with singular cases well. Hence, we use Pytorch3D [12] to calculate the matrix logarithm. For the LieMLR layer, $P _k$ is optimized by the Riemannian SGD on $\mathrm{SO}(3)$, while $A _k$ is optimized by Euclidean SGD.

In the original LieNet, classification is carried out by Euler axis and angle representation. We substitute this layer with our LieMLR. We call LieNet with our LieMLR as LieNetLieMLR. Following LieNet, we adopt the G3D-Gaming dataset [13], and follow all the settings of LieNet-3blocks in the original paper of LieNet. Similar to LieNet, we train the network by the standard cross-entropy loss.

Results

The results are presented in the following table. Due to different software, our reimplemented LieNet is slightly worse than the performance reported in [8]. However, we still can observe the improved performance of LieNetLieMLR over LieNet.

Table 2: Results on LieNet with or without LieMLR

[1] Ganea O, Bécigneul G, Hofmann T. Hyperbolic neural networks.

[2] Nguyen X S, Yang S. Building Neural Networks on Matrix Manifolds: A Gyrovector Space Approach.

[3] Chen Z, Song Y, Liu G, et al. Riemannian Multiclass Logistics Regression for SPD Neural Networks.

[4] Thanwerdas Y, Pennec X. O (n)-invariant Riemannian metrics on SPD matrices.

[5] Lin Z. Riemannian geometry of symmetric positive definite matrices via Cholesky decomposition.

[6] Hartley R, Trumpf J, Dai Y, et al. Rotation averaging[J]. International journal of computer vision.

[7] Thanwerdas Y, Pennec X. The geometry of mixed-Euclidean metrics on symmetric positive definite matrices.

[8] Huang Z, Wan C, Probst T, et al. Deep learning on lie groups for skeleton-based action recognition.

[9] Boumal N, Absil P A. A discrete regression method on manifolds and its application to data on SO (n).

[10] Chakraborty R. Manifoldnorm: Extending normalizations on Riemannian manifolds.

[11] https://github.com/hjf1997/LieNethttps://github.com/hjf1997/LieNet

[12] Ravi N, Reizenstein J, Novotny D, et al. Accelerating 3d deep learning with pytorch3d.

[13] Bloom V, Makris D, Argyriou V. G3D: A gaming action dataset and real time action recognition evaluation framework.

---

For brevity, we refer to reviewers $\textcolor{red}{tGbz}$ as $\textcolor{red}{R1}$, $\textcolor{brown}{fSLb}$ as $\textcolor{brown}{R2}$, $\textcolor{green}{Qd7L}$ as $\textcolor{green}{R3}$, $\textcolor{blue}{ur4M}$ as $\textcolor{blue}{R4}$, respectively.