RMLR: Extending Multinomial Logistic Regression into General Geometries

We thank Reviewer $\textcolor{blue}{dBzm}$ for encouraging feedback and valuable comments. Below, we address the comments in detail. 😄

---

1. Details on the parameters learning of the MLR.

Due to page limits, the optimization details are discussed in Apps. G.1.3 and G.2.3. Generally, our RMLR (Eq. (11)) requires Riemannian optimization for each manifold-valued parameter $P _k$. For the SPD MLR, we use geoopt [a] to optimize the SPD parameter $P _k$. For the Lie MLR, $P _k$ is a rotation matrix, whose Riemannian computation is not supported by geoopt. We, therefore, further extend the geometry in geoopt to include rotation matrices to update our Lie MLR.

2. Hyperparameters in SPD MLRs: $\theta$ > $(\alpha, \beta)$ .

The hyperparameter selection has been discussed in App. G.1.4. For a specific SPD MLR, there are at most three kinds of hyperparameters: deformation factor $\theta$, $\alpha$, and $\beta$. The general order of importance should be $\theta > (\alpha, \beta)$.

• The most significant parameter is the deformation factor $\theta$. As we discussed in Sec. 4.1, $\theta$ interpolates between different types of metrics ($\theta=1$ and $\theta \rightarrow 0$). Therefore, one can select its value around the deformation boundary, which has been systematically presented in Fig. 1. Generally speaking, the recommended candidate values for $\theta$ in AIM, PEM, and LCM are $\{0.5, 1, 1.5\}$, while the ones for $2\theta$ in BWM are $\{0.25, 0.5, 0.75\}$.

• The less important parameters are $(\alpha,\beta)$. Recalling Tab. 12, $(\alpha, \beta)$ only affects the Riemannian metric tensor, i.e., the inner products over tangent spaces. For our SPD MLRs in Thm 4.2, they only affect inner products, which should have fewer effects. Our experiments indicate that $(\alpha,\beta)=(1,0)$ is saturated for most cases.

3. Model complexities.

As the logics of RMLR are similar under different geometries, we focus on the SPD MLR in the following for simplicity.

• Memory complexities: Recalling Thm. 4.2, each class $k$ requires an SPD parameter $P _k$ and a Euclidean parameter $A _k$ in the SPD MLR. The memory costs depend on the number of class $C$ and the dimension of $P _k$ or $A _k$. Supposing there is $C$ class and the dimension of $P _k$ is $n \times n$. SPD MLR needs to store 2C matrix parameters of $n \times n$, i.e, $2Cn^2$. On the other hand, the classifier (LogEig MLR) on the tangent space at the identity requires $C \times \left(\frac{n(n+1)}{2}+1 \right)$, where $\frac{n(n+1)}{2}$ is the dimension of tangent space. Although our SPD MLR requires more parameters than the vanilla LogEig MLR, we achieve much better performance across different network architectures.

• Computational complexities: The efficiency of our SPD MLRs against the vanilla nonintrinsic LogEig MLR has been discussed in App. G.1.5. The key factor of the computational complexity of SPD MLRs under different metrics lies in the number of matrix functions, such as matrix power, logarithm, Lyapunov operator, and Cholesky decomposition. These matrix functions are divided into two categories: one is based on eigendecomposition, and the other is the Cholesky decomposition. Note that Cholesky decomposition is more efficient than eigendecomposition. Tab. A summarizes the number of matrix functions in each SPD MLR. With deformation, the general efficiency of SPD MLR should be LCM>EM>LEM>AIM>BWM, while without deformation, the order should be EM>LCM>LEM>AIM>BWM. The following Tab. B comes from Tab. 16 in App. G.1.5, which reports the average training time (in seconds) per epoch of each classifier. Please refer to App. G.1.5 for more details. Tab. B shows that EM-, LEM-, and LCM-based SPD MLR are more efficient than AIM- and BWM-based MLR. Notably, we also observe that our LCM achieves even comparable efficiency with the vanilla LogEig MLR on the radar and Hinss2021 datasets.

Table A: Number of matrix functions for each class $k$ in different SPD MLRs. a (b) means the number of matrix functions in the SPD MLR under the deformed (standard) metric.

Table B: Training efficiency (s/epoch) of different classifiers.

Reference

[a] Riemannian adaptive optimization methods. ICLR, 2018.

We thank Reviewer $\textcolor{brown}{jvDJ}$ for the constructive suggestions and insightful comments! In the following, we respond to the concerns in detail. 😄

---

1. Parallel transport is not the only option for determining $\tilde{A} _k$ in the RMLR (Eq. (11)).

• The ways to determine $\tilde{A} _k$ are flexible. As discussed in lines 142-150, as $P _k$ varies, we can not directly optimize $\tilde{A} _k \in T _{P _k} \mathcal{M}$ by the Euclidean optimization. The key is to use a map $f: T _{Q}\mathcal{M} \rightarrow T _{P _k}\mathcal{M}$ to generate $\tilde{A} _k$ from a fixed tangent space. The choice of $f$ is pretty flexible. In lines 142-150, we discussed four instances of $f$, including parallel transport, vector transport [a] (the approximation of parallel transport), differential of Lie group translation, and differential of gyro group translation. Apart from these, we believe there will be alternative proper $f$. For a specific manifold, the suitable choice of $f$ depends on factors such as the geometry of the manifold, simplicity, and numerical stability.

• We mainly focus on parallel transport due to its advantageous properties and theoretic convenience. Parallel transport has many nice properties, such as preserving inner product across tangent spaces [b, Ch. 3.3]. Furthermore, in our SPD and Lie MLRs involving parallel transport, parallel transport can be canceled out, and the MLR expression can be further simplified. For instance, Although the parallel transport under AIM could be complex (as shown in Tab. 12), it is canceled out and further simplifies the final expression of AIM-based SPD MLR. Please check the SPD MLRs in Thm. 4.2 (except the one under BWM) and Lie MLR in Thm. 5.2, as well as their proofs, for more details.

• Furthermore, we also use the differential of Lie group translation for better numerical stability. As detailed in App. F.2.1, the parallel transport under BWM is backpropagation-unfriendly. Therefore, we use the differential of Lie group translation to determine $\tilde{A} _k$ for the BWM-based SPD MLR as a more stable alternative.

• Finally, the projection map might omit crucial aspects of the latent geometry. We assume the projection map you mentioned is the orthogonal projection for the Riemannian submanifold or quotient manifold [a, Ch. 3.6]. It is widely used in Riemannian optimization, as it is equivalent to transforming the Euclidean gradient to Riemannian counterparts. However, the projection map only captures the horizontal part and discards the vertical part, leading to information loss or redundancy. For instance, it fails to preserve the inner product. Besides, it is not bijective, potentially introducing many-to-one redundancy in $\tilde{A} _{k}$. Therefore, although it can be numerically used to determine $\tilde{A} _{k}$, we did not use this map. Nevertheless, we will explore other maps, which properly map the Euclidean vector into the tangent space for determining $\tilde{A} _k$.

In summary, parallel transport is not a necessary ingredient for determining $\tilde{A} _k$, and there are different alternative choices. The specific choice depends on theoretical rationality and computational simplicity & stability.

2. Eq. (8) is not from point to point, but from point to hyperplane; therefore, the axioms of metric space cannot be applied to our distance.

• The Riemannian margin distance in Eq. (8) differs from the distance in a metric space. The distance in a metric space is defined as $\mathcal{M} \times \mathcal{M} \rightarrow \mathbb{R}$. However, the distance in Eq. (8) is $\mathcal{M} \times \{ \tilde{H} _{\tilde{A},P}\} \rightarrow \mathbb{R}$, where $\{ \tilde{H} _{\tilde{A},P}\}$ denotes the set of all Riemannian hyperplanes. Therefore, the axioms of metric space can not be applied to our Eq. (8). Nevertheless, Eq. (10) indicates the positiveness of our Riemannian margin distance.

• Besides, our Riemannian margin distance is a direct generalization of Euclidean margin distance. When the latent manifold $\mathcal{M}=\mathbb{R} ^n$, Eq. (8) is reduced to the familiar Euclidean margin distance (Eq. 7).

Reference

[a] Absil P A, Mahony R, Sepulchre R. Optimization algorithms on matrix manifolds. Princeton University Press, 2008.

[b] Do Carmo M P, Flaherty Francis J. Riemannian geometry[M]. Boston: Birkhäuser, 1992.

We thank Reviewer $\textcolor{green}{5QCn}$ for the careful review and the suggestive comments. Below, we address the comments in detail. 😄

---

1. Linearization by a single fixed tangent space or coordinate system fails to capture global geometry; Our RMLR adopts distinct dynamic tangent spaces for each class respecting Riemannian geometry.

Lines 29-31 summarize several previous classifiers in Riemannian neural networks, which linearize the whole manifold as a fixed Euclidean space, such as a fixed tangent space (at the identity matrix) or fixed coordinate system. In contrast, our Riemannian Multinomial Logistic Regression (RMLR) does not identify the manifold with a flat Euclidean space. Instead, it directly builds the classifier based on Riemannian geometry.

• Theoretically, our RMLR extends the Euclidean MLR into manifolds by Riemannian geometry. Def. 3.1 extends the Euclidean distance to the hyperplane in Eq. (7) into manifolds by Riemannian trigonometry and geodesic distance. Thm. 3.2 offers a general solution for this hyperplane distance. Putting this solution into Eq. (4), RMLR is introduced in Eq. (11). Although Eq. (11) uses Riemannian logarithm, it is derived from distances to Riemannian hyperplanes, respecting the underlying Riemannian geometry.

• Numerically, our RMLR in Eq. (11) adopts distinct dynamic tangent spaces for each class:

$$$$

where $P _k \in \mathcal{M}$ and $\tilde{A} _k \in T _{P _k} \mathcal{M}$. Each class $k$ involves a distinct calculation of $\left\langle\operatorname{Log} _{P _k} S, \tilde{A} _k\right\rangle _{P _k}$ in the tangent space at $P _k$, i.e., $T _{P _k} \mathcal{M}$. More importantly, each tangent space $T _{P _k} \mathcal{M}$ is dynamically learned, as each $P _k$ is a learnable parameter. Besides, $\left\langle\operatorname{Log} _{P _k} S, \tilde{A} _k\right\rangle _{P _k}$ respects the Riemannian hyperplane (Eq. (5)), which generalizes the Euclidean decision plane (Eq. (3)). As illustrated in Figs. 2-3, the Riemannian hyperplane for each class is a curved surface.

2. Core technical contribution: We derive the Riemannian marginal distance by the Riemannian trigonometry and geodesic distance, which only requires matrix logarithm.

As discussed in lines 104-118, the core challenge in building Riemannian MLRs is reformulating the Eucliedan marginal distance in Eq. (7) into Riemannian counterparts. Previous Riemannian MLRs mainly resort to gyro structures and pullback metrics from the Euclidean space. These approaches fail to handle general geometries due to the strong requirements of the metrics.

• As the core technical contribution, we re-interpret the Riemannian marginal distance using Riemannian trigonometry and geodesic distance, which only requires matrix logarithm. As discussed in lines 120-121, the matrix logarithm is the minimal requirement for building Riemannian MLRs, and it exists in the most popular geometries in the machine learning community. Based on this, our Thm. 3.3 provides a solution for RMLR under general geometries. We thus see our RMLR is easy to use and can handle general geometries. Further, as shown in Tab. 1, many previous MLRs are special instantiations of our RMLR under different metrics. We will emphasize this core contribution in the revised introduction.

• Further remark: As we claimed, given a Riemannian metric, our RMLR can be implemented in a plug-in manner, by simply putting the required Riemannian logarithm into Eq. (11). Therefore, our RMLR can also be implemented on other manifolds, such as correlation matrices [a] and special Euclidean groups [b], as the required Riemannian operators have explicit expressions. We will explore these in the future.

3. Experimental explanations in the appendix will be briefly summarized in the main paper.

Due to page limits, we left implementation details and RMLR efficiency in the appendix. In the final version, we will briefly summarize our findings in the main paper and provide proper references to the appendix.

References

[a] Thanwerdas Y. Permutation-invariant log-Euclidean geometries on full-rank correlation matrices. SIAM Journal on Matrix Analysis and Applications, 2024.

[b] Murray R M, Li Z, Sastry S S. A mathematical introduction to robotic manipulation[M]. CRC press, 2017.