Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry

3. Explanations: Pow-EMLR vs. Pow-TMLR

$$**Tangent Classifier Pow-TMLR: ** \mathcal{F}\left(f_{\text {vec }}\left(\frac{1}{\theta}\left(S^{\theta}-I\right)\right)\right),$$ $$**Riemannian Classifier Pow-EMLR: ** \mathcal{F}\left(f_{\text {vec }}\left(S^{\theta}\right)\right).$$

As recapped above, Pow-EMLR and Pow-TMLR differ only by a constant transformation $f(X)=\frac{1}{\theta}\left(X-I\right)$. However, this difference introduces both theoretical and numerical disparities.

• Theoretical Explanations. Pow-TMLR operates as a tangent classifier on the SPD manifold. By identifying the entire manifold with a single flat tangent space, it fails to respect the complex SPD geometry. In contrast, Pow-EMLR is equivalent to the SPD MLR, which is grounded in the Riemannian geometry of the SPD manifold. As such, Pow-EMLR naturally has theoretical advantages over the approximated tangent-based Pow-TMLR. Several studies have demonstrated the effectiveness of intrinsic Riemannian MLRs across various geometries, including SPD manifolds [b-d] and hyperbolic manifolds [h-j]. These findings serve as the main supporting evidence for the superiority of Pow-EMLR over Pow-TMLR.
• Numerical Explanations. Beyond the explanation from Riemannian geometry, as discussed in App. G.2, although Pow-EMLR and Pow-TMLR can be aligned during the forward process (by adjusting initialization), their gradient backpropagation differs. This subtle difference makes Pow-EMLR and Pow-TMLR distinct in the training dynamics of the network, further explaining the observed performance gap. Please refer to App. G.2 for more details.

Remark: Initially, during this work, we attempted to use tangent classifiers to explain the role of matrix functions in GCP. Despite fine-tuning efforts, Pow-TMLR usually failed to achieve performance comparable to Pow-EMLR. This counterintuitive result also puzzled us. However, we believe there must be some other more fundamental reasons. When we delved into this myth deeper, we finally discovered that Pow-EMLR is implicitly a Riemannian MLR, which is a fundamentally different mechanism from the tangent perspective. This finding explains not only the mechanism underlying matrix functions in GCP but also the unexpected performance gap observed. We hope our findings will inspire further research of GCP from a geometric perspective.

---

References

[a] The geometry of mixed-Euclidean metrics on symmetric positive definite matrices

[b] RMLR: Extending multinomial logistic regression into general geometries

[c] Building neural networks on matrix manifolds: A Gyrovector space approach

[d] Riemannian multinomial logistics regression for SPD neural networks

[e] A Lie group approach to Riemannian batch normalization

[f] The Gyro-structure of some matrix manifolds

[g] Why approximate matrix square root outperforms accurateSVD in global covariance pooling

[h] Hyperbolic neural networks

[i] Hyperbolic neural networks++

[j] Fully hyperbolic convolutional neural networks

2. Equivalence of Eq. (15) or (16) to the SPD MLR

Let us first review the related equations. For each class $\forall k \in 1, \cdots, C$, we have:

$$**Eq. (15): ** \left\langle \log(S) - \log(P _k), A _k \right\rangle \overset{\text{Prop. 5.1 [d]}}{\Longleftrightarrow} **Log-EMLR: ** \mathcal{F} \left( f _{\mathrm{vec}} (\log(S)) \right),$$ $$**Eq. (16): ** \frac{1}{\theta}\left\langle S ^\theta-P _k ^\theta, A _k\right\rangle \overset{\text{Thm. 2}}{\Longleftrightarrow} **ScalePow-EMLR: ** \mathcal{F}\left(f _{\text {vec }}\left(\frac{1}{\theta} S ^\theta\right)\right),$$ $$\text{ScalePow-EMLR} \underset{\text{Empirically: Sec. 6}}{\overset{\text{Theoretically: App. G.1}}{\Longleftrightarrow}}**Pow-EMLR (Used in GCP): ** \mathcal{F} \left(f _{\mathrm{vec}} \left(S ^\theta \right) \right),$$

where $S$ is the input SPD matrix, $P _k \in \mathcal{S} _{++} ^n$ is an SPD parameter, and $A _k \in \mathcal{S} ^n$ is a symmetric matrix parameter. Here, $\mathcal{F}$ represents the FC layer, and $f _{\text{vec}}$ denotes the vectorization. Without loss of generality, we omit the softmax operation.

The equivalence between Eq. (15) and Log-EMLR has been rigorously proven by [d, Prop. 5.1]. Our focus now shifts to Pow-EMLR. Specifically, we first prove the equivalence between Eq. (16) and ScalePow-EMLR (as presented in Thm. 2) and then demonstrate the equivalence between ScalePow-EMLR and Pow-EMLR both theoretically and empirically.

• Eq. (16) and ScalePow-EMLR. As each $P_k$ in Eq. (16) is an SPD parameter, it is non-trivial to directly identify Eq. (16) with ScalePow-EMLR. Different from Euclidean parameters, $P_k$ requires Riemannian optimization, such as the Riemannian SGD or Riemannian gradient. However, Thm. 2 shows that when $P_k$ is optimized bt PEM-based Riemannian optimization, Eq. (16) and ScalePow-EMLR become equivalent. As proven in App. I, Eq. (16) is identical to ScalePow-EMLR during each optimization step.
• ScalePow-EMLR and Pow-EMLR. The difference between ScalePow-EMLR and Pow-EMLR lies only in a scaling factor, which should have minor practical effects. Although the scaling factor affects the gradient, these two formulations can become equivalent under carefully tuned initialization and learning rates. For rigorous theoretical analysis, please refer to App. G.1.
• Conclusion. We claim that Pow-EMLR is implicitly the SPD MLR described in Eq. (16), as it is theoretically equivalent to ScalePow-EMLR under scaled settings. Experimental results in Sec. 6 further validate this claim, demonstrating that Pow-EMLR achieves performance comparable to ScalePow-EMLR. We summarize this reasoning as follows:

$$\text{Eq. (16)} \overset{\text{Thm. 2}}{\Longleftrightarrow} \text{ScalePow-EMLR} \underset{\text{Empirically: Sec. 6}}{\overset{\text{Theoretically: App. G.1}}{\Longleftrightarrow}} \text{Pow-EMLR (Used in GCP)}.$$

We thank reviewer $\textcolor{red}{2skh}$ for the valuable feedback. Below is our detailed response. 😄

---

1. Analysis of the Empirical Difference between Matrix Power and Logarithm

• Empirical Difference and Latent Geometries. This paper demonstrates that matrix power and logarithm in GCP respect the Power-Euclidean Metric (PEM) and Log-Euclidean Metric (LEM), respectively. The empirical differences between them can thus be attributed to the underlying metrics. As shown in [a, Sec. 3.1], PEM converges to LEM as $\theta \to 0$, meaning PEM serves as a balanced metric between LEM and the Euclidean Metric ($\theta=1$), offering greater flexibility. GCP applications might benefit from this balancing effect. Supporting evidence from [b, Tabs. 4 and 7] also suggests that PEM may outperform LEM in certain tasks, highlighting its practical effectiveness.
• Treating Metrics as Hyperparameters. Rather than focusing solely on matrix functions, our work uncovers the latent geometries they respect. A key insight from this work is that the choice of metric should be treated as a hyperparameter, as it is difficult to identify a universally optimal metric for all applications due to the non-convex nature of deep networks. The current empirical superiority of matrix power over logarithm may stem from PEM being better suited to the GCP application than LEM. Previous studies on the SPD manifold have shown that the optimal metric often varies across tasks [b-f]. We hope our work will inspire further exploration of alternative geometries on the SPD manifold to design more effective GCP classifiers.
• Numerical Issues of Matrix Logarithm. Beyond theoretical considerations, the matrix logarithm can excessively stretch eigenvalues [g], which may lead to suboptimal performance in high-dimensional settings, such as those encountered in GCP. In contrast, the matrix power in PEM exhibits more moderate eigenvalue scaling. Tab. B illustrates a numerical comparison between matrix logarithm and square root in terms of eigenvalue behavior.

Thanks for your further reply and insightful comments!

We agree that our core contribution lies in uncovering the underlying mechanism of the Riemannian classifier. We will emphasize this point in the revised introduction and point to the detailed discussion in Sec. 5.3. The choice of a power of 0.5 is indeed more favorable for GCP applications, especially in the large-scale setting. While the reason why power=0.5 performs best is not fully addressed, our work sheds light on the mechanism of the Riemannian classifier, providing a novel perspective. In future work, we plan to delve deeper into understanding why PEM, particularly with power=0.5, outperforms LEM.

We would like to discuss more about the desired properties of Riemannian metrics and future directions of manifold learning beyond SPD manifolds:

Geometries within the SPD Manifold

Three properties of metrics on the SPD manifold are particularly crucial: invariance, closed-form Riemannian operators, and isometry to the Euclidean metric. These properties enhance the representation power, ease of implementation, and computational efficiency of Riemannian classifiers.

• Invariance, such as $\mathrm{O}(n)$-invariance, is a critical property for modeling covariance matrices. Metrics with $\mathrm{O}(n)$-invariance enable the construction of more powerful Riemannian classifiers. In addition to LEM and PEM, the recently developed Bures-Wasserstein metric [A, Tab. 6] also possesses $\mathrm{O}(n)$-invariance. Moreover, as shown in [A, Tab. 5], the classic Affine Invariant metric has been generalized into a two-parameter $\mathrm{O}(n)$-invariant metric. Riemannian classifiers under these metrics have closed-form expressions [B, Thm. 4.2]. Exploring the adaptation of such classifiers to GCP applications could be an interesting direction.
• Closed-form expressions enable the practical development and implementation of Riemannian classifiers. All seven metrics discussed in Sec. 2 possess closed-form expressions for their Riemannian operators.
• Isometry to the Euclidean metric facilitates fast and simple computations. Riemannian computations under such metrics reduce to Euclidean computations, modulo the calculation of isometric diffeomorphisms [C, Tab. 2]. This property could significantly simplify the final expression of Riemannian classifiers, making them well-suited for GCP applications.

Geometries beyond the SPD Manifold

Beyond SPD matrices such as covariance, correlation matrices can be viewed as normalized SPD matrices, serving as natural, compact alternatives to SPD matrices in statistical analysis. Recently, these structured matrices have been recognized as a manifold, and several Riemannian metrics have been identified [D, E]. These metrics enjoy the three properties mentioned above. Therefore, it is promising to develop Riemannian classifiers based on this manifold for GCP applications.

Thanks again for the encouraging feedback. We will explore both geometries of and beyond SPD manifolds and find out more in our future work!

---

References

[A] O(n)-invariant Riemannian metrics on SPD matrices

[B] RMLR: Extending Multinomial Logistic Regression into General Geometries

[C] The geometry of mixed-Euclidean metrics on symmetric positive definite matrices

[D] Theoretically and computationally convenient geometries on full-rank correlation matrices 2022

[E] Permutation-invariant log-Euclidean geometries on full-rank correlation matrices 2024

---

Thanks for the instant reply and further comments! We make the following explanations.

---

1. Novel Insights: From Matrix Functions to Matrix Geometries

We discover that matrix log and pow in GCP respect Log Euclidean Metric (LEM) and Power Euclidean Metric (PEM), respectively. Therefore, the empirical difference can be attributed to the difference in their Riemannian metrics. Apart from preserving several properties of LEM, PEM offers greater flexibility.

• Shared Properties: PEM shares several key properties with LEM. Both are $\mathrm{O}(n)$-invariant [A, Tab. 2], a crucial property for modeling covariance matrices. Additionally, their Riemannian computations (e.g., Riemannian logarithm, exponentiation, and parallel transport) are similar, modulo the computation of matrix log or power.
• Flexible Deformation: PEM is a deformed metric of LEM. It approaches LEM with power factor $\theta \rightarrow 0$.

The above suggests that PEM functions as a balanced metric, maintaining the essential properties of LEM while introducing greater flexibility. This added flexibility could become particularly advantageous in high-dimensional settings, such as those encountered in GCP applications, potentially explaining why PEM outperforms LEM in these cases.

Remark: [$\mathrm{O}(n)$-invariance] When a random vector $x$ with covariance matrix $\Sigma$ is transformed by $O x$, the resulting covariance matrix becomes $O \Sigma O^\top$. $\mathrm{O}(n)$-invariance is to characterize this relation between covariance matrices:

$

d(P,Q) = d(O P O^\top,O Q O^\top), \quad \forall P, Q \in \mathcal{S} _{++} ^n, \forall O \in \mathrm{O}(n),

$

where $d$ denotes the geodesic distance.

---

2. Pow-TMLR vs. Pow-EMLR --> Tangent Classifiers vs. Riemannian Classifiers

Although the numerical expressions of Pow-TMLR and Pow-EMLR differ by a constant transformation, they differ fundamentally in theory: Pow-TMLR is a tangent classifier, whereas Pow-EMLR is a Riemannian classifier.

• Tangent Classifier: The tangent classifier treats the entire manifold as a single tangent space at the identity matrix. When mapping data into this tangent space, critical structural information, such as distances and angles, cannot be preserved. This distortion undermines classification performance. In contrast, Riemannian MLR is constructed based on Riemannian geometry, fully respecting the manifold's geometric structure.

• Tangent as a Special Case of Riemannian Classifier: The tangent classifier can be seen as a reduced case of the Riemannian classifier. For example, consider Eq. (16) (PEM):

$

**Eq. (16): ** P(y=k|S) \propto \exp \left[\frac{1}{\theta}\left\langle S ^\theta-P _k ^\theta, A _k \right\rangle\right], \quad \forall k =1, \cdots, C.

$

When all SPD parameters $P_k$ are set to the fixed identity matrix, Eq. (16) exactly corresponds to Pow-TMLR.

In summary, the Riemannian classifier enjoys significant theoretical advantages over the tangent classifier while incorporating the tangent classifier as a special case.

Remark: We have added the above discussion into App. G.3 in our revised manuscript.

---

Further Comments Are Very Welcome

If you have any additional comments, we are happy to discuss them further.

---

References

[A] RMLR: Extending multinomial logistic regression into general geometries

We thank Reviewer $\textcolor{blue}{24iT}$ for valuable comments. Below, we address the comments in detail. 😄

---

1. Practical Significance: From Matrix Functions to Various Latent Geometries

• From Matrix Functions to Latent Geometries. Previous work primarily views the matrix functions in GCP as numerical normalization methods. While this numerical interpretation provides empirical insights, it lacks an in-depth theoretical foundation. Covariance matrices, however, lie on the SPD manifold, which is endowed with rich Riemannian structures. A natural idea is that Riemannian features may benefit more from operations grounded in Riemannian geometry rather than Euclidean manipulations. This idea aligns with findings from recent studies across various geometries [a-i]. Inspired by these advancements, we collectively interpret matrix power and logarithm as special SPD MLRs under the Power-Euclidean Metric (PEM) and Log-Euclidean Metric (LEM), respectively. In this way, the empirical performance of power and log can be attributed to the metrics they respect. We expect this novel view can shed light on the GCP research.
• From PEM and LEM to Other Geometries. Our discovery hints at the potential to design classifiers in GCP by leveraging Riemannian classifiers. As summarized in Sec. 2, there are five other metrics on the SPD manifold beyond PEM and LEM. We expect that our work could inspire the development of GCP based on these alternative metrics.

The above impact has been briefly mentioned in our conclusion. To further highlight our significance, we have added the following to the introduction (L 93-95):

• We expect our work to pave the way for a deeper theoretical understanding of GCP from a Riemannian perspective and inspire more research to explore the rich SPD geometries for more effective GCP applications.

---

2. Our Work Also Explains the Empirical Superiority of Matrix Power over Element-Wise Power

As demonstrated in Thm. 2, matrix power, including the matrix square root, implicitly constructs the intrinsic SPD MLR under the Power-Euclidean Metric (PEM). In contrast, element-wise power applied to SPD matrices is merely a numerical trick. It neither connects to nor respects any of the seven Riemannian metrics (reviewed in Sec. 2) on the SPD manifold. Previous works [a-i] have demonstrated that Riemannian features benefit from Riemannian operations on different geometries. Therefore, this theoretical distinction provides a novel explanation for the empirical superiority of matrix power over element-wise power observed in previous work [j].

---

3. Changed Caption of Fig. 1

Fig. 1 is intended to refute the discussions in L 53-54. Previous works [j,k] claim that since the distances under PEM and LEM are similar, and the co-domain of the matrix logarithm is a Euclidean space, it is reasonable to use a Euclidean classifier after the matrix power. However, Fig. 1 demonstrates a significant gap between the distances under PEM ($\theta=0.5$) and LEM, challenging the validity of this assumption.

Thank you for the suggestion. We have revised the caption of Fig. 1 to clarify the implications of this observed gap:

• This indicates that matrix power is not proximate to LEM for classification under the widely used $\theta=0.5$.

---

References

[a] RMLR: Extending multinomial logistic regression into general geometries

[b] Building neural networks on matrix manifolds: A Gyrovector space approach

[c] Riemannian multinomial logistics regression for SPD neural networks

[d] A Lie group approach to Riemannian batch normalization

[e] The Gyro-structure of some matrix manifolds

[f] Hyperbolic neural networks

[g] Hyperbolic neural networks++

[h] Fully hyperbolic convolutional neural networks

[i] Riemannian residual neural networks

[j] What deep CNNs benefit from global covariance pooling: An optimization perspective

[k] Is second-order information helpful for large-scale visual recognition?

Following up on Response 1

Further clarifications are more than welcome

As opposed to tangent classifiers (e.g., Eq. $C$) that approximate the manifold as a flat space, the SPD MLR Eq. (D) resorts to the idea of margin distance to the hyperplane characterizing each class [a, Eqs. (4-5)], which has shown success in different geometries, including SPD [a-c], Symmetric Positive Semi-Definite (SPSD) [d], and hyperbolic [e-f] manifolds.

If you are interested in delving further into extending the Euclidean MLR Eq. (B) into manifold [a-g] or more explanations on Thm. 2, we would be more than happy to provide additional clarifications.

2. Bias is not disentangled.

In the original SPD MLRs Eqs. (E-F), $P _k$ is the biasing parameter. Thm. 3.2 demonstrates that Eq. (E) is equivalent to

$$**ScalePow-EMLR: ** \mathcal{F} \left(f _{\mathrm{vec}} \left( \frac{1}{\theta} S ^{\theta} \right) \right),$$

where $\mathcal{F}$ is the standard FC layer. In our code, $\mathcal{F}$ in Pow-EMLR, Pow-TMLR, and ScalePow-EMLR are all implemented by torch.nn.Linear. There is no separate manipulation of the bias. The following is our pseudo-code.

# Pseudocode for Covariance Pooling and ResNet with Covariance Pooling

import torch.nn as nn

# Covariance Pooling Module
class CovPooling(nn.Module):
    def __init__(self, args):
        """
        Initializes the CovPooling module.
        Args:
            args: Configuration or arguments required for initialization.
        """

    def forward(self, x):
        """
        Forward pass for Covariance Pooling.
        Args:
            x: Input feature tensor.
        Returns:
            x_vec: Vectorized representation of the transformed SPD matrix.
        """
        # Step 1: Apply covariance pooling
        x = self._cov_pool(x)

        # Step 2: Compute matrix square root
        x_sqrt = self._sqrtm(x)

        # Step 3: Apply transformations
        if self.transformed_mode == 'Pow-TMLR':
            # $\frac{1}{\theta}=2$ for sqrt; Self.I is the identity matrix
            tmp = 2 * (x_sqrt - self.I)  
            x_transformed = tmp
        elif self.transformed_mode == 'ScalePow-EMLR':
            # $\frac{1}{\theta}=2$ for sqrt
            x_transformed = 2 * x_sqrt  
        elif self.transformed_mode == 'Pow-EMLR':
            x_transformed = x_sqrt  # For Pow-EMLR

        # Step 4: Vectorize the resulting matrix
        x_vec = self.vec(x_transformed)

        # Step 5: Return the vectorized matrix
        return x_vec

# ResNet with Covariance Pooling
class ResNetCovPooling(nn.Module):
    def __init__(self, args, fc_input_dim, num_classes):
        """
        Initializes the ResNetCovPooling model.
        Args:
            args: arguments for ResNet and CovPooling
            fc_input_dim is the dim of vectorized SPD matrices.
            num_classes: Number of output classes.
        """
        super(ResNetCovPooling, self).__init__()
        # Step 1: Set up backbone, representation, and classifier
        self.feature = ResNet(args.ResNet)  # ResNet backbone
        self.representation = CovPooling(args.CovPooling)  # Covariance pooling representation
        self.classifier = nn.Linear(fc_input_dim, num_classes)  # Fully connected layer

    def forward(self, x):
        """
        Forward pass for the ResNetCovPooling model.
        Args:
            x: Input feature tensor.
        Returns:
            out: Class predictions.
        """
        # Step 1: Extract features using ResNet backbone
        x = self.feature(x)

        # Step 2: Apply Covariance Pooling for representation
        x = self.representation(x)

        # Step 3: Classify the representation
        out = self.classifier(x)

        # Step 4: Return class predictions
        return out

We thank Reviewer $\textcolor{green}{JvSJ}$ for the careful review and the suggestive comments. Below, we address the comments in detail.😄

---

1. Reparameterization: manifold counterparts of $\langle a_k, x-0 \rangle - b_k$ and $\langle a_k, x-p_k \rangle$ are significantly different

Given $x \in \mathbb{R} ^n$, the two reparameterizations of $\langle a_k, x \rangle - b_k$ are equivalent in Euclidean space:

$$**Eq. (A): ** \langle a_k, x - 0 \rangle - b_k, \quad **Eq. (B): ** \langle a_k, x - p_k \rangle,$$

with $a _k \in \mathbb{R} ^n$, $p _k \in \mathbb{R} ^n$, and $b _k \in \mathbb{R}$.

However, their extensions to manifolds are significantly different and lead to fundamentally distinct interpretations.

Reformulation via Eq. (A) is the tangent classifier discussed in Sec. 4

Given an SPD matrix $S \in \mathcal{S} _{++} ^n$, extending Eq. (A) to the SPD manifold $\mathcal{S} _{++} ^n$ is

$$\langle a _k, x - 0 \rangle - b _k \Rightarrow \langle \mathrm{Log} _I(S), A _k \rangle_I - b _k, \quad \forall k = 1, \ldots, C, \quad **Eq. (C)**$$

where $I$ is the identity matrix, $A _k \in T _I \mathcal{S} ^n _{++} \cong \mathcal{S} ^n$, and $b _k \in \mathbb{R}$. Here, $\mathcal{S} ^n$ is the Euclidean space of symmetric matrices, while $\langle , \rangle _{I}$ and $\mathrm{Log} _I$ are the Riemannian metric and logarithm at $I$. Eq. $C$ corresponds to mapping $S \in \mathcal{S} ^n _{++}$ to the tangent space $T_I \mathcal{S} ^n _{++}$ using the Riemannian logarithm $\mathrm{Log} _I(S)$, followed by applying a Euclidean FC layer in the tangent space.

Under PEM and LEM on the SPD manifold, Eq. $C$ exactly reduces to Pow-TMLR and Log-EMLR:

• Pow-TMLR: $\mathcal{F} \left(f _{\mathrm{vec}} \left( \frac{1}{\theta} \left(S ^\theta - I \right) \right) \right)$,
• Log-EMLR: $\mathcal{F} \left(f _{\mathrm{vec}} \left( \log(S) \right) \right),$ where $\mathcal{F}$ denotes the FC layer and f _{\mathrm{vec} is the vectorization.

Comparing this to Pow-EMLR in GCP:

$$**Pow-EMLR: ** \mathcal{F} \left(f_{\mathrm{vec}} \left(S^\theta \right) \right),$$

Pow-TMLR and Pow-EMLR are quite similar. However, experimental results in Tabs. 3 and 5 consistently show that Pow-TMLR performs worse than Pow-EMLR. This observation suggests a more fundamental mechanism underlying the matrix functions in GCP. The above is what we discussed in Sec. 4.

Riemannian MLR via Eq. (B) is different from the tangent MLR via Eq. (A)

Extending Eq. (B) to the SPD manifold results in a fundamentally different formulation, leading to SPD MLRs [a, Thm. 3.3]:

$$\langle a _k, x - p _k \rangle \Rightarrow \langle \mathrm{Log} _ {P _k}(S), \widetilde{A} _k \rangle _{P _k}, \quad \forall k = 1, \ldots, C, \quad **Eq. (D)**$$

where $P _k \in \mathcal{S} ^n _{++}$ and $\widetilde{A} _k \in T _{P _k} \mathcal{S} ^n _{++}$ are the parameters. It can hardly be said that Eq. $C$ and (D) are equivalent, although their Euclidean prototypes are equivalent. A simple intuition is that there are multiple tangent plain in Eq. (D), while Eq. $C$ only has a single tangent space. After several steps of simplification, the SPD MLR under PEM and LEM [c, Thm. 4.2] for the SPD input $S \in \mathcal{S} ^n _{++}$ can be expressed as follows:

• LEM-based SPD MLR:

  $$p(y = k \mid S) \propto \exp \left( \left\langle \log(S) - \log(P_k), A_k \right\rangle \right), \quad \quad **Eq. (E)**$$

  where $P_k \in \mathcal{S}^n_{++}$, and $A_k \in \mathcal{S} ^n$.

• PEM-based SPD MLR:

  $$p(y = k \mid S) \propto \exp \left( \frac{1}{\theta} \left\langle S^\theta - P_k^\theta, A_k \right\rangle \right), \quad \quad **Eq. (F)**$$

  where $\theta > 0$, $P_k \in \mathcal{S}^n_{++}$, and $A_k \in \mathcal{S} ^n$.

While SPD MLRs differ from tangent classifiers, they are also distinct from the Pow-EMLR and Log-EMLR used in GCP, as there is an SPD parameter $P _k$ for each class. However, our Thm. 2 demonstrates that the matrix power in GCP implicitly respects the SPD MLR under PEM. Similar results also hold for the matrix logarithm and LEM [b, Prop. 5.1]. This theoretical insight bridges the gap between practical GCP matrix functions and SPD MLRs.

Under trivial bias, Riemannian MLR becomes the tangent classifier

When the bias in Eq. (D) is set as the fixed "zero" element (identity matrix), the Riemannian MLR reduces to the tangent classifier described in Eq. (C). Unlike in Euclidean space, the bias plays a more critical role in Riemannian MLR, as it determines the anchor point of the tangent space corresponding to each class, which is sourced back to the definition of margin hyperplane [a, Eq. (5)] or [e, Eq. (22)].

(To be continued)

Thank you very much for the positive feedback and suggestive comments!

These discussions will be shown in our final version, as we cannot upload pdf now. We will further revise our presentation to clarify our key contributions! 😄

We thank Reviewer $\textcolor{purple}{SGk2}$ for the encouraging feedback and the constructive comments! In the following, we respond to the concerns point by point. 😄

---

1. Nuanced Discussion of LEM and PEM

Notations. $(\theta, 1, 0)$-EM refers to the Power-Euclidean Metric (PEM), while $(1, 0)$-LEM corresponds to the standard Log-Euclidean Metric (LEM). For simplicity, we use PEM and LEM in the following discussion.

Comparison of LEM and PEM

Below is a refined discussion of Sec. 5.3, as summarized in Tab. A.

Table A: A summary of key differences between LEM and PEM.

• Overly Stretched Eigenvalues in LEM. The matrix logarithm in LEM can excessively stretch eigenvalues [a], which may lead to suboptimal results, especially in high-dimensional settings like those encountered in GCP. In contrast, PEM exhibits more moderate eigenvalue scaling. Tab. B provides a numerical illustration.

Table B: A numerical comparison of log and square root functions. Given eigenvalues $a=10^3$ and $b=10^{-3}$, the log excessively compresses the magnitude.

• Swelling Effect in PEM. PEM suffers from the swelling effect, which may affect certain applications, such as diffusion tensor imaging. While LEM does not exhibit swelling [b], PEM partially alleviates this issue but cannot eliminate it. Following [b], we analyzed the swelling effect on geodesic interpolation under different metrics. Tab. C shows that LEM demonstrates no swelling, while the swelling in PEM reduces as $\theta$ decreases.

Table C: Determinants of geodesic interpolations on the $3 \times 3$ SPD manifold under PEM and LEM. The swelling effect is indicated by increasing determinant values. Here, $\theta$-PEM denotes PEM with power equal to $\theta$.

• Completeness. As shown in [c, Tab. 2], LEM is geodesically complete, while PEM is incomplete. This limitation may affect applications requiring the Riemannian exponential map to be globally defined. However, this limitation does not impact GCP, as GCP does not require completeness.

• Relation Between PEM and LEM. As shown in [d, Sec. 3.1], PEM converges to LEM as power $\theta \to 0$. Tab. C also demonstrates this, where smaller values of $\theta$ reduce swelling, making PEM resemble LEM. This suggests that PEM can be viewed as a balanced metric between LEM and the Euclidean Metric (EM).

Empirical Applications

The limitations of LEM and PEM depend on specific applications:

• High-dimensional SPD Matrices: The eigenvalue stretching in LEM can make it less suitable for high-dimensional settings, such as those in GCP.
• Very Small SPD Matrices: Conversely, the swelling effect in PEM may become unsatisfactory for very small SPD matrices, such as the $3 \times 3$ matrices used in diffusion tensor imaging.

Overall, the choice of metric should be treated as a hyperparameter. Due to the non-convex nature of deep networks, it is challenging to identify a universally optimal metric across all tasks. Our work highlights the tradeoffs between LEM and PEM, providing a theoretical framework for practitioners to explore metrics tailored to specific applications.

---

2. Comparison of Pow-EMLR, Pow-TMLR, and SaclePow-EMLR under different power factors.

Table. D: Results of different powers under the ResNet-50 backbone on the Aircrafts and Cars dataset.

Following your suggestion, we conducted additional experiments with varying power factors on the Aircrafts and Cars datasets, i.e., $\theta = 0.25, 0.5, 0.7$. For these experiments, we employed accurate Singular Value Decomposition (SVD) for calculating the matrix power and used Padé approximants for backpropagation [e]. The results are summarized in Tab. D. Note that, since we used SVD to compute all the matrix power in these experiments, the results for $\theta = 0.5$ might differ from those reported in Tab. 5 of the main paper.

From the results, it is clear that Pow-EMLR consistently achieves performance comparable to ScalePow-EMLR while outperforming Pow-TMLR across all power factors. This provides further evidence supporting our claim that the mechanism underlying matrix functions in GCP should be attributed to the Riemannian MLRs they implicitly respect.

Remark: The above has been added to Sec. 6.2 in our revised paper.

---

3. Experiments on the Second-order Transformer (SoT)

Table E: Comparison of Pow-EMLR against Pow-TMLR under the SoT-7 backbone on the ImageNet-1k dataset.

We appreciate the reviewer’s suggestion to evaluate our findings on ViT. Following [f], we conduct experiments using the Second-order Transformer (SoT) [g] on the ImageNet-1k dataset. Specifically, we use the 7-layer SoT (SoT-7) architecture and train the model up to 250 epochs with a batch size of 512, keeping other settings consistent with [f]. The results are presented in Tab. E.

Pow-EMLR still outperforms Pow-TMLR under the SoT-7 backbone. These results further support our claim that tangent classifiers cannot adequately explain the matrix functions used in GCP, while the underlying mechanism can be better explained by our Riemannian perspective.

Remark: The above has been added into Sec. H.3 in our revised paper.

---

References

[a] Why approximate matrix square root outperforms accurate SVD in global covariance pooling

[b] Log‐Euclidean metrics for fast and simple calculus on diffusion tensors

[c] RMLR: Extending multinomial logistic regression into general geometries

[d] The geometry of mixed-Euclidean metrics on symmetric positive definite matrices

[e] Why approximate matrix square root outperforms accurate SVD in global covariance pooling?

[f] Fast differentiable matrix square root

[g] SoT: Delving Deeper into Classification Head for Transformer

Additional Experiments on Second-order Transformer (SoT)

Dear reviewer, we have trained SoT-7 with ScalePow-EMLR to further validate our claim. The following table renews the previous Tab. E. Pow-EMLR still achieves similar performance to ScalePow-EMLR, but outperforms Pow-TMLR. These results further support our claim that tangent classifiers cannot adequately explain the matrix functions used in GCP, while the underlying mechanism can be better explained by our Riemannian perspective.

We have revised our manuscript accordingly.

Table F: Comparison of Pow-EMLR, ScalePow-EMLR and Pow-TMLR under the SoT-7 backbone on the ImageNet-1k dataset.