EqNIO: Subequivariant Neural Inertial Odometry

W1. The abstract describes EqNIO as leveraging "canonical displacement priors" to generalize across arbitrary IMU orientations, but it lacks a clear technical explanation of how these priors work in practice. Generalization is claimed to stem from "canonical gravity-aligned frames" and "equivariant yaw frames," but the abstract could benefit from a more precise explanation of these transformations and their operationalization in the model.

Thanks for the valuable suggestion. What we and previous work refer to as “neural displacement priors” are neural networks that take IMU measurements as input and generate displacement and covariance predictions as outputs, i.e. a prior probability distribution on the expected displacement, given a set of measurements. These priors can be incorporated into probabilistic filters to estimate the state of the IMU over time. When referring to “canonical priors”, we mean priors (i.e. predicted displacement and covariances) expressed in a canonical frame. We uniquely define the canonical frame via three axes: the first is the gravity axis (making the frame gravity aligned), and the second and third are predicted by an “equivariant frame model” that takes IMU data as input. Note that this frame is not necessarily right-handed, and quantities expressed in this frame can, therefore, potentially undergo a reflection and rotation. This model is termed “equivariant” since it commutes with rotation around or reflection across the gravity axis (O(2) roto-reflection). This means that if the input is O(2) roto-reflected by an arbitrary transformation, the resulting output is roto-reflected by the same transformation.

We reworked the abstract to lay more emphasis on the explanations above and provide further details on the operationalization of the canonical frames in Q1 below.

W2. The learnable yaw orientation in canonical frames is a promising feature but lacks clarity on how it resolves yaw drift or improves orientation estimation, given that yaw is typically the most challenging to estimate accurately in inertial odometry due to the absence of an absolute reference.

We believe that the term “learnable yaw” has generated some confusion. The canonical frame is specified by gravity and two learnable orthogonal vectors perpendicular to gravity. It does not define a yaw measurement and is thus not incorporated directly into the filter. Instead, this frame is used to canonicalize the input, generate the displacement, and then transform it back into the original frame. As a result, our network generates more consistent outputs (see Fig. 1) when data is observed under different O(2) roto-reflections within the plane perpendicular to gravity. Since the network does not need to learn to generate consistent results across O(2) roto-reflections, it can overall generate better displacement and covariance estimates and, thus, improve the trajectory tracking in the EKF (when the frame is applied to TLIO). Since the measurement equation of the EKF depends on the IMU yaw orientation (see Eq. 14 in Liu et al. 2020, and Appendix A.6.4), better displacement measurements should result in better IMU yaw estimation and thus lower yaw drift. However, experimentally we observed this effect is very small, showing only a small improvement on the Aria Dataset (Tab. 3 see 2.073 deg average yaw error (AYE) for TLIO + rot. aug. vs. 2.059 deg AYE for TLIO + O(2) Eq. Frame).

W1. It is not clear if ICLR is the best venue for such research, since learning components here is rather limited to a regression module.

Equivariance has been a prevalent topic in ICLR. Because the canonicalization is a preprocessing step and yields a group action, the data transformed with this action can be fed into any architecture beyond regression.

W2. Uncertainty modelling assumes diagonal covariance. Validity of these assumptions are tested by looking at the final performance that it helps. Perhaps an in-depth analysis on this step could help, despite not the core focus of the paper. For example, there has been many evaluation tools from uncertainty quantification literature, and can be presented here.

We apologize for not formulating this in a clear way. The covariance is diagonal in the canonical frame but after back-transforming it with a roto-reflection (predicted by the canonicalization) the resulting covariance becomes non-diagonal. This is a feature of our system that the meaning of our canonical frames is the orientation of the covariance matrix.

In addition to supporting this design choice empirically by running our neural network (see Tab. 3), we add additional statistical analyses below, using two more techniques: (1) we report the percentage of displacement predictions by our network that have an error outside of the 3-sigma bound, in x,y, and z direction (denoted $\delta_x,\delta_y,\delta_z$ in percent), where sigma is given by our network, and (2) we compute the median negative log-likelihood (median NLL) of the ground truth samples given the mean and covariance provided by our method. All results were calculated in the canonical frame and pertain to the Aria Dataset.

In addition, we highlight the importance of learning the covariance, by introducing two additional baselines: First, similar to Liu et al 2020, we use our pre-trained method to generate displacement measurements, but use a constant isotropic covariance (sigma=0.01) instead of the learned one (indicated with +constant cov), and secondly we train our model only on MSE for 20 epochs (indicated with (mse)+constant cov) and deploy it with the same constant covariance.

We see that TLIO + O(2) Eq. Frame, TLIO + O(2) Eq. Frame + P and TLIO + O(2) Eq. Frame (mse) + constant cov, all show low outlier counts. However, low outlier counts can also be achieved by predicting high covariances. However, this strategy increases the median NLL, as seen in TLIO + O(2) Eq. Frame + P and TLIO + O(2) Eq. Frame (mse) + constant cov, showing that they overestimate covariances. Our method (TLIO + O(2) Eq. Frame) shows the highest median NLL overall methods, with low outlier counts.

For completeness, we also report the tracking performance of the new methods below

We see that, first, O(2) Eq. Frame + constant cov performs worse than our method, indicating that our learned covariance adapts to the specific learned displacements. Second, we see that (mse)+constant cov performs even worse. This is likely due to the network being overconfident in its displacement prediction before MLE finetuning, which results in significant outlier prediction.

We have included this analysis in Appendix A.17 of the revision. We look forward to your suggestions to further analyze the various covariance parameterizations used in the paper.

W1. The primary weakness of this paper is the clarity of its writing. I’m unable to fully understand the major differences between this work and RIO: Rotation-equivariance Supervised Learning of Robust Inertial Odometry.

While our work and RIO both aim to address the equivariance of neural inertial odometry by modeling rigid transformations of trajectory and IMU data, there are several distinct differences.

(1) Most significantly, RIO addresses approximate equivariance, while our method enforces strict equivariance. Approximate equivariance enforces equivariance via an equivariance loss which penalizes inconsistencies in predictions on data from four rotated trajectories. By contrast, strict equivariance enforces inherent consistency by tailoring specific neural network components to be exactly equivariant. This work proposes several specialized linear and non-linear layers to guarantee the strict equivariance of the predicted canonical frame, while RIO uses traditional layers.

(2) RIO adopts a Test-Time Training strategy, which adds computational overhead and thus increases the inference time as they are required to store multiple versions of the prediction model and augmented trajectories. Since our method is strictly equivariant, we do not require a Test-Time Training strategy. Moreover, despite not using such a strategy our method outperforms RIO.

(3) RIO only handles $2D$ displacement outputs and equivariance to rotations in SO(2), i.e. $2D$ planar rotations. By contrast, our method can handle both $2D$ and $3D$ outputs and thus addresses equivariance in $2D$ and subequivariance in $3D$.

(4) Last, we also extend the modeling to roto-reflections, i.e. the group $O(2)$, which comprises rotations and reflections in the plane perpendicular to gravity and requires a novel bijection for angular rate preprocessing.

To highlight the difference in equivariance types (strict/approximate), we included a visualization of the different strategies in the revision Appendix A.4.

W2. While the key idea of this paper is clear, it’s difficult to discern how it specifically diverges from the previous work. I strongly recommend that the authors begin by clearly outlining the main concepts, followed by a detailed description of the methodology. This structure would greatly help readers in understanding the unique contributions of this work.

As discussed previously, our work significantly diverges from the previous works mentioned above (RIO) in four major ways (i) approximate vs. strict equivariance, (ii) test-time training vs. no test-time training, and (iii) $2D$ vs. $3D$ modeling and (iv) equivariance to rotations in $SO(2)$ vs. roto-reflections $O(2)$. Please see our previous response for a recap.

We also thank the reviewer for the recommendation of reworking our method, and have tried our best to include the preliminary theory that is necessary to understand the concepts in this work in Appendix A.4. We supplemented this with additional pointers to textbooks and prior works. Regretfully, we believe, due to space considerations, that it is out of scope for us to include more preliminary theory in the main text.

Q1. What is the roto-reflection group, and why is it important? A more detailed explanation of this concept and its relevance would be helpful.

The Roto-reflection group $O(n)$ is a set of $n\times n$ matrices $R$ (in this work $n=2$), which are orthogonal, i.e. $RR^T = I_n$. Their determinant can be 1 or -1. This set of matrices forms a group because (i) it has an identity element $I_n$, (ii) the product of any two matrices from $O(n)$ is in $O(n)$, and (iii) every matrix $R\in O(n)$ has an inverse $R^T\in O(n)$. As shown in Figure 3a, the orange trajectory is obtained by reflecting the reference trajectory, and the purple one is obtained by rotating the reference trajectory. The reflections and rotations are the transformations in the roto-reflection group. We illustrate how this transformation acts on the inputs of the neural network in Figure 3b.

W1. The general canonicalization scheme has been proposed before by Kaba, Sékou-Oumar, et al. "Equivariance with learned canonicalization functions." ICML 2023, and cannot not be presented as a contribution. The original work must be cited.

Thanks for pointing out the original work, which we cited in the revision (Section 2). As correctly pointed out, we achieve equivariance via the canonicalization scheme presented in Kaba et al. We want to highlight that our novelty is in characterizing and addressing the subequivariant structure in neural inertial odometry where the learned network predicts the displacement priors. To this end, we design a subequivariant framework by applying canonicalization and designing specific basic layers.

We made the following modifications in the revision: (i) cite the work of Kaba et al. in the related work section, and add a separate related work section on canonicalization in Appendix A.1 due to space constraints, including citations of previous and subsequent papers on canonicalization. and (ii) modify our introduction to make clear that our novelty is in applying canonicalization to IMU data, and that, to achieve this, we tailor specific basic equivariant layers and a bijective map.

W2. While there is a reduction in drift compared to the baselines, the remaining drift is still significant (>2m) which suggests that the main problem in IO is not in exact equivariance but elsewhere (most likely sensor noise).

The lack of principled subequivariance modeling in previous approaches accounts for a significant increase in drift, and we address this with the presented work. In fact, by correctly accounting for this geometric constraint, EqNIO reduces the MSE* by 57%, the ATE* by 12%, and the RTE* by 11% on the Aria Dataset (see Tab. 1).

As rightly pointed out, there still exists drift compared to VIO, and this highlights the fact that pure IO is a historically hard, but equally exciting problem to study. Incorrect biases in addition to sensor noise are two known drivers of drift in learning-based IO. To address this, Brossard et al., 2020a, Brossard et al., 2020b and Buchanan et al., 2023 train models to predict either IMU biases or debiased IMU data directly, which reduces drift by a significant margin. We believe that our framework can readily incorporate such methods to replace the current way of debiasing IMU measurements, which relies on factory-calibrated bias values.

Q1. Why is this canonical. equiv. scheme chosen over other equiv. choices? e.g. frame averaging (Puny et al. ICLR '22) also allows adapting existing non-equiv. Architectures.

We thank the reviewer for this suggestion and present results for an additional baseline following the frame-averaging technique in Puny et al. ICLR ‘22 applied to our O(2) model. We first perform PCA (using the torch.pca_low_rank() implementation) on the IMU data, which results in a set of four frames, corresponding to the four solutions of PCA. We then use the equivariant format in Eqn.5 of Puny et al. ICLR ‘22 to average the projected predictions. We show the results below, marked as TLIO + frame averaging, and include them in Appendix A.16 of the revision:

We see that while TLIO + frame averaging achieves a low error on the TLIO dataset it fails to generalize to the Aria Dataset. Furthermore, our equivariant method (TLIO + O(2) Eq. Frame) outperforms it on both datasets and across metrics. We believe that the subpar performance of TLIO + frame averaging stems from the noise sensitivity of PCA. Due to this sensitivity, it likely overfits to the specific noise level present in the TLIO dataset, which does not match the one in the Aria dataset. Lastly, we want to note that the frame averaging technique significantly increases the inference time by a factor of four, as the model needs to be inferred several times, once for every constructed frame.

Thank you for addressing all my questions and comments.

The accuracy improvements still seem marginal to me (ATE / RTE are more informative performance metrics than the MSE loss) compared to the increased complexity and run-time (on-device run-time will be even higher). Nonetheless, this work is still an interesting and novel application of learned canonicalization that is well developed and has potential for future work. Thus I have increased my score from 5 to 6.