ROMEO: ROBUST METRIC VISUAL ODOMETRY

1. The technical contribution and insights appear limited. The integration of additional depth estimation models to enhance the performance of monocular VO or SLAM systems has become a well-recognized and practical solution. The primary distinction of this paper compared to previous literature is the incorporation of both monocular depth estimation models and multi-view stereo (MVS) models.
2. The paper presents a less rigorous concept that could potentially mislead the community. The authors claim that existing monocular VO approaches lack robustness and fail to generalize to unseen data, particularly in recovering metric-scale poses. However, this paper overstates its contributions, as it relies on predefined scale and shift hyperparameters to convert predictions from the DPT model to metric or absolute scale. These parameters must be computed using known ground truth to achieve reasonable performance across different benchmarks, rendering the method neither robust nor generalizable to unknown data.
3. The operational speed of this system (3-20 fps) is significantly slower compared to the baseline DPVO, which claims to achieve speeds of 60-120 fps.
4. The schematic diagram, specifically Figure 2, is cluttered, making it difficult to discern the data flow for training and inference.

1. The pipeline flow is not completely clear from the method description and Figure 2. It would be beneficial to separate the trained or frozen components (Depth, Flow, MVS Networks) from the test-time optimization (differentiable BA, sliding window optimization) in the figure and the text. A figure for the Depth-guided BA (Section 3.1) seems necessary to understand the optimization procedure.
2. No demo is provided to demonstrate the proposed system's pipeline step-by-step and performance.

• The paper's motivation is questionable. Although the results are insightful, the motivation is not sufficiently compelling. The authors claim that introducing pre-trained monocular depth and multi-view stereo neural networks helps recover scene scale. However, the scale of each frame is merely a scalar, and using dense depth priors for this purpose significantly increases the system's computational load (with system efficiency below 30 FPS, making it non-real-time). A more elegant, well-designed "scale estimation network" might achieve similar results with less overhead.

• The discussion in this paper is lacking as follows:

  1. The reasons why pre-trained monocular depth and multi-view stereo neural networks aid in scale recovery are not fully explored. It could be due to the proposed strategy or the inherent capabilities of the networks used. Although some intuitive explanations are provided in the ablation study section, more extensive experimental evidence is needed;
  2. The introduction of pre-trained monocular depth and multi-view stereo neural networks did not significantly improve accuracy in indoor scenes. This suggests that in scenarios with good initial values and minimal scale variation, depth priors may not substantially enhance accuracy. This might be due to the difficulty of obtaining good initial values in outdoor scenes or rapid scale changes;
  3. The upper performance limits of the algorithm should be explored, such as providing ground-truth initialization or absolute scale input for comparison methods;
  4. The efficiency of the algorithm varies notably between indoor and outdoor environments. In Table 5, RoMeO-VO achieves FPS values of 3.54 on the KITTI dataset and 7.65 on the TUM-RGBD dataset, well beyond the range influenced by noise, yet the authors do not address this discrepancy.

main issue: Several of the comparisons are problematic. The teaser compares to a method that does not estimate absolute scale (Fig.1 (b) right-side) and the proposed method is explicitly overfitted to KITTI data (uses a depth method which was explicitly overfitted to KITTI, which is particularly problematic for a driving setting with a fixed camera-road configuration throughout the whole dataset which allows almost trivially perfect scale estimates, especially for the road in front of the vehicle). Notice that the better absolute depth can also help relative pose estimates as it provides consistent estimates across views, so this might not only affect ATE, but also RTE errors. Therefore, Fig. 1 (a) should be recomputed excluding any KITTI data. Similarly Fig. 4 is fundamentally problematic due to the KITTI overfitting issue (of DPT). All Tables should excluded KITTI data, and have recomputed averages. The overfitting can directly be seen from e.g. Table 2 which still has good performance for 4Seasons and Cambridge, but nowhere close as good as on Kitty. Kitty odometry results should NOT be used given the explicit overfitting in DPT (with outdoor hyperparameters), and thus also not be included in any average. Also, separate averages should be computed between indoor and outdoor datasets given the huge difference in scale (otherwise results are dominated by outdoor errors).

No ATE values should be provided for methods that do not attempt to compute metric scale!

other issues: Equation (1) does not seem to be invariant to the overall scale of the observed scene with the first term being photometric and the second term proportional to absolute depth. It is suprising that this works well on indoor and outdoor datasets, but might be thanks to C_i learning to switch off predictions that are further away (although they would carry useful information).

Also problematic is that different between absolute depth is used as an error, while for this typically inverse depth is much better (as the depth uncertainty from images is proportional to inverse depth or inverse squared depth depending on the scenarios).

The authors incorrectly equate their uncertainty weighting to Mahalanobis distance. Also, it is not clear why monocular depth estimates are also weighted by uncertainty on flow, given they should be independent.

It was not clear how the initial absolute depth estimate is obtained given l.194 stating that for initialization depth priors are turned off. Assuming the authors do incorporate depth priors, this might need clarification/reformulation to make this part more clear.

l. 232 indoor KITTI and outdoor TUM is swapped. l. 249 0.1m is an absolute threshold which is unlikely to generalize well between indoor wearable and outdoor vehicles, as well as drones, etc. A metric that is proportional to observed parallax would make more sense. It is also not clear why the authors would turn off MVS guidance when the camera travels straight... This makes no sense. Notice this would actually turn of MVS guidance for most of KITTI and other outdoor datasets (driving down straight roads).

The major concern is that the paper in the existing form provides little knowledge advances beyond better results. I don't doubt its efficacy on these evaluated datasets, but the straightforward solution requires better analysis to uncover the major issues regarding leveraging a pre-trained depth estimation model:

• What are the typical failure modes on diverse datasets with a naive DROID-Metric3D baseline?
• Is there any domain gap between different datasets?
• What are the differences between monocular depth estimation and MVS results?
• As there is no regularization term regarding the scale consistency, is the metric scale provided by Metric3D or DPT good enough to be ignored?

Besides the understanding of the robust metric visual odometry problem, the paper lacks sufficient understanding of its own strategies. All strategies to ensure 'robustness' rely heavily on hyperparameters ($\alpha$ in L207, Eq. 3, 20% in L279, small learning rate in L287, parameters in L310 to switch between indoor and outdoor scenes). It is unclear if the system is sensitive to these parameters. Besides, several arguments are not convincing to me:

• Why depth regularization is not enabled during the initialization (L194)?
• Why better depth estimation models lead to worse results as in Tab. 1 and Tab. 4 (DPT $\rightarrow$ Metric3D)? I don't see any relevance between the accuracy and the overhead as the system does not reduce the number of frames for depth estimation to balance the efficiency in this experiment.
• What about the computational cost of MVS depth estimation?
• Sec. 3.4 is not well validated. There is no evidence that 'purposely maintaining noise' (L284) leads to better robustness. It is also weird to ignore MVS depth during fine-tuning to 'ensure sufficient depth noise' (L288).
• More trajectory results in different datasets should be provided to understand how other baselines fail. Video demonstration of the online dense reconstruction may better help the readers understand the major challenges of the 'robust metric visual odometry'.

Without a clear analysis of missing pieces toward robust metric visual odometry', the paper fails to address the motivations behind its system design, but instead adopts a simple fashion to only leverage reliable (heuristically determined) depth results for depth regularization. The major improvement regarding the pose estimation is on outdoor datasets where DROID-SLAM performs badly, where the robustness in indoor environments due to the proposed method is not clearly presented. The current form is not sufficient enough to be spread in the ICLR community. I list several suggestions in the following [Questions] section so the authors can improve the paper's quality with a more in-depth analysis.