RESfM: Robust Deep Equivariant Structure from Motion

This paper studies an important problem in multi-view structure from motion, specifically aiming to recover 3D points and camera poses from input point tracks. To this end, based on an existing permutation equivariant architecture, an outlier classification module is integrated to cope with outliers. A robust Bundle Adjustment method is also proposed, based on recursive potential outlier removal. Experiments are conducted on multiple datasets, demonstrating the robustness of the proposed method against existing ones.

In this work, authors robustify the existing deep equivariant SfM method from [1] by incorporating an inlier-outlier prediction module and robust Bundle Adjustment. Specifically, deep SfM uses projective factorization on a 2D point track matrix, but the method in [1] assumes that this matrix is complete and outlier-free. Authors in this work argue that these assumptions are unrealistic. Hence, authors proposed a deep equivariant SfM that works on a large collection of internet photos and works with a matrix of point tracks that might have outliers. In their method, they first train an end-to-end outlier prediction and camera poses + 3D point prediction network from a matrix of 2D point tracks. At test time, they use "inlier-outlier" predictions from the pre-trained network to remove outliers and fine-tune the network for 3D pose + 3D points recovery using unsupervised losses for each scene.

Authors show impressive results across different datasets. Their method achieves the best results among deep SfM methods and achieves comparable results to the best classical SfM methods.

References:

[1] Moran, Dror, et al. "Deep permutation equivariant structure from motion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

The paper presents an architecture for Multiview Structure from Motion (SfM) focusing on the robust recovery of camera pose and 3D scene structure from large, uncontrolled image collections that contain numerous outliers. Traditional and deep-based SfM approaches often struggle with outlier point tracks resulting from viewpoint variations, illumination changes, and repetitive structures, significantly impacting their performance. The authors propose an enhancement to the deep network architecture proposed by Moran et al. in 2021, incorporating an outlier classification module within a permutation equivariant framework. The revised architecture also integrates a robust bundle adjustment step to correct classification errors. The method has been tested on challenging datasets like MegaDepth and 1DSFM and has shown good accuracy and competitive runtimes against both contemporary deep-learning methods and established classical techniques.

We deeply appreciate the reviewers for their insightful feedback!

We would like to emphasize the importance of our contribution:

Significance: The removal of outlier matches is a critical step in Structure-from-Motion (SfM). Methods like ESFM (Moran et al., 2021) and GASFM (Brynte et al., 2023), which do not effectively handle outliers, have been applied only in limited scenarios—such as the outlier-free Olsson’s dataset (Olsson & Enqvist, 2011), which was captured with a single camera, small motion, and no light changes. When these methods are applied to more challenging datasets, such as those consisting of internet photos, they result in poor accuracy. Our work introduces an effective, deep network-based approach to outlier removal in a global SfM setting, addressing this critical gap.

Novelty: We have introduced the novel task of multi-view outlier removal and will make our benchmark and datasets publicly available. We believe future research can build on our work to further improve performance. Similar to the advancements seen in classical global methods, which culminated in GLOMAP (Pan et al., 2024) being on par with the state-of-the-art incremental COLMAP, we hope our contribution will serve as a foundation for future progress.

Approach: Our goal is to simultaneously recover both camera poses and 3D structure. To handle outliers in this context, we integrate an inlier/outlier classification module within a sets-of-sets permutation-equivariant architecture. This architecture efficiently manages point tracks while respecting their symmetries, thus eliminating the need for data augmentation. We further explain the details of this equivariant architecture below.

Importance of Equivariance: Our network classifies individual track points as inliers or outliers by considering both their track and image memberships. The sets-of-sets-equivariant architecture allows the hidden layers for each track point to interact with other points both within and across cameras. This eliminates the need for exhaustive augmentation processes. Our comparison to a PointNet-based architecture in Table 7 demonstrates the importance of leveraging sets-of-sets equivariance in improving performance.

Results: Our results demonstrate that our method can efficiently handle challenging datasets, including those with hundreds of internet photos, achieving great accuracy. In contrast, existing deep learning methods like ESFM and GASFM fail to handle such datasets due to their inability to remove outliers effectively. Other robust deep learning-based methods also struggle with large-scale datasets of internet photos. We believe this marks an important step in advancing deep learning as a leading technique for large-scale SfM.

We have made the following revisions in the text:

1. Thanks to the insightful question from reviewer oVEX, we identified a bug in the GLOMAP code. Specifically, in some reconstructions, unregistered cameras (cameras without any associated 3D points) were not filtered out. This oversight negatively impacted GLOMAP’s performance. We have corrected this issue and updated the GLOMAP results, which affected the 1DSfM and MegaDepth datasets (Tables 1, 2, 9, and 10). While the updated results show an improvement for GLOMAP, the overall conclusions of our comparisons remain unchanged.

2. The best-performing method is now marked in bold and underlined, we hope this makes the tables clearer for readers.

3. At the request of the reviewers, we have added the mean errors to Tables 1, 2, 9, and 10 for the 1DSfM and Megadepth datasets to enhance clarity.

Summary

The paper presents a new architecture for Multiview Structure from Motion (SfM) that improves the recovery of camera pose and 3D scene structure from large, uncontrolled image collections with outliers. The architecture incorporates an outlier classification module and a robust bundle adjustment step, achieving good accuracy and competitive runtimes against contemporary deep-learning methods and classical techniques.

##Strengths

• Integrates a new inlier-outlier classification branch and a robust structure from motion (SfM) mechanism.
• Extensive experiments conducted over various datasets to prove claims, demonstrating the efficacy of these components.
• The proposed inlier-outlier prediction head shows improvement compared to ESFM.
• Uses unsupervised reprojection losses for easy fine-tuning for different scenes.
• The paper is well-presented with a comprehensive literature review and extensive experiments.

Weaknesses

• The ablation study section needs more consideration, as it should focus on why error handling designs are better than proving the base model is better.
• Concerns arise regarding the reliability of the results, as the rotational metrics are reported in degrees, not radians.
• The authors did not use Mast3R [2] as a baseline method due to its limitations in working with large number of images.
• The paper's contribution is largely built on existing network architecture, with the addition of outlier classification output channels.
• The proposed solution for handling outliers incurs additional overhead, lacks theoretical justification, comprises inliers recall, and can not reject outliers well.

Conclusions

Based on the reviews and author's feedback, the paper shows an interinting approach and should be accepted as most of the concerns were addressed by the authors.

Accept (Poster)