F2M-Reg: Unsupervised RGB-D Registration with Frame-to-Model Optimization

Q1: The claim about 'unsupervised RGB-D Registration' / 'unsupervised learning'

Following previous work [1] [2] [3] [4], unsupervised RGB-D registration aims to training the registration model without the supervision of the ground-truth poses. This line of work usually first computes the pseudo poses between the point cloud pairs, and trains the model based on the pseudo poses. Our method also follows this paradigm, with a novel frame-to-model optimization method to obtain accurate pseudo poses. And we propose a model initialization method based on synthetic data. On the other hand, zero-shot learning focuses on predicting novel classes or domains without the corresponding data, which is different from this task.

Q2: More evaluations on different datasets.

We compare our method with PointMBF on TUM RGB-D under the $50$-frame setting. And our method surpasses PointMBF by $9.4$ pp on Rotation Accuracy@$5^{\circ}$, $12$ pp on Translation Accuracy@$5\text{cm}$, and $9.5$ pp on Chamfer Accuracy@$1\text{cm}$. These results have proven the strong generality of method to new datasets. We will add these results in the future version.

Q3: Work on the unordered data.

It is common practice for unsupervised RGB-D registration methods [1] [2] [3] [4] to train the registraion model with RGB-D sequences. The main reason here is to guarantee that there is reasonable overlap between two frames, which is difficult to achieve with unordered data. It is the overlap that counts, rather than the orderness. And this is the same for Co-SLAM part. Theoretically, if the unordered data are organized to guarantee the overlap between frames, our method should also work. Besides, our method only leverages RGB-D sequences during training, and we do not rely on RGB-D sequences during inference and thus can handle various multi-view RGB-D data in applications.

Q4: The reason on why F2M approach superior to F2F approach.

Neural implicit field still has challenges in handling multi-view inconsistency, especially in extreme cases. However, compared to the traditional rasterization process used in the frame-to-frame methods, neural implicit field is well-known for its stronger capability to handle multi-view inconsistency as noted in NeRF [5], NeuS [6], etc. By encoding the view direction information, the neural implicit field can render more geometrically and photometrically consistent images, which helps optimize a more accurate pose to train the registration model. On the contrary, the frame-to-frame method only project one frame into the other and compare the consistency between two images, which is more easily affected by the multi-view inconsistency.

I really appreciate the comprehensive responses from the authors. Some of my concerns regarding the F2F and F2M, the integration of Co-SLAM, and the training strategies are sufficiently addressed. The authors provide additional experiments on TUM-RGBD and showing their superiority over the baseline method. However, no experiments on the recommended ScanNet++ (Yeshwanth et al.) are provided. Considering the performance issues also highlighted in other reviews, I keep my score unchanged.

A minor suggestion: The PDF file can be modified and reuploaded to reflect the revisions. Instead of promising future changes, it is more straightforward to show and highlight the revisions directly.

Yeshwanth, Chandan, et al. "Scannet++: A high-fidelity dataset of 3d indoor scenes." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

The additional experiments present a solid demonstration for the effectiveness of the proposed method. I suggest incorporating both the TUM-RGBD and ScanNet++ results into the revised paper. Considering the thorough response, I am pleased to raise my score to 6 (or 7 if that were an option) to reflect the good quality of the paper.

Q1: Where the improvements come from.

We have separately studied the improvements from each component in Tab.2, Tab.3, Tab.4 and Tab.5 in the main paper. Here, we reorganize the results in the table below for a better understanding. BS refers to the synthetic bootstrap, F2M refers to frame-to-model optimization, and F2F refers to the frame-to-frame optimization. Independently applying synthetic bootstrap and frame-to-model optimization both achieves significant improvements compared with the frame-to-frame baseline, demonstrating the strong effectiveness of our designs. And applying both of them further improves the results, achieving the improvements of $6$ pp over the BS-only model and $3$ pp over the F2M-only model on most metrics. We will refine the descriptions in the future version.

Q2: The effectiveness of the neural implicit field-guided mechanism.

The results have already been shown in the Tab.2 and Tab.4 of the main paper and we put them together in the table below. Under the $50$-frame setting, our method without bootstrap surpasses the previous state-of-the-art method PointMBF by $14$ pp on Rotation Accuracy@$5^{\circ}$, $6.8$ pp on Translation Accuracy@$5\text{cm}$ and $9.6$ pp on Chamfer Distance Accuracy@$5\text{cm}$, showing strong superiority.

We further show the comparisons under the $20$-frame setting in the table below. As this setting is relatively easy as the frame pairs share larger overlap, our method without bootstrap still outperforms the PointMBF baseline.

Q1: About the performance of w/o warm-up registration model.

A: As already described in the paper, our frame-to-model method is more sensitive to model initialization as it needs to track the poses of more frames to optimize a neural implicit field, as we design a synthetic bootstrap method to solve this problem. To further investigate the effectiveness of our approach, we compared the results of applying synthetic warm-up to the frame-to-frame method, PointMBF. The results are shown in the tables below, where both methods were trained on 3DMatch and ScanNet. The first table presents the comparison under the 50-frame-apart setting, while the second table corresponds to the 20-frame-apart setting. Across both settings and training datasets, our method consistently outperforms the frame-to-frame baseline, even with synthetic warm-up applied.

It is also worth mentioning that our method was trained for only 1 epochs on 3DMatch, whereas PointMBF was trained for 12 epochs. To provide a clearer comparison, we report the results after training for 4 epochs without bootstrapping in the following table. As shown, our method continues to demonstrate improvements, further validating its effectiveness.

Q2: About the results under the 20-frame setting.

A: The $20$-frame setting is more like a easy setting instead of a common one as the viewpoint changes are small, and thus the performance tends to be saturated. However, in real applications, the $20$-frame setting does not always hold. For example, in the scenes with fast motion or low frame rate, the frame-to-frame methods could fail but our method still provides accurate pose estimations.

And further we further report the results on 3DMatch in the table below.

Q3: About the low-overlap cases.

A: First of all, we would note that our method and PointMBF share the similar registration model, but differ in the training methods, i.e., frame-to-model v.s. frame-to-frame. It is one of the most disadvantages of the frame-to-frame methods that they cannot estimate accurate poses for the low-overlap pairs during training, which leads to suboptimal convergence of the model. On the contrary, our frame-to-model can provide better poses for the low-overlap pairs, which contributes to a more robust model, as shown in Tab.2 of the main paper. These results explicitly demonstrate the superiority of our method. Morever, we compare the poses during training estimated by both methods on pairs with rotation > $20^{\circ}$ on ScanNet scene0000_00 in the below table. It can be seen that our F2M method achieves obviously better pose optimization during training, providing the registration model with better supervised signals.

Q4: About the differences between ScanNet and 3DMatch.

A: We show the statistics of the rotations, the translations, and the overlap ratios between the training pairs for both datasets. It can be seen that the two datasets do not differ much in data distribution. However, the size of the two datasets shows significant differences, i.e., 1045 vs 71, which could be the main cause of the performance differences.

Q1: The effectiveness of the warming up dataset.

In the table above, we compare PointMBF, F2M-Reg w/o bootstrapping, and full F2M-Reg under the $20$-frame setting on ScanNet. Our method still outperforms PointMBF even without bootstrapping, especially it can reduce extremely erroneous registrations. We further show the results under the $50$-frame setting (see the table below) with more severe multi-view inconsistency due to light variations and geometric occlusion. In this setting, our improvements are more significant, i.e., $14$ pp on Rotation Accuracy@$5^{\circ}$, $6.8$ pp on Translation Accuracy@$5\text{cm}$, and $9.6$ pp on Chamfer Distance Accuracy@$5\text{cm}$. These results further demonstrate the superiority of our frame-to-model design.

Q2: 50 frames apart in the evaluation phase.

The $50$-frame setting aims to evaluate the performance under low overlap, which is common in real applications with fast motion or low frame rate. It is more challenging due to the severe interference of the non-overlap region. In the evaluation, we exhaustively include all consecutive pairs which are $50$ frames apart, e.g., frame $1$ and $51$, frame $51$ and $101$, etc. We do not exclude any pairs even if they have extremely low overlap or no overlap. This further increase the difficulty of this setting. In the paper, we do not show the mean and the median errors in Tab.2 of the main paper, which we show below. The table below show the mean and median errors of the registration model trained on ScanNet. The results are clearly worse than those under the $20$-frame setting, but our method still outperforms PointMBF by a large margin.

Q3: The influence of the warming up dataset.

About the dataset size. To investigate the influence of the dataset size, we use $20$, $40$, $60$ and $80$ scenes to bootstrap the model and directly evaluate without fine-tuning. The results are shown in the table below. It is observed that increasing the dataset size increases the performence, but the improvements are not significant. This means the synthetic bootstrap does not rely on a large synthetic dataset.

About the similarity. Actually, our Sim-RGBD dataset is already very different from the real datasets. First, our dataset is constructed from ShapeNet, which contains the objects which cannot appear together in real scenes, such as planes and guitars, buses and chairs, etc. This induces inconsistent semantics and geometries between the synthetic and the real datasets. Second, our dataset puts the objects densely and randomly in the space, without collision detection or gravity, so there are severe layout differences between the synthetic and the real datasets. Nevertheless, the synthetic bootstrap still effectively provides a good initialization of the registration model, which demonstrates that the synthetic and the real datasets do not have to be very similar.

About other synthetic dataset. We conducted an ablation study by training on the Replica dataset and testing on ScanNet. The results of these experiments are summarized in the table below. The findings indicate that training on Replica yields performance comparable to training on Sim_RGBD. This demonstrates the robustness and generality of our bootstrapping stage for utilizing simulation data across different datasets.

Q4: Time efficiency.

Yes, our method increases the training time, mainly coming from the tracking and the mapping stages during optimizing the neural implicit field and the relative poses. However, we can reduce the tracking iterations (which we use $100$ in the experiments) for a balance between accuracy and efficiency. And as shown in Tab.7 of the main paper, reducing the tracking iterations only marginally influences the performance, but effectively reduces the training time. Moreover, we would also note that PointMBF uses $20\times$ training data than ours. PointMBF uses all pairs which are $20$ frames apart but we sample the training samples in a consecutive manner. For this reason, our method is more data-efficient than PointMBF.

Thanks the authors for providing these extra results.

From the table, we can conclude that the improvement on ScanNet of 20 frames over PointMBF mostly comes from the extra training data. From the table, I may not agree with the conclusion that "Our method still outperforms PointMBF even without bootstrapping, especially it can reduce extremely erroneous registrations." In some metrics, Trans(5) Chamfer(5), the method is worse. I can say the two methods are about the same performance.

From Table2 and Table4 on 3DMatch of 50 frames apart setting, it also shows without introducing extra training data, the performance of the method measured by strict metrics standard is also similar (or worse?) to PointMBF. In the setting of 20frames in Table 1, will the result of the method with introducing extra dataset also similar to PointMBF or worse？ Rot5 Tran5 Chamfer 1 PointMBF 59.3 34.2 42.9 The method 59.0 30.4 43.2

Yet, the method has shown improvements on erroneous cases. Among all the comparison on different datasets, the method shows improvements on ScanNet on 50 frames apart and as authors said in the setting of 20 frames, the method also shows improvements over pointMBF on less strict metrics like large error threshold (Rot10Trans10) .

Q3: About the performance of w/o warm-up registration model.

A: As already described in the paper, the frame-to-model approach is more sensitive to the model initialization as it needs to track the poses of more frames to optimize a neural implicit field. This sensitivity prompted the design of our synthetic warm-up method to address the issue. For more comparisons, we apply the synthetic warming-up on both F2F and F2M approaches on both datasets in the following tables: the first table corresponds to the 50-frame setting, and the second table to the 20-frame setting. These comparisons highlight the superiority of the F2M approach over the F2F approach, demonstrating its effectiveness in training the registration model.

Q1: About the definition of 'bootstrap'.

We use the term 'bootstrap' in this paper to describe the process of initializing the model for more effective unsupervised training. This process is achieved by leveraging the synthetic data. We admit that the usage of this term is somewhat inaccurate, and we will replace this term in the future version.

Q2: Synthetic Bootstrapping achieves more improvements on 3DMatch than ScanNet.

The reasons for this inconsistency are two-fold. First, as we use ScanNet for testing in all the experiments, there is naturally a domain gap for the model trained on 3DMatch which limits its performance. Second, ScanNet contains more scenes than 3DMatch ($1045$ v.s. $71$ for training), which also affects the generalization of the model. By leveraging the pre-training, the various synthetic data and the supervised training manner helps fill this gap and improve the generalization of the model.

Q3: Effectiveness in dynamic scenarios.

Yes, this paper focuses on the unsupervised RGB-D registration For the pre-training stages,in static scenarios like previous work UR&R [1], BYOC [2], LLT [3], PointMBF [4] and unsupervised RGB-D registration in dynamic scenarios is out of the scope of this paper. However, we would note that there are also neural implicit fields for dynamic scenes, which could be integrated into our pipeline to solve the dynamic problem. We will leave this as the future work.

[1]: Mohamed El Banani, Luya Gao, and Justin Johnson. Unsupervisedr&r: Unsupervised point cloud registration via differentiable rendering. CVPR 2021, pp. 7129-7139.

[2]: Mohamed El Banani and Justin Johnson. Bootstrap your own correspondences. ICCV 2021, pp. 6433-6442.

[3]: Ziming Wang, Xiaoliang Huo, Zhenghao Chen, Jing Zhang, Lu Sheng, and Dong Xu. Improving rgb-d point cloud registration by learning multi-scale local linear transformation. ECCV 2022, pp. 175-191.

[4]: Mingzhi Yuan, Kexue Fu, Zhihao Li, Yucong Meng, and Manning Wang. Pointmbf: A multi-scale bidirectional fusion network for unsupervised rgb-d point cloud registration. ICCV 2023, pp. 17694-17705.

Thank you very much for recognizing our work and for the honor of us being able to address your concerns!