3D Focusing-and-Matching Network for Multi-Instance Point Cloud Registration

We appreciate your diligent review and valuable feedback to help improve our paper.
 

Q1: The contribution and innovation of the method.

A1: Our contributions lie in three aspects:
(1) Our primary contribution does not lie in the network architecture but rather in proposing a new pipeline to address the multi-instance point cloud registration problem. Existing methods (such as PointCLM and MIRETR) mainly learn correspondence between the one CAD model and multiple objects (one-to-many paradigm), while our method decompose the one-to-many paradigm into multiple pair-wise point cloud registration (multiple one-to-one paradigm) by first detecting the object centers and then learning the matching between the CAD model and each object proposal.
(2) Our new pipeline is simple yet powerful, achieving the new state-of-the-art on both Scan2CAD and ROBI datasets. Especially on the challenging ROBI dataset, our method significantly outperforms the previous SOTA MIRETR by about 7% in terms of MR, MP, and MF.
(3) The progressive decomposition approach of transforming multi-instance point cloud registration into multiple pair-wise registrations, as proposed in our paper, also holds significant insights for other tasks, such as multi-target tracking and map construction.
 

Q2: Reasons for not using density information (such as PointCLM) for clustering.

A2: PointCLM uses the feature density of contrastive learning to filter out matching pairs that do not belong to the instance before clustering. However, our method decomposes the one-to-many paradigm into multiple one-to-one paradigm by first detecting the object centers and then learning the matching between the CAD model and each object proposal. For the first detection stage, we can filter lots of background points by detecting the object centers and generating object proposals. For the second matching stage, the instance mask learning and overlap mask learning further filter the noisy points. To this end, we count the inlier ratio of points falling on objects and the total correspondence on the ROBI dataset. The following table show the results:

It can be observed that our method achieves the highest inlier ratio, which further demonstrate that our multiple one-to-one paradigm can effectively improve inlier ratio.
 

Q3: The article seems to be less theoretical, with a large number of formulas used to describe the structure of the network.

A3: As mentioned in A1, our contribution primarily lies in proposing a new paradigm for solving the multi-instance point cloud registration problem. Our method is simple yet powerful, achieving new state-of-the-art performance on both Scan2CAD and ROBI dataset.
 

Q4: I think this method of finding the center point first and then doing multi-instance point cloud registration is time consuming.

A4: Generally, two-stage methods are time-consuming than one-stage methods. In Table 2 of the main paper, we compare our method (3DFMNet) with existing methods (such as PointCLM, MIRETR):

Our inference speed of our two-stage 3DFMNet (0.54s) is lower than one-stage MIRETR (0.40s), while our inference speed is higher than two-stage PointCLM (0.63s). Although each method has different inference speeds, the inference speeds of these methods are all at the same level. In order to reduce the overall time consumption, one possible strategy for reducing the inference time of our two-stage method is to use parallel optimization to simultaneously match multiple pair-wise registrations. In the future, we will further consider reducing the inference time of our method.
 

Q5: The time spent of the center point detection and the registration.

A5: We supplement the detailed inference time of each stage in the following table:

It can be observed that the first stage (Focusing Model) tasks up a few time while the second stage (Matching Model) takes up a larger proportion of the time. Compared with the inference speed 0.400s of MIRETR, our total inference speed 0.540s is slightly lower than MIRETR.

The discussion period will end in approximately 5 hours. We sincerely hope that you could review our rebuttal and respond accordingly. Our rebuttal offers comprehensive explanations to the questions you raised, and we believe that our work shows a new yet powerful pipeline and high performance in multi-instance point cloud registration. We have also provided the discussion of density information and inference time analysis. We are confident that our responses will alter your perception of this paper. We look forward to your reply to our rebuttal. Once again, thank you for your dedication and efforts in the review process.

We thank the reviewer for the diligent comments to improve the paper.
 

Q1: How does the pair-wise registration model manage these falsely detected objects?

A1: Our method is a two-stage approach, so we analysis the falsely detected objects in both two stage.
(1) For the 3D multi-object focusing model (first stage), we compute the mean recall and mean precision of detect object centers. Specifically, we usex the distance between the predicted center and the ground truth center ($distance \leq 0.1 * r_{instance}$, $r_{instance}$ is the radius of instance) as a metric to determine the success of the 3D multi-object focusing module. The results are as following table:

On Scan2CAD and ROBI, it can be observed that our method achieve very high precision (98.85% and 99.99%), which means that our method can successfully detect the object centers.
(2) For the 3D dual-masking instance matching module (second stage), the instance mask learning and overlap mask learning further filter the falsely detected objects. Specifically, we study the influence of the false detected objects on the Scan2CAD dataset. According to the above table, it can be observed that there are 1.05% objects are falsely detected, i.e., there are 25 cases of failed detected objects on the Scan2CAD dataset. After the instance mask learning and overlap mask learning, the 22 cases are filtered. Since instance mask learning and overlap mask learning cannot effectively achieve the matching points between the falsely detected objects and the CAD model, there are few matching points obtained for falsely detected objects, resulting in SVD being unable to solve for the relative pose. Therefore, falsely detected objects slightly affect our final results.
 

Q2: Details of the metrics.

A2: We refer to the settings used in MIRETR and previous work to determine whether an instance is recognized as correctly registered based on RTE and RRE [1][2]. Specifically, we consider a match successful when $RTE \leq voxelsize * 4$ and $RRE \leq 15^\circ$. Following existing methods, such as MIRETR, the voxel sizes of Scan2CAD and ROBI dataset are set to 0.025m and 0.0015m, respectively.

[1] Yuan M, Li Z, Jin Q, et al. PointCLM: A Contrastive Learning-based Framework for Multi-instance Point Cloud Registration. In ECCV, 2022.
[2] Yu Z, Qin Z, Zheng L, et al. Learning Instance-Aware Correspondences for Robust Multi-Instance Point Cloud Registration in Cluttered Scenes. In CVPR, 2024.
 

Q3: Experimental results related to the 3D multi-object focusing module are needed.

A3: To evaluate 3D multi-object focusing module, we compute the mean recall (MR) and mean precision (MR) and root mean square error (RMSE) of detect object centers. Specifically, we usex the distance between the predicted center and the ground truth center ($distance \leq 0.1 * r_{instance}$, $r_{instance}$ is the radius of instance) as a metric to determine the success of the 3D multi-object focusing module. The results are as following table:

It can be observed that our 3D multi-object focusing module has good results in terms of MR, MP, and RMSE, which ensures that our method can successfully detect the object centers in the first stage.
 

Q4: The generalizability of the 3D multi-object focusing module to the unseen scenes.

A4: For the unseen scenes, we follow MIRETR and use ShapeNet dataset [1] (total 55 categories) to conduct experiments on invisible semantics. Specifically, we use CAD models of ShapeNet dataset from the first 30 categories for training and the remaining 25 categories for testing to assess generalization to new categories. To prevent class imbalance, we follow MIRETR to randomly sample up to 500 models per category. Each point cloud pair consists of a randomly selected CAD model and a scene model created by applying 4–16 random poses. This results in 8,634 pairs for training, 900 for validation, and 7,683 for testing. The results of testing set are as following table:

It can be observed that our method show good generalizability on unseen scenes compared to previous SOTA MIRETR. We will add the results in the revised manuscript.

[1] Chang A X, Funkhouser T, Guibas L, et al. ShapeNet: An Information-Rich 3D Model Repository. arXiv preprint arXiv:1512.03012, 2015.

We thank the reviewer for the detailed comments to improve the paper.
 

Q1: Detailed analysis of why the inference speed is slightly lower than MIRETR.

A1: In the main paper, we have measured the total inference time of our method (0.540s) and MIRETR (0.400s) per scene on the ROBI dataset. Since our method is a two-stage method, we supplement the detailed inference time of each stage in the following table:

It can be observed that the first stage (Focusing Model) tasks up a few time while the second stage (Matching Model) takes up a larger proportion of the time. Compared with the inference speed 0.400s of MIRETR, our inference speed 0.540s is slightly lower than MIRETR.
 

Q2: Details of training and testing strategies.

A2: During training, we use the ground truth center as supervision to train the 3D multi-object focusing module and use the point cloud around the ground truth center as the training data to train the matching network. During testing, we use the center predicted by the 3D multi-object focusing module and its surrounding point cloud as the input to the 3D dual-masking instance matching module to regress the final pose.
 

Q3: Possible reasons behind the phenomenon of MR not SOTA on the Scan2CAD.

A3: In Table 1 of the main paper, it can be seen that in the Scan2CAD dataset, our method improves the mean precision (MP) by about 3% while reducing the mean recall (MR) by about 1%. For Scan2CAD dataset, it may be due to some objects in the scene being too close to each other and relatively small in scale compared to the scene, which cause the multiple objects to be detected as a single object, leading to a decrease in MR. In order to better evaluate MP and MR, we also adopt the mean F1 score (MF), which is the harmonic mean of both MP and MR. In Table 1 of the main paper, it can be observed that our method improve MF by about 1.3%, which further demonstrate the effectiveness of our method.
 

Q4: Analyze the reasons why current methods cannot achieve good results on the ROBI dataset.

A4: Due to the fact that the ROBI dataset is generated from monocular images, occlusions and other factors can cause the instance point clouds to be quite incomplete, making it more challenging. This is the main reason that the performance of all methods is not ideal. Nonetheless, our method significantly outperforms the previous SOTA MIRETR by about 7% in terms of MR, MP, and MF.
 

Q5: Potential plan to solve the problem of inference time.

A5: In the future, we plan to use multiple GPUs for parallel computation to address this issue. We believe that the time taken to process a single matching proposal will be the upper limit of our optimization.
 

Q6: Discuss the computational complexity.

A6: We provide the inference time and the number of parameters in the ROBI dataset in the following table:

It can be our running time is slightly higher than MIRETR. Although the number of parameters is about 2$\times$ of MIRETR, the performance has been greatly improved by about 8% in terms of MR, MP, and MF on the ROBI dataset.

The rebuttal from the authors addressed most of my questions. Since the findings in this paper are interesting, the interpretations are rationale, and the experimental results are convincing to me, I would like to keep the score to "Weak Accept".