Mining and Transferring Feature-Geometry Coherence for Unsupervised Point Cloud Registration

We first would like to thank the reviewer for giving us valuable comments.

Q1: Robustness of Initial Teacher Model: the strategies used for teacher initialization and its robustness in adverse conditions or with suboptimal initialization.

A: The initialization strategy is detailed in Sec 3.2. We first split an input point cloud into two partially overlapping parts, apply a random rigid transformation to one part, and use it as the ground truth. To mimic real-world LiDAR scans, we employ a periodic sampling technique. We also apply commonly used data augmentations, such as random scaling and noise. Visualizations of the generated pairs are provided in Sec. A.5 (L594-600).

The performance of INTEGER indeed relies on the initial teacher. However, INTEGER remains robust even with suboptimal initialization. Our ablation study (Sec 4.3) shows that INTEGER maintains high-quality pseudo-labels, even with EYOC-style initialization (achieving 71.9% IR compared to EYOC's 53.2% IR). This superior robustness may be attributed to the proposed FCEM module, which dynamically adapts the teacher, enhancing it before mining pseudo-labels.

To further demonstrate the robustness, we have compared (1) a Fully-trained teacher, (2) a Under-trained teacher, and (3) a Randomly-initialized teacher. We measured the Inlier Ratio of the teacher in the first epoch to assess the quality of pseudo-labels:

We observed only a minor performance drop when the teacher wasn't fully trained, and the randomly-initialized teacher performed only slightly worse than the well-trained EYOC. In this very extreme case, FCEM's adaptation process ensures that the teacher still performs surprisingly well, highlighting both the superiority of the FCEM design and the robustness of our method. We would also like to highlight that after training for several epochs, the teacher's IR improves significantly, with minimal negative impact on the student's convergence.

Indeed, more complicated registration networks, such as Predator, slightly struggle with current initialization strategies. During training, they show suboptimal performance at the beginning of teacher-student training but eventually catch up after more training epochs, highlighting the superior effectiveness of the proposed FCEM module. We speculate that the complicated networks may suffer from overfitting the synthetic data. In further work, we will devise a better strategy to address this issue.

Regarding adverse conditions, we plan to investigate common adverse conditions in autonomous driving scenarios, such as extreme weather, in our future work. This is a critical aspect of real-world applications that requires thorough investigation.

Q2: Broader Comparisons: Additional SOTA unsupervised methods that employ fundamentally different approaches.

A: Our INTEGER focuses on outdoor scenes; only a few unsupervised methods, namely EYOC and RIENet, are applicable in this context. We have included both in our comparison (see Sec 4.1, Table 1). Among these unsupervised methods, RIENet takes a fundamentally different approach. Our INTEGER not only outperforms EYOC, but also suppresses RIENet by a large margin.

Q3: Insights into computational efficiency and scalability of INTEGER compared to both supervised and unsupervised counterparts.

A: As mentioned in Limitations (Sec.A.1), our method is slightly slower to obtain pseudo-labels than EYOC because of the proposed iterative method used in the FGC-Branch of the FCEM module. However, with only 33% more time cost on KITTI, our INTEGER has much more accurate pseudo-labels with 28.1% higher IR (81.3% v.s. EYOC's 53.2%), and considerable improvements on RR% of very distant pairs. (54.2% v.s. EYOC's 52.3%). We show additional results for supervised baselines on the nuScenes dataset and generalizability tests in part 2 in the general response, which shows that our method has better scalability than SOTA supervised and unsupervised methods. Please refer to it for details.

Q4: Scalability with increasingly large datasets or more complex environments and the discussion on computational resources and runtime; Generalizability Across Different Datasets.

A: Our method targets autonomous driving scenarios, so we tested it on KITTI and nuScenes, standard datasets in this field. These datasets vary in density, occlusion, and noise levels, enabling us to evaluate INTEGER's scalability and robustness.

To further demonstrate the scalability of INTEGER, we have also evaluated the performance of INTEGER on ETH, an outdoor dataset with rural and forest scenes, which are primarily unstructured and complex, differing from the urban environments of KITTI and nuScenes datasets. Moreover, we also conducted additional experiments in indoor scenes as requested by other reviewers. Please refer to part 2 and 3 of the general response for results. These results also show the superior generalizability of INTEGER.

Q5: Robustness against different types of noise or outliers.

A: In outdoor scenes, registration methods are often challenged by extremely low overlap and density variation, which introduces noise in correspondences, especially for distant pairs. Our method demonstrates superior robustness against these challenges, as shown in Sec 4.1, Table 1.

Q6: Discussion on Failure Cases and Limitations.

A: The robustness of the proposed INTEGER against different types of noise and outliers, including the noise introduced by extreme weather, requires further investigation. These situations are particularly relevant for autonomous driving. We will discuss these limitations and future work in the conclusion.

Dear Reviewer 2VKD,

We sincerely thank you for your precious time and efforts in reviewing our paper. We greatly appreciate your insightful and detailed feedback, and we have carefully addressed the concerns in the rebuttal. Please let us know if any aspects remain unclear, and we are happy to provide further clarification.

We are looking forward to your reply.

Best regards,

The authors of submission 1909

The author's answer here is convincing and solves some of my previous questions. Although only a few unsupervised methods are compared, it can be seen that the proposed method has advantages over those methods. If possible, it would be better to compare more unsupervised methods. Although the performance of the proposed method is lacking in some aspects compared to other methods, it also shows that the method proposed by the author has a good effect overall, which is surprising. I keep my original score.

We would like to thank the reviewer for the valuable comments. For concerns about the efficiency and effectiveness of FCEM, please refer to the "6. Efficiency and effectiveness of FCEM" part in the general response.

Q1: Similarity to EYOC.

A: Our method significantly differs from EYOC in several key aspects: (1) Novel Insight: We innovatively introduce a new perspective by demonstrating that inlier matches are closer to positive anchors than negative ones, which summarizes existing inlier and outlier features, respectively. (2) Pseudo-Labeling with Multilevel Information: Our innovative FCEM module dynamically adapts the teacher and integrates low-level geometric and high-level contextual information, whereas EYOC only uses low-level geometric cues. (3) Robust Knowledge Transfer We propose ABCont for effective and robust teacher-student knowledge transfer, which is not present in EYOC. (4) Density Invariance: We introduce MDS to explicitly address density variation in outdoor scenarios, a concern overlooked by EYOC. Experiments show that our method outperforms EYOC in two datasets by a considerable margin.

Q2: Clarity on pseudo-labeling mechanisms and the use of Spatial Compatibility Filtering(SCF) in relation to EYOC.

A: We would like to clarify a potential misunderstanding. While SCF in GSA-Branch appears to be similar to EYOC regarding rejecting outliers, our intention completely differs from EYOC: We aim to improve the teacher for the current data batch, whereas EYOC uses it for pseudo-label extraction for student. Moreover, experiments(Sec4.2, L272-285) show that SCF can be implemented using other pose estimators rather than SC2-PCR estimators used by EYOC.

Moreover, we do not solely rely on SCF to incorporate both low- and high-level information; this is achieved through the entire FCEM, particularly the FGC-Branch(Sec 3.3, L186-216). We iteratively identify latent inliers using feature-space anchors and exclude outliers via SCF. The contributions are novel and not found in EYOC. In our manuscript(L174-175), we cited EYOC to credit their work. We will add detailed discussions on the differences to EYOC in the revised version.

Q3: Impact of distances [d1, d2] on registration.

A: Our evaluation follows EYOC's existing protocol, focusing on pairs within distances [d1,d2], consistent with our method's design for autonomous driving data. The differences between frames, indicated by the distance between them, are crucial for demonstrating robustness and generalizability in real-world applications like collaborative perception. Increased distances mainly present two challenges:(1) Overlap Ratio: Increasing distances reduce overlap ratios, complicating the search for reliable correspondences. (2) Density Variation: Increasing distances often lead to significant point density variations, challenging feature matching without density-invariant features.

Q4: Dataset Differences and Overlap Ratios.

A: Fig.2 in the PDF attachment of the general response illustrates the overlap ratios w.r.t distances between frames in KITTI and nuScenes datasets, indicating increasing distances reduce overlap ratios. Furthermore, KITTI and nuScenes differ in LiDAR resolution(64 LiDAR beams in KITTI and 32 LiDAR beams in nuScenes), resulting in denser point cloud data of the KITTI dataset. The sparsity of nuScenes point cloud data makes it more challenging for registration, especially for feature extraction.

Q5: Performance on Various Datasets.

Q9: Generalization to Indoor Environments.

A: We address Q5 and Q9 together here. We conduct additional experiments in indoor(3DMatch/3DLoMatch dataset) and non-urban environments(ETH dataset). Please refer to parts 2 and 3 of the general response for details.

Q6: Lack of Training Time Details.

A: For most experimental results with FCGF, the full training procedure takes approximately 76 hours. For the additional results with the Predator presented in the Appendix, the training procedure takes about 140 hours. The inference efficiency is the same as that of the registration network(0.16s for FCGF and 0.30s for Predator per pair) because our method does not introduce any test-time modules. Please refer to part 5 of the general response for details.

Q7: Direct Comparison with EYOC if it also uses synthetic pretraining.

A: EYOC doesn't use synthetic pretraining. Instead, it assumes minimal transformation between two consecutive LiDAR frames, using identity transformation as the ground truth for the teacher's pretraining. In contrast, our method employs synthetic pretraining to initialize the teacher, as detailed in Sec3.2(L149-156). Table 4 of the general response shows that EYOC's assumption does not hold in real-world data: it may introduce significant noise, leading to a suboptimal initial teacher of EYOC.

For direct comparison, we evaluate both methods using EYOC's Approximation approaches and Synthetic-Pretraining.

Our method consistently outperforms EYOC even when initialized the same as EYOC. While synthetic pretraining benefits EYOC, the improvement is limited.

Q8: Clarification on Predator architecture and its use in the framework.

A: Predator doesn't employ a dual-branch structure. It consists of a single-branch KPConv encoder-decoder structure with an overlap-attention module in between, and we do not divide the architecture into teacher and student *components. Regarding Predator's results in the Appendix, we utilize the entire Predator model described in Predator's paper, pre-training it fully as the teacher, and then supervise the student with the same architecture. We will clarify this in the revised version.

Dear Reviewer SEpR,

We sincerely thank you for your precious time and efforts in reviewing our paper. We greatly appreciate your insightful and detailed feedback, and we have carefully addressed the concerns in the rebuttal. Please let us know if any aspects remain unclear, and we are happy to provide further clarification.

We are looking forward to your reply.

Best regards,

The authors of submission 1909

Dear Reviewer SEpR,

We sincerely thank you for your precious time. We are happy to see that our work related to PREDATOR has been understood.

We would like to clarify a potential misunderstanding of the the two parts of our loss, namely the anchor-based auxiliary loss $\mathcal{L} _ {\text{aux}}$ and the registration loss $\mathcal{L} _ {\text{reg}}$. We proposed $\mathcal{L} _ {\text{aux}}$ to increase the feature similarity to positive anchors from the teacher, and decrease the feature similarity to negative anchors from the teacher. It is not Circle Loss, although they may share some similarities in their mathmatical expressions. The detailed definition of these anchors are given in L106-117 of our manuscript. Overall, $\mathcal{L} _ {\text{aux}}$ is designed to regularize the feature space of the student with teacher's extracted features. $\mathcal{L} _ {\text{reg}}$, in PREDATOR's case, involves the Circle Loss. However, it cannot be removed because it is the primary supervision signal to supervise the student with the pseudo-labels from the teacher. In Table 3 of our ablation study, we remove the $\mathcal{L} _ {\text{aux}}$ and find a performance drop. However, with $\mathcal{L} _ {\text{reg}}$ and our more accurate pseudo-labels comapred with existing methods, we are still able to outperfrom exsiting SOTAs (our 83.5% mRR v.s. EYOC's 83.2%), which demonstrates the importance of $\mathcal{L} _ {\text{reg}}$. For the student, indeed, we need to train both encoders and decoders, as well as the overlap attention module in between. As you said, the loss passes through both encoders and decoders, so it can adequatly supervise the learning of all components of the student. During teacher-student training, we prevent gradients from being propagated to the teacher, as is typically done by works involving distillation.

We sincerely appreciate your thorough review. Indeed, it is crucial to understand how gradients are back-propagated and their effects on different modules of the network. We will conduct additional experiments and incorporate your suggestions into the final version.

Best regards,

The authors of submission 1909

Dear Reviewer SEpR,

We sincerely appreciate the time and effort the reviewer has dedicated to evaluating the work. As the reviewer-author discussion phase is coming to the end, we hope that our responses and additional experiments have effectively addressed the concerns raised. We respectfully request that the reviewer re-evaluate our work and kindly reconsider the ratings. Thank you for the continued support and understanding.

Best regards,

The authors of submission 1909

We sincerely thank you for your precious time and efforts. We are happy that we have addressed most of your concerns. The reviewer is still concerned about the loss calculation for both the source and target point clouds. Indeed, as stated in the PREDATOR paper, PREDATOR is a two-stream encoder-decoder network, which produces point-level features for both the source and target point clouds. We provide more details about the loss for the source and target point clouds as follows:

As is elaborated in Sec.3.4(L236-238) in our manuscript, the overall training loss for student is a weighted combination of multiple $\mathcal{L} _ \text{ABCont}$ introduced by ABCont (see Sec. 3.1), which is defined as $\mathcal{L} _ \text{ABCont} = \mathcal{L} _ \text{reg} + \lambda _ \text{aux} \mathcal{L} _ \text{aux}$.

• For the auxiliary loss part $\mathcal{L}_\text{aux}$ proposed by us, as we have stated in L142-143 of the manuscript, we calculate it from the two stream (i.e.,for both source and target point clouds): $\mathcal{L} _ \text{aux} = \frac{1}{2} (\mathcal{L} _ \text{aux}^\mathbf{P} + \mathcal{L} _ \text{aux}^\mathbf{Q})$

• For the registration loss part $\mathcal{L}_\text{reg}$ , as we have stated in L137 of the manuscript, we directly adopt the loss of the registration networks, such as PREDATOR and FCGF used in our experiments. Therefore, for experiments involving PREDATOR, we calculate the registration loss in the same way as in their paper. As stated in their paper, they calculate the loss from the two stream (i.e.,for both source and target point clouds), and average them to form the total registration loss.

In summary, we calculate the loss from the two streams in the experiments involving PREDATOR during teacher-student training. As for synthetic pretratining, we also directly adopt the loss of the registration network, and thus calculate the loss from the two stream (i.e.,for both source and target point clouds) for PREDATOR.

We hope this further explanation would address your questions on the loss calculation. Thank you again for your constructive suggestions. We sincerely appreciate your thorough review, and we will incorporate your suggestions into the final version, including further clarification on loss calculation.

We first would like to thank the reviewer for giving us valuable comments.

Q1: The key insight is more presented as an empirical observation, is there any chance that such can be justified in a more concrete way, even on some toy examples?

A: Thanks for your valuable comments. Fig.1 in our manuscript provides qualitative results concerning our key insight. We also conduct an additional simple quantitative experiment. For a sample pair from KITTI, we report the IR% of pseudo-labels at the first iteration in the FGC-Branch under different thresholds based on the feature-space distance to positive anchors in the following table. Features are normalized before calculating distances.

The table shows that most inliers are close to positive anchors in feature space, and outliers are likely far away from positive anchors. This result conforms to our key insight.

Notably, despite the high IR% of correspondences very close to anchors at the first iteration, there are too few correspondences to supervise the student, and some outliers still exist. This fact calls for our iterative design of FCEM to include potential inliers and update the anchors iteratively.

We will refine the presentation of our key insight by (1) providing detailed quantitative results that cover more pairs from both the KITTI and the nuScenes datasets and (2) adding some toy examples to Fig.1 in the revised version.

Q2: Missing Report on the training cost and inference efficiency in the submission.

A: As an unsupervised registration network, our method's training cost and inference efficiency are highly dependent on the choice of registration network. For most experimental results with FCGF, the full training procedure takes approximately 76 hours. For the additional results with the Predator presented in the Appendix, the training procedure takes about 140 hours. The inference efficiency is the same as that of the registration network(0.16s for FCGF and 0.30s for Predator per pair) because our method does not introduce any test-time modules. We will add these details in the revised version.

Q3: INTEGER applies to cases with significant rigid transformations. For example, object-level cases in robotics.

A: Although our work focuses on outdoor scene-level data rather than object-level data, we are willing to demonstrate the effectiveness of our method in scenarios involving significant rigid transformations. Specifically, we augment the rotation around the specified axis by a randomly-sampled angle between [a1,a2]°.

• Results with Augmented Z-axis Rotation on KITTI: Transformation that has been presented in the training set.

In outdoor settings, the rotation around the Z-axis is the most common transformation. However, significant rotations as large as 180° are unseen in the model during training, which makes them a challenge.

Compared to the supervised FCGF, our method has shown superior performance in terms of robustness against significant rigid transformations. Note that we follow the common practice of previous works to compute RRE and RTE only for pairs that are successfully registered. Therefore, RRE and RTE are not necessarily large even when RR% is very low. The results are as follows:

Ours:

FCGF (supervised) trained on KITTI:

• Results with Augmented X-axis Rotation on KITTI: Transformation that hasn't been presented in the training set.

In outdoor settings, the rotation around the X-axis is almost absent from the dataset. Such transformations are thus considered a challenge for registration models, both in terms of generalization and robustness. In this case, we have observed that our method outperforms the supervised FCGF by a very large margin (+18.9 RR% at most), demonstrating the superior generalizability of our method. The results are as follows:

Ours:

FCGF (supervised) trained on KITTI:

Overall, these results indicate that our method can effectively handle cases with significant rigid transformations, even when such transformations are not adequately presented in the training dataset.

We have acknowledged the importance of handling object-level data, such as ModelNet40. However, we are not able to conduct experiments on object-level datasets due to the lack of point cloud sequences in these datasets. Our method relies on progressive training (see Sec.3) and is designed to train with point cloud sequences provided in outdoor datasets. We will consider this in our future work.

Q4: Insight/estimation on the cost of the pre-training stage.

A: For the KITTI dataset, we train the teacher for 30 epochs in the pretraining stage, which takes roughly the same time as teacher-student training. However, the time cost for a single epoch is longer because there are more point cloud pairs (we generate a synthetic pair for every frame in the training set). We will investigate the necessity and devise more efficient sampling strategies in our future work.

Q5: Minor points: L161, missing space.

A: Thanks for pointing out it. We will correct it in the revised version.

Dear Reviewer FNCk,

We sincerely thank you for your precious time and efforts in reviewing our paper. We greatly appreciate your insightful and detailed feedback, and we have carefully addressed the concerns in the rebuttal. Please let us know if any aspects remain unclear, and we are happy to provide further clarification.

We are looking forward to your reply.

Best regards,

The authors of submission 1909

We first would like to thank the reviewer for giving us valuable comments.

W1: Problems with the figures. For example, Fig. 1 is not very easy to understand. Fig.2 and Fig.3 include too much content and make it hard to focus.

A: Thanks for pointing out the figure issues. A simplified version of Fig.2 is shown in the PDF attachment of the general response. In Fig.1, we visualize the feature space by reducing the feature dimension to 2 with t-SNE. Here, we provide an explanation:

• Contour lines and colors in between denote the feature-space distance from a point to its nearest anchor.
• The "o" and "x" markers denote points forming inlier and outlier correspondence, respectively.
• From Fig.1, we can observe that almost all points close to positive anchors in the feature space belong to inlier correspondences.. The observation conforms to our key insight.

We will further improve the clarity of these figures in the revised version. Thank you for bringing this to our attention.

Q2: Reasons for fewer baseline methods presented on nuScenes compared to those on KITTI, as well as the generalization test from KITTI to nuScenes.

A: We excluded some supervised baselines (e.g., GeoTransformer, CoFiNet, and Predator) in experimental results of nuScenes and generalization tests for two main reasons:

• We aimed to align our experimental setup with previous research, such as EYOC, which similarly omits these baselines.
• The omitted baselines are included in KITTI's results to demonstrate that state-of-the-art supervised methods fail when the overlap ratio is very low and density variation is huge. Since KITTI and nuScenes are both autonomous driving datasets and share similar challenges, we believe including these baselines may not significantly add to the completeness of our experiments.

We recognize the value of a comprehensive evaluation and have provided additional baselines in part 1 of the general response. Overall, these supervised SOTA methods still fail to perform on par with unsupervised counterparts with distant point cloud pairs. These experimental results will also be added to the revised version of our paper for clarity.

Q3: Reasons indoor scenes are excluded? Additional experiments to further demonstrate the superiority of the proposed method?

A: We chose to focus on outdoor scenes for two reasons: (1) A rich set of unsupervised indoor registration methods have been proposed, and their performance is already on par with supervised indoor methods. Still, they fail to tackle outdoor challenges due to their reliance on RGB-D data, high overlap assumptions, and LiDAR data's sparsity in outdoor scenes. We aimed to extend the success of unsupervised methods to outdoor scenes, explicitly addressing the unique challenges of these scenarios, such as the varying point density of the same object in different scans, as often encountered in autonomous driving. Given the very distinct applications and sensor characteristics (RGB-D cameras v.s. LiDARs), which may hinder a "unified method" from excelling in both indoor and outdoor environments, we only present the results on outdoor datasets in our paper. (2) Our method relies on progressive training (see Sec.3) and is designed to train with point cloud sequences that are commonly provided in outdoor autonomous driving datasets.

We acknowledge the significance of indoor scenes and have conducted additional experiments to explore our method's potential in these environments. Specifically, we tested generalization from the outdoor KITTI dataset to the indoor 3DMatch/3DLoMatch dataset, demonstrating that our framework, despite being unsupervised, outperforms the supervised FCGF by a significant margin (26.4% RR vs. FCGF's 19.7% RR). Please refer to part 2 of the general response for details.

Dear Reviewer Bwh6,

We sincerely thank you for your precious time and efforts in reviewing our paper. We greatly appreciate your insightful and detailed feedback, and we have carefully addressed the concerns in the rebuttal. Please let us know if any aspects remain unclear, and we are happy to provide further clarification.

We are looking forward to your reply.

Best regards,

The authors of submission 1909

Thanks for your time and effort to read our response. We are happy to be able to address your concerns, and we are more than grateful for your accept recommendation.