A Consistency-Aware Spot-Guided Transformer for Versatile and Hierarchical Point Cloud Registration

Thank you for your constructive review and valuable suggestions. Below, we offer detailed responses to each of your comments and questions. If there are any points where our answers don't fully address your concerns, please let us know, and we will respond as quickly as possible.

• Weakness 1: In line 123, “third level” and “fourth level” need to be explained more or illustrated in the Figure 1.

For feature extraction based on KPConv, we need to down-sample the raw point clouds with a series of voxel sizes $v, 2v, 4v, 8v$ before each encoder layer, resulting in the first level, the second level, the third level and the fourth level points. As the feature maps in the decoder of these points are with the size of $1/k$ of that of the input point cloud $X$ where $k=1,2,4,8$, respectively, we will introduce the notation $X^{1/k}$ and $F_X^{1/k}$ to symbolize the points and their features to make our maniscript more easy to understand. We have carefully revised our writing and figures of Section.3 Method and released this version as a general response, and it would be appreciated if you can read our revised overview part.

• Weakness 2: Please clear state the proposed contribution or difference with respect to the existing work.

Existing work widely adopts self-attention and cross-attention for coarse matching, however, CoFiNet and GeoTransformer adopt global cross-attention for feature aggregation, which inevitably attends to similar yet irrelevant areas, resulting in misleading feature aggregation and inconsistent correspondences. DiffusionPCR follows an iterative matching scheme that explicitly selects overlapped regions to attend to, leading to much longer inference time. Additionally, these methods focus on matching among very coarse nodes without considering geometric consistency, which is not tight enough for fine matching. Instead, we focus on feature aggregation among semi-dense features leveraging both local and global geometric consistency to tackle the sparsity and looseness of coarse matching. To be specific, our consistency-aware self-attention only attends to salient nodes sampled from a global compatibility graph, while our spot-guided cross-attention only attends to nodes selected based on local consistency, i.e., the spot.

As for geometric consistency, existing work only uses it to search consistent correspondences (outlier rejection) from a given correspondence set for robust pose estimation, such as PointDSC, spectral matching, etc. However, our method leverages geometric consistency in feature aggregation during coarse matching, instead of outlier rejection. For fine matching, we adopt the geometric consistency for inlier prediction, which is similar to PointDSC but without time-consuming hypothesis-and-verification pipelines. Therefore, our attention-based modules and the way using geometric consistency are very different from existing work.

• Weakness 3: I would recommend the generalizability of the proposed method can be discussed.

We present our results of generalizability and discuss the potential in realistic applications in part 3 of author rebuttal. Please refer to it, thank you for your suggestion!

Although our method trained on 3DMatch fails to deploy on ETH due to the out of memory error, we evaluate the generalizability of different methods when trained on KITTI (outdoor) and tested on ETH (outdoor). Note that these datasets use Velodyne-64 3D LiDAR and Hokuyo 2D LiDAR, respectively, leading to very different appearance of point clouds, hence we believe it is solid to show the generalizability of different methods. Our method achieves satisfying accuracy and robustness, showcasing better generalizability than GeoTransformer, due to the help of consistency. But it falls behind SpinNet and BUFFER using RANSAC mainly owing to the point-wise FPN backbone and lightweight yet learnable fine matching. Furthermore, we conduct an unsupervised domain adaptation experiment, indicating that our model easily adapts to an unseen domain and achieves robust and accurate performance after a short period of unsupervised tuning (only 20min for a epoch!). For applications, we believe generalizing or quickly adapting from an outdoor LiDAR dataset to another is a more realistic setting than generalizing from an RGBD camera dataset 3DMatch to a LiDAR dataset ETH as many papers.

• Q: Instead of introducing the geometric consistency to the feature matching process, why not making it as a separate and following process?

A: Our overall design using a complex coarse matching module and a lightweight fine matching module is motivated by the efficiency, accuracy and scalability requirements from real-time applications such as LiDAR odometry and SLAM. In a LiDAR odometry system, it is a consensus that the real-time frame-to-map registration is essential for accuracy, instead of frame-to-frame registration as we do in this task. However, all of the existing coarse-to-fine methods fail to achieve real-time performance. We believe the key is only using a lightweight fine matching process, because coarse matching is not necessary in odometry with a small pose deviation. Existing hypothesis-and-verification pipelines such as RANSAC and PointDSC are robust but time-consuming. Hence, we propose a lightweight fine matching module allowing independent deployment without coarse matching to meet these requirements. Nevertheless, coarse matching is essential for place recognition and global re-localization in SLAM without real-time requirement, hence we introduce the consistency to it to make the whole system robust for large-scale pose deviation.

Thank you for your constructive review and valuable suggestions. Below, we offer detailed responses to each of your comments and questions. If there are any points where our answers don't fully address your concerns, please let us know, and we will respond as quickly as possible.

• Weakness 1-3 about writing.

We have carefully revised our writing and figures of Method Section in the general comments, and it would be appreciated if you can read it. Specifically, we adjust our Figure 1 to clearly label the coarse and fine matching modules and include more details such as the prediction of overlap scores. We explicitly describe our pipeline in both Figure 1 and an extra Architecture subsection in the same order. I believe our revised version can address most of your concerns.

• Weakness 4 about performance in low overlapping scenes.

We update our results on 3DLoMatch and discuss the performance of our method in 3DLoMatch in part 1 of author rebuttal. Please refer to it, thank you!

• Weakness 5 about our claim of potential in large-scale applications.

The FLOPs of our method is significantly lower than GeoTransformer and RoITr, indicating its efficiency. Although FLOPs of other methods are lower, their runtime is much more. We believe the runtime on RTX3080Ti is more direct than FLOPs only focusing on deep models.

By the way, "large-scale applications" in our manuscript may be misleading, thank you for pointing it out! Instead of large-scale datasets, it mainly stands for SLAM in our context, which requires frame-to-map registration for accuracy, instead of frame-to-frame registration as we do in this task. Compared with our task, the frame-to-map registration is large-scale. Here we would like to clarify why our design has greater potential in this application.

At present, existing coarse-to-fine methods fail to achieve real-time performance in SLAM. Moreover, their fine matching tightly couples with coarse matching, which requires node-based partitioning and really time-consuming optimal transport. Therefore, they are not scalable to SLAM. We believe the key is only using a lightweight fine matching process, because coarse matching is not necessary in odometry with a small pose deviation. Hence, we design a sparse-to-dense fine matching pipeline allowing independent deployment without coarse matching to meet these requirements, which can efficiently establish virtual correspondences for any point and their neighbors in the other point cloud. Moreover, we validate its feasibility using a learnable inlier classifier instead of existing time-consuming pose estimators (RANSAC, MAC, etc). Nevertheless, coarse matching is essential for place recognition and global re-localization in SLAM without real-time requirement, hence we introduce the consistency to it to make the whole system robust for large-scale pose deviation.

• Weakness 6: lacks visualizations for the spot-guided attention.

We add visualizations of vanilla global cross-attention and our spot-guided cross-attention of nodes from both salient areas and non-salient areas, as is shown in Figure 2 in the supplementary pdf, which demonstrates its effects to select instructive areas to attend to according to local consistency, while vanilla cross-attention may be attracted by many irrelavant areas.

• Weakness 7: The manuscript lacks critical references. Comparisons with state-of-the-art methods cited in references [2] and [3] are also necessary.

Thank you for pointing it out. We add the mentioned reference in our revised version in general comments. Comparisons with these SOTA registration methods are shown in Table 1 in author response, which further validate the efficacy of our method. For fairness, all of these methods use the FCGF descriptor rather than coarse-to-fine methods. Rather than focusing on feature extraction or matching as the baselines included in our manuscript, these methods focus on how to search a consensus set for robust pose estimation, using a pipeline more complex than our sparse-to-dense fine matching and LGR in GeoTransformer. We believe this is why existing papers only compare these registration methods with each other using the same descriptors or correspondences.

• Q1: The paper does not specify if the loss supervises the overlap scores, it does not require it?

A: Our coarse matching loss $\mathcal{L}_c$ in Equation (12), Line 233 simultaneously supervises the coarse matching and the overlap scores, since the final matching score is the product of feature similarities and overlap scores, and we also include the negative log-likelihood terms of overlap scores whoose labels are "0". Besides, we add an ablation study by ablating this overlap head as is shown in Table 2 in the supplementary pdf, which demonstrates its importance.

• Q2: The term "partial tokens" needs clarification. Details on computing the coarse matching for spot-guided attention are also needed.

A: "Partial tokens" means instead of attending to all nodes, our attention only select some of them to attend to. To be specific, the consistency-aware self-attention only attends to salient nodes sampled from a global compatibility graph, while the spot-guided cross-attention only attends to nodes selected based on local consistency, i.e., the spot. We have carefully revised our writing and figures of Method Section in the general comments and supplementary pdf, which clearly provides the details of how to match the semi-dense nodes and evaluate their consistency in Subsection Architecture.

• Suggestion

Thank you for pointing it out! We will ensure consistency in font sizes across Tables 3 and 4. And we also replace the terminology "patches" in line 123 or any similar cases with "nodes", which is more appropriate.

I appreciate the authors' clarifications. My concerns have been addressed.

Thank you for your constructive review and valuable suggestions. Below, we offer detailed responses to each of your comments and questions. If there are any points where our answers don't fully address your concerns, please let us know, and we will respond as quickly as possible.

• Weakness 1.3.4 about writing.

Thank you for your valuable suggestions. We have carefully revised our writing and figures of Method Section and released them in the supplementary pdf of author rebuttal and general comments, respectively, and it would be appreciated if you can read this version.

• • Specifically, we adjust our Figure 1 to clearly label the coarse and fine matching modules with as well as submodules, and include more details to achieve greater clarity and precision.
• • The inputs and outputs of each module are explicitly stated in our revised version and depicted in the revised Figure 1. For instance, the inputs of the sparse matching are coarse correspondences $\hat{C}=\\{(x_j^S, y_j^S): x_j^S\in X^{1/4}, y_j^S\in Y^{1/4}\\}$ and the semi-dense nodes $X^{1/4},Y^{1/4}$ with features $F^{1/4}$ from FPN, while the outputs of the single-head attention and the compatibility graph embedding (submodules) are virtual correspondences and their confidence weights, respectivel, which are also the outputs of sparse matching. Then these virtual correspondences with weights along with dense features $F^{1/2}$ are the inputs of dense matching, which estimates the pose for coarse alignment and further local matching, then it outputs the final pose estimate.
• • Finally, we have added an extra Architecture subsection to introduce the pipeline and kept the writing in the same same order as the figure, for instance, the consistency-aware self-attention is before the spot-guided cross-attention. The architecture of our coarse matching module is designed as a sequence of blocks for attention-based multi-scale feature aggregation. For each block with both semi-dense features $F^{1/4}$ and coarse features $F^{1/8}$ as inputs, we first feed $F^{1/8}$ into a self-attention module and a cross-attention module. Then $F^{1/4}$ and $F^{1/8}$ are fused into each other. Finally, semi-dense features are fed into a consistency-aware self-attention module and later a spot-guided cross-attention module at the end of each block. I believe our revised version can address most of your concerns about writing and presentation.

I believe our revised version can address most of your concerns about writing and presentation.

• Weakness 2 about the description of spot.

We have carefully revised this part and you can check it in the supplementary pdf (Figure 2) of author rebuttal and general comments. Here I would like to briefly describe the definition and effect of spot. Before spot-guided cross-attention, we compute a coarse correspondence set by matching, denoted as $C^{(l)}$. For each node $x_i^S$ such that $(x_i^S,y_i^S)\in C^{(l)}$, we select a subset $\mathcal{N}_s(x_i^S)$ as seeds from its neighborhood $\mathcal{N}(x_i^S)$ and construct a region of interest for it as $\mathcal{S}(x_i^S)=\bigcup _{x_k^S\in \mathcal{N}_s(x_i^S)} \mathcal{N}(y_k^S)$, namely its spot. $\mathcal{N}_s(x_i^S)$ selects $x_i^S$ and only its neighbors with reliable correspondences, and we propose a consistency-aware matching confidence criterion to rank the neighbors. Effect: As shown in Figure 2 of the supplementary pdf, global cross-attention tends to aggregate features from wrong regions under the disturbance of many similar yet irrelevant regions, leading to false correspondences. Our formulation of spot is inspired by local consistency that the correspondences of adjacent 3D points remain close to each other. Hence, the spots defined above are likely to cover the true correspondences of query nodes, providing guidance for feature aggregation without interfering with irrelevant areas.

• Weakness 5: In Formula 4, is the pre-defined threshold $\sigma_c$ set differently for datasets of varying sizes?

Yes, the threshold $\sigma_c$ for evaluating the consistency should be set differently for datasets of varying sizes. For simplification, we set it as the 6 times of the initial voxel size for down-sampling.

• Weakness 6: How generalizable is the method proposed in this paper? If trained on the 3DMatch dataset, how would the results be when tested on outdoor datasets such as ETH?

We present our results of generalizability and discuss the potential in realistic applications in part 3 of author rebuttal. Although our method trained on 3DMatch fails to deploy on ETH due to the out of memory error, we evaluate the generalizability of different methods when trained on KITTI and tested on ETH. Note that these datasets use Velodyne-64 3D LiDAR and Hokuyo 2D LiDAR, respectively, leading to very different appearance of point clouds, hence we believe it is solid to show the generalizability of different methods. Our method showcases better generalizability than GeoTransformer, due to the help of consistency. But it falls behind SpinNet and BUFFER using RANSAC mainly owing to the point-wise FPN backbone and lightweight yet learnable fine matching. Moreover, we conduct an UDA experiment, indicating that our model easily adapts to an unseen domain and achieves robust and accurate performance after a short period of unsupervised tuning (only 20min for a epoch!). For applications, we believe generalizing or quickly adapting from an outdoor LiDAR dataset to another is a more realistic setting than generalizing from an RGBD camera dataset 3DMatch to a LiDAR dataset ETH as many papers.

• Weakness 7: The method designed in this paper seems reasonable and effective. It is suggested that the authors make the code open-source for others to learn from.

Thank you for your suggestion! Our source codes have been submitted as the supplementary material before rebuttal, and we will make it open source once the paper is accepted.

Thank you for your constructive review and valuable suggestions. Below, we offer detailed responses to each of your comments and questions. If there are any points where our answers don't fully address your concerns, please let us know, and we will respond as quickly as possible.

• Weakness 1: Novelty

Our method roughly follows the pioneering coarse-to-fine matching framework CoFiNet (NeurIPS21'), but there are significant improvements besides performance. We believe this is a novel way incorporating both local and global consistency in feature aggregation rather than outlier removal as existing registration methods such as SC2-PCR and PointDSC. And it is also important that our method makes learning-based registration more efficient and scalable for real-time applications such as odometry and SLAM.

Ever since the publishment of CoFiNet (NeurIPS21'), many works roughly follow the pipeline and revise some of the modules, including GeoTransformer(CVPR22'), OIF-Net (NeurIPS22'), PEAL (CVPR23'), etc. Although these works widely adopt self-attention and cross-attention for coarse matching, however, CoFiNet and GeoTransformer adopt global cross-attention for feature aggregation, which inevitably attends to similar yet irrelevant areas, resulting in misleading feature aggregation and inconsistent correspondences. Additionally, these methods focus on matching among very coarse nodes without considering geometric consistency, which is not tight enough for fine matching. Instead, we focus on feature aggregation among semi-dense features leveraging both local and global geometric consistency to tackle the sparsity and looseness of coarse matching. To be specific, our consistency-aware self-attention only attends to salient nodes sampled from a global compatibility graph, while our spot-guided cross-attention only attends to nodes selected based on local consistency. We believe this is a novel way to introduce geometric consistency to feature aggregation rather than outlier rejection such as PointDSC and SC2-PCR.

More importantly, our overall design using a complex coarse matching module and a very lightweight fine matching module is motivated by the efficiency, accuracy and scalability requirements from real-time applications such as LiDAR odometry and SLAM. In a LiDAR odometry system, it is a consensus that the real-time frame-to-map registration is essential for accuracy, instead of frame-to-frame registration as we do in this task. However, all of the existing coarse-to-fine methods fail to achieve real-time performance. Moreover, their fine matching tightly couples with coarse matching, which requires node-based partitioning and really time-consuming optimal transport for patch-to-patch correspondences. Therefore, existing methods are not scalable to SLAM. We believe the key is only using a lightweight fine matching process, because coarse matching is not necessary in odometry with a small pose deviation. Hence, we design a sparse-to-dense fine matching pipeline allowing independent deployment without coarse matching to meet these requirements, which can efficiently establish virtual correspondences for any point and their neighbors in the other point cloud. Moreover, we validate the feasibility of this scheme using a learnable inlier classifier instead of existing time-consuming pose estimators (RANSAC, PointDSC, MAC, etc). Nevertheless, coarse matching is essential for place recognition and global re-localization in SLAM without real-time requirement, hence we introduce the consistency to it to make the whole system robust for large-scale pose deviation.

Finally, extensive experiments validates the accuracy, robustness, and efficiency on both indoor and outdoor scenes. It is noteworthy that our model can easily adapts to an unseen domain and achieve robust and accurate performance after a short period of unsupervised tuning (only 20min for a epoch!), which is detailed in part 3 in author rebuttal. The model sizes of popular methods are also reported as follow:

• Weakness 2: Performance on indoor datasets

We update our results on 3DLoMatch and discuss the performance of our method in low overlapping scenes in part 1 of author rebuttal. Please refer to it, thank you for your comment!

It is noted that on indoor benchmark 3DMatch, our method achieves the highest RR of 95.2%. On 3DLoMatch, our method achieves 75.1% RR, surpassing all descriptors and all non-iterative correspondence-based methods except OIF-Net. As for DiffusionPCR and PEAL showing higher RR than others, they iteratively use a variant of GeoTransformer with overlap priors for multi-step matching, which is extremely time-consuming (10x runtime of ours), and PEAL even uses extra information from 2D images. Notably, our method achieves such superior performance only with the lowest time consumption.

Spot-Guided Cross-Attention. As shown in Figure 2, global cross-attention tends to aggregate features from wrong regions under the disturbance of many similar yet irrelevant regions, leading to false correspondences. Inspired by local consistency that the correspondences of adjacent 3D points remain close to each other, we design the spot-guided cross-attention as depicted in Figure 2. For each node $x_i^S$ such that $(x_i^S,y_i^S)\in C^{(l)}$, we select a subset $\mathcal{N}_s(x_i^S)$ as seeds from its neighborhood $\mathcal{N}(x_i^S)$ and construct a region of interest for it as $\mathcal{S}(x_i^S)=\bigcup _{x_k^S\in \mathcal{N}_s(x_i^S)} \mathcal{N}(y_k^S)$, namely its spot. $\mathcal{N}_s(x_i^S)$ selects $x_i^S$ and only its neighbors with reliable correspondences. We propose a consistency-aware matching confidence criterion to rank the neighbors, which is formulated as the sum of the matching score and the normalized generalized degree in the compatibility graph. This criterion incorporates feature similarity and geometric consistency to properly measure the reliability of correspondences for seed selection. Finally, semi-dense features attend to their spots for feature aggregation according to Eq.3. Under the guarantee of local consistency, the spots are likely to cover the true correspondences, providing guidance for feature aggregation without interfering with irrelevant areas.

Sparse-to-Dense Fine Matching

Given a coarse correspondence set $\hat{C}=\\{(x_j^S, y_j^S): x_j^S\in X^{1/4}, y_j^S\in Y^{1/4}\\}$ selected as mutual nearest neighbors from the final coarse matching scores (Eq.1), we propose a lightweight sparse-to-dense fine matching module for hierarchical pose estimation without optimal transport, maintaining scalability and efficiency. For sparse matching, we first adopt k-nearest neighbor (kNN) search centered on semi-dense nodes $X^{1/4}$ to group patches containing dense points from $X^{1/2}$, then we use an attentive keypoint detector [7] to predict a repeatable keypoint with a descriptor from each patch. Each keypoint of point cloud $x$ is assigned to its nearest node, and each node $y_j^S\in Y^{1/4}$ with a correspondence in $\hat{C}$ groups a patch $\mathcal{P}(y_j^S)$ of keypoints via kNN. Then, a keypoint $x_i$ assigned to $x^S_j$ will correspond to the patch $\mathcal{P}(y_j^S)$, forming a keypoint-to-patch correspondence. Finally, we utilize a single-head attention layer for each keypoint-to-patch correspondence to predict virtual correspondences for keypoints. Denote the descriptor of $x_i$ as $d^X_{i}$, the virtual correspondence $\hat{y}_i$ with feature $\hat{d}_i^Y$ is predicted from keypoints in $\mathcal{P}(y_j^S)$:

E^{(l+1)}=\text{softmax}\left(\frac{1}{\sqrt{D_e}} E^{(l)}W_Q^{(l)}
(E^{(l)}W_K^{(l)})^\mathsf{T} \odot B \right)E^{(l)}W_V^{(l)}, E^{(0)}_i = \text{MLP}([ x_i,d^X_i, \hat{y}_i, \hat{d}^Y_i ]), (10)$$

where $E^{(l)}$ is the correspondence-wise embedding of the $l$-th layer with learnable weights $W_Q^{(l)},W_K^{(l)},W_V^{(l)}$. Finally, the embedding is fed into an MLP to classify if a correspondence is an inlier. The predicted inlier confidences serve as the weights of keypoint correspondence for pose estimation formulated as Eq.1, which can be analytically solved by weighted Kabsch algorithm [8]. It is noteworthy this process is really lightweight and scalable to large-scale registration tasks.

After aligning two point clouds based on sparse matching, we propose to refine the transformation based on dense matching. We still utilize local attention (Eq.9) to predict the correspondences of dense points $X^{1/2}$ from its neighbors in $Y^{1/2}$ within a radius $R_d$, and we simply set the confidence weight of a correspondence with a distance $d$ as $w = [1 - d/R_d]^+$. By solving Eq.1 again with both sparse and dense correspondences, we can achieve more accurate pose estimation efficiently.

• [7] Fan Lu et al. "RSKDD-Net: Random sample-based keypoint detector and descriptor." Advances in Neural Information Processing Systems (NeurIPS), 2020.
• [8] Wolfgang Kabsch et al. "A solution for the best rotation to relate two sets of vectors." Acta Crystallographica447 Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 1976.

Architecture. As both spot-guided cross-attention and consistency-aware self-attention are sparse attention lacking of abundant global context, we propose to enhance the semi-dense features via multi-scale feature fusion with coarse features. Hence, the architecture of our coarse matching module is designed as a sequence of blocks for attention-based multi-scale feature aggregation. For each block with both semi-dense features $F^{1/4}$ and coarse features $F^{1/8}$ as inputs, we first feed $F^{1/8}$ into a self-attention module (Eq.5) and a cross-attention module (Eq.3). Then $F^{1/4}$ and $F^{1/8}$ are fused into each other based on nearest up-sampling and distance-based interpolated down-sampling [4]:

P_{ij}^{(l)} = \underset{k\in \{1,\cdots,M'\}}{\text{softmax}}(S _{kj}^{(l)})_i \underset{k\in \{1,\cdots,N'\}}{\text{softmax}}(S _{ik}^{(l)})_j, S^{(l)}=\hat{F}_X^{(l)} (\hat{F}_Y^{(l)})^{\mathsf{T}}. (7)$$

Then the correspondence of each node can be obtained as the node from another point cloud with the highest matching score, forming a correspondence set $C^{(l)}=\\{(x_i^S,y_i^S): x_i^S\in X^{1/4},y_i^S\in Y^{1/4}\\}$. An insight about the consistency among correspondences is that the distance between two points is invariant after transformation. Hence, geometric compatibility is adopted as a simple yet effective measure of consistency [5,6], which is based on the length difference between pairwise line segments. Given a pre-defined threshold $\sigma_c$, the pair-wise geometric compatibility of $C^{(l)}$ is formulated as: