FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators

We appreciate the reviewer's insightful comments and address all concerns below.

Q1: The motivation to use diffusion networks rather than other vision foundation backbones (VFB).

A1: Existing VFBs are designed to process 2D RGB images and are not directly applicable to 3D point cloud data. Treating depth maps as RGB images and feeding them into these VFBs [1,2,3] for feature extraction and feature matching leads to poor performances as shown in Table 1 below. We have added this experiment in Sec.A.4.1 of the revised supplementary material.

Table 1: Results on 3DMatch utilizing features from other VFBs

[1] Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193, 2023.

[2] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PMLR, 2021: 8748-8763.

[3] Zhang J, Herrmann C, Hur J, et al. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence[J]. arXiv preprint arXiv:2305.15347, 2023.

Q2: ''FreeReg does not require task-specific training and finetuning and exhibits strong generalization ability to both indoor and outdoor scenes'' is not solid, as the feature extractors are already trained on some datasets.

A2: We use "task-specific" to denote the training on the image-to-point cloud registration task. We agree that our pre-trained models are trained on other tasks. We make this clearer in the revised version by using "FreeReg does not need any training or fine-tuning on I2P registration task."

Q3.1: FreeReg seems not solid outperform the counterpart 2D3D-Matr.

A3.1: FreeReg already achieves 0.9% higher RR than 2D3D-Matr using only pre-trained models. Note 2D3D-Matr is fully supervised on the image-to-point cloud registration task and is a concurrent work to our submission. In Sec.A.5.1 of the revised supplementary material, we show that we fine-tune ControlNet on the image generation task (rather than the I2P registration task) for 1 hour and the resulted model outperforms 2D3D-Matr more significantly as shown in Table 2 below.

Table 2: Comparison of 2D3D-Matr and FreeReg(-Ft).

Q3.2: Comparision with additional baselines including CorrI2P, EP2P-Loc, C2FI2P, E2E-I2P, and CoFiI2P.

A3.2: Many thanks for your recommendation. EP2P-Loc (ICCV 2023), C2FI2P (arXiv), E2E-I2P (arXiv), and CoFiI2P (arXiv) are concurrent works to FreeReg and have not been open-sourced. We have added discussions of these concurrent works in Sec.2 of the revised main paper.

CorrI2P [4] is the only open-sourced work and we tried to compare FreeReg with it. The official code of CorrI2P is released but no pretrained model is available. We thus train it on the provided training set. The trained model achieved an average rotation error of 7 degrees on their provided test dataset. However, the model almost fails on our KITTI-DC test data because CorrI2P relies on the intensity information of the point cloud as input, which is not available in our test data. We add such a discussion in Sec.A.2 of the revised supplementary material.

[4] Ren S, Zeng Y, Hou J, et al. Corri2p: Deep image-to-point cloud registration via dense correspondence[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 33(3): 1198-1208.

Q4: FreeReg performances under $w$=0 and $w$=1 in Table 4.

A4: In our submission, FreeReg-D or FreeReg-G means $w=1$ or $w=0$ for feature matching respectively, for which we have reported their performances using PnP or Kabsch in Table 1. The performances with $w=1$ and $w=0$ using Kabsch are reported in Table 3 below and have been added to Table 4 of the revised main paper.

Table 3: FreeReg performances with w = 0 or 1.

We appreciate the reviewer's insightful comments and our responses are listed below.

Q1: Only comparable performances with the concurrent work 2D3D-Matr.

A1: In Sec. A.5.1 of the revised supplementary material, we fine-tune ControlNet on the image generation task (rather than the difficult I2P registration task) for only 1 hour and the resulted model outperforms 2D3D-Matr more significantly as shown in Table 1 below.

Table 1: Comparison of 2D3D-Matr and FreeReg(-Ft).

Q2: How to accelerate FreeReg?

A2: Many thanks for your inspiring comments. There are two possible ways to accelerate FreeReg.

1. DDIM sampling speed-up. The most time-consuming part of FreeReg registration is the forward sampling process of ControlNet and we can enlarge the sampling step interval for fewer sampling iterations for acceleration. As shown in Table 2 below, with only 5 sampling iterations, FreeReg achieves a registration time reduction of $\sim 50\%$, with only a $1.4\%$ decrease in RR. This might be further optimized with the emergence of diffusion sampling speed-up strategies[2].

   Table 2: FreeReg acceleration with fewer DDIM [1] sampling iterations.

   DDIM-Iters (#)	FMR (%)	IR (%)	IN (#)	RR (%)	Time (s)
   5	93.6	47.8	72.1	62.4	4.7
   10	95.0	46.6	83.0	63.5	6.4
   15	94.5	47.1	82.4	63.0	8.1
   20 (FreeReg)	94.6	47.0	82.8	63.8	9.3

2. Knowledge distillation. We can use FreeReg as a teacher network to guide the feature estimation of a lightweight student image/point cloud feature extractor. An analysis has been added in Sec.A.5.2 of the revised supplementary material.

[1] Song J, Meng C, Ermon S. Denoising diffusion implicit models[J]. arXiv preprint arXiv:2010.02502, 2020.

[2] Luo S, Tan Y, Huang L, et al. Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference[J]. arXiv preprint arXiv:2310.04378, 2023.

Q3.1: What's the initial rotation error before registration?

A3.1: The average initial rotation error between unposed images and posed point clouds is 115.4 degrees and 68.3 degrees on 3Dmatch and KITTI-DC, respectively.

Q3.2: Is the rotation error of 10 or 20 degrees a good threshold for recall?

A3.2: We agree that 10/20 degrees is a large rotation threshold and we select it based on the following considerations:

1. Why 10/20 degrees: The rotation threshold is typically set to 15/5 degree for the indoor/outdoor point cloud registration task[3]. As a cross-modality task, Image-to-point cloud registration is more difficult. Thus, we use a looser threshold at 20 degrees for indoor I2P registration and a 10-degree threshold for outdoor registration, which is the same as DeepI2P.
2. Performance under different thresholds: We also report the performance of FreeReg under different thresholds in Fig. 1 and Table 5 in the revised supplementary material. On more than 50% data, our method achieves a rotation error less than 10/5 degrees for indoor/outdoor data.

[3] Choy C, Park J, Koltun V. Fully convolutional geometric features[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 8958-8966.

Q3.3: Is a rotation error of 10 or 20 degrees useful in practice?

A3.3: Yes. 10/20 degree rotation accuracy can support applications like positioning and navigation in a mall, which do not require very high accuracy. We believe improving FreeReg's accuracy will enlarge the application scope of I2P registration.

Q4: Experimental results of straightforward/direct implementations in the 4th and 5th paragraph of Introduction.

A4: In the fourth paragraph, we analyzed that naively using ControlNet to directly transform input point clouds into the image modality failed to reliably match with input images. This corresponds to ControlNet+SuperGlue(CN+SG) in Sec.4. In the fifth paragraph, we discussed using Zoe-Depth to transform input images into the point cloud and matching between the point clouds. This corresponds to Zoe-Depth+FCGF, i.e., FreeReg-G in Sec.4. The experimental results indicate that both baselines perform worse than FreeReg. We have improved the introduction about these two baselines and recall the above discussions in Section 4.1 to make it clearer.

We appreciate the reviewer's encouraging comments and make responses below.

Q1: Use FreeReg features in other cross-modality problems.

A1: Many thanks for your insightful comments. In Sec.A.5.3 of the revised supplementary material, we show that the FreeReg features have strong semantic consistency, which suggests the potential ability of these features for other semantic tasks.

Q2: How to accelerate FreeReg?

A2: There are two possible ways to accelerate FreeReg.

1. DDIM sampling speed-up. The most time-consuming part of FreeReg registration is the forward sampling process of ControlNet and we can enlarge the sampling step interval for fewer sampling iterations for acceleration. As shown in Table 1 below, with only 5 sampling iterations, FreeReg achieves a registration time reduction of $\sim 50\%$, with only a $1.4\%$ decrease in RR. This might be further optimized with the emergence of diffusion sampling speed-up strategies[2].

   Table 1: FreeReg acceleration with fewer DDIM [1] sampling iterations.

   DDIM-Iters (#)	FMR (%)	IR (%)	IN (#)	RR (%)	Time (s)
   5	93.6	47.8	72.1	62.4	4.7
   10	95.0	46.6	83.0	63.5	6.4
   15	94.5	47.1	82.4	63.0	8.1
   20 (FreeReg)	94.6	47.0	82.8	63.8	9.3

2. Knowledge distillation. We can use FreeReg as a teacher network to guide the feature estimation of a lightweight student image/point cloud feature extractor. An analysis have been added in Sec.A.5.2 of the revised supplementary material.

[1] Song J, Meng C, Ermon S. Denoising diffusion implicit models[J]. arXiv preprint arXiv:2010.02502, 2020.

[2] Luo S, Tan Y, Huang L, et al. Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference[J]. arXiv preprint arXiv:2310.04378, 2023.

Q3: Can FreeReg work with only diffusion or geometric features by setting $w$ to 1 or 0.

A3: In our submission, FreeReg-D or FreeReg-G means $w=1$ or $w=0$ for feature matching respectively, for which we have reported their performances using PnP or Kabsch in Table 1. The performances with $w=1$ and $w=0$ using Kabsch are reported in Table 2 below and have been added to Table 4 of the revised main paper.

Table 2: FreeReg performances with w = 0 or 1.