Generative Point Cloud Registration

Q1: Clarification on rigid transformations.
A1: Thank you for your valuable suggestion. In the revised introduction, we will explicitly state that our work focuses exclusively on the rigid point cloud registration problem to set clear expectations for the readers.

Q2: The discrepancy between the noticeable improvement in mean errors and the lack of significant enhancement in median error in Table 2 requires further explanation.
A2: Thank you for the insightful comment. We clarify that this discrepancy arises because mean and median errors capture different aspects of performance. Specifically, the mean error reflects the overall precision and is sensitive to outliers. The noticeable improvement in mean error stems from a substantial reduction in large-error cases, indicating that our method can effectively handle challenging scenarios. Instead, the median error reflects the model’s performance on the majority of cases, which was already strong and thus shows minimal change. We will clarify this distinction in the revised manuscript to avoid confusion and better highlight the strengths of our approach.

Q3: Comparison with newer state-of-the-art.
A3: We appreciate the reviewer’s valuable suggestion. Following the reviewer’s recommendation, we have conducted an additional comparison with a recent state-of-the-art method, PARE-Net (ECCV 2024), and included the results in the table below. The new results show that our proposed Generative FCGF (SD) consistently outperforms PARE-Net, confirming our superior registration performance. We will include these comparisons in our revised version.

Q4: The concern is that ControlNet may have been trained on easier datasets like ScanNet, which could simplify the generation task. It's important to test the proposed method on more challenging datasets to confirm its generalizability.
A4: Thank you for the thoughtful comment. (i) We would like to clarify that, although the training dataset for the depth-conditioned ControlNet model has not been publicly released, the depth maps used in that model were generated by the MiDaS depth estimation model. By contrast, our experiments utilize real depth maps captured directly by depth sensors. Despite the significant domain gap between the MiDaS-estimated and sensor-captured depth maps, our method still achieves impressive generation quality, highlighting its robustness and generalization capability; (ii) To further validate the generalizability and robustness of our Match-ControlNet in more challenging scenarios, we conducted an additional experiment where we casually captured low-overlap photos of a cluttered, unconstrained indoor environment (i.e., the author's room) using a mobile phone. We then estimated their depth maps using DUSt3R for subsequent Match-ControlNet generation (without any model fine-tuning). The figure (https://anonymous.4open.science/r/rebuttal-688D/wild_vis.pdf) demonstrates that, even under these challenging in-the-wild conditions with a different depth-map source, our method still achieves impressive cross-view consistency and generation quality. These results confirm the practical effectiveness and excellent generalizability of our approach.

Q1: Lack of joint distribution modeling of multi-view images?
A1: We respectfully clarify that our coupled denoising mechanism has implicitly modeled the joint distribution of multi-view images (i.e., cross-view images in our task). Formally, the likelihood over the cross-view image pair can be expressed as $\mathbb{E}[p_\theta({x}^{P}, {x}^Q)]$ (we omit conditional variables for simplicity). By concatenating the cross-view images into a unified variable ${x}^{PQ} = Cat([{x}^P, x^Q])$, we can model the joint distribution of cross-view images via the likelihood over the concatenated representation: $\mathbb{E}[p_\theta({x}^{PQ})]$. As such, we can derive a variational lower bound on the data log-likelihood for optimization: L=E_q [log p_θ(x_0^{PQ} | x_1^{PQ}) - ∑{t>1} D_KL(q(x{t-1}^{PQ} | x_t^{PQ}, x_0^{PQ}) || p_θ(x_{t-1}^{PQ} | x_t^{PQ})) - D_KL(q(x_T^{PQ}| x_0^{PQ}) || p(x_T^{PQ}) )].

Through the reparameterization trick, maximizing this lower bound is equivalent to minimizing our training loss in Eq. 5. This derivation demonstrates that our coupled denoising implicitly models the joint distribution of cross-view images, thereby effectively enforcing cross-view consistency generation. We will clarify this in our revision.

Q2: The proposed method is a workaround and exhibits instability? Absence of explicit attention-based modules like Zero123++?
A2: We respectfully clarify that our method is not a workaround, but an efficient and elegant design that enables effective cross-view information fusion and consistency learning with minimal overhead. Unlike methods like Zero123++, which introduce an additional complex attention modules for fusion, our attention design is native. We employ a novel coupled denoising mechanism that naturally extends the inherent intra-image self-attention to both inter- and intra-image attention, without any architectural modifications (see Eq. 4). We note that this design effectively unlocks the zero-shot consistency generation ability of Stable Diffusion, acquired through extensive large-scale pretraining, thereby enhancing both generation quality and cross-view consistency. As evidenced by extensive experiments, our method consistently delivers stable results across diverse scenarios (see Table 5, 6, and Fig. 5). Moreover, we tested on low-overlap, in-the-wild data (from the author's room) captured by a mobile phone. The results (https://anonymous.4open.science/r/rebuttal-688D/wild_vis.pdf) further confirm its stability in unconstrained conditions (please see A4@P29t for details).

Q3: Real vs. generated images comparison.
A3: As shown in Table 3 (third block) in our manuscript, we had compared ColorPCR using ground-truth images with our generative ColorPCR that utilizes generated images. Surprisingly, our generated images can even achieve superior performance across most metrics. We attribute this improvement to that our generated images effectively mitigate calibration errors and lighting challenges inherent in real-world data, as illustrated in Fig. 6.

Q4: Comparison with image-only methods (e.g., DUSt3R).
A4: We would like to emphasize that our paper targets the point cloud matching rather than the image matching. Therefore, comparisons with image-only methods actually fall outside the scope of our research. Below, we report the score of SOTA image-only method, DUSt3R, on 3DMatch. It shows that, due to the absence of geometric and scale information inherent in point clouds, DUSt3R demonstrates limited precision, highlighting the importance of point cloud features for accurate registration.

Q5: Variance analysis across generated images.
A5: Thank you for your suggestion. We repeated the evaluations on 3DMatch using varying random seeds (123, 1234, 12345) for Match-ControlNet. The mean and variance of Generative FCGF are 93.7 ± 0.6 for Rot@5 and 54.1 ± 0.1 for Trans@5, which still exhibits a significant performance gain over the baseline, validating the robustness of our method.

Q6: More visualization results of point cloud overlap after matching.
A6: We provide more registration visualisation results in figure (https://anonymous.4open.science/r/rebuttal-688D/reg_vis.pdf), qualitatively showing our excellent precision. We will include them into our revised version.

Q1: The approach heavily relies on the quality of the generated 2D images, which could introduce artifacts or inconsistencies in certain scenarios?
A1: The image generation quality is indeed crucial to the overall performance. Notably, our Match-ControlNet successfully unlocks the generalizable zero-shot consistency generation ability inherently learned through the large-scale pretraining of Stable Diffusion and ControlNet (see Sec.3.2 and Sec.3.3). This large-scale pretraining on consistency generation can effectively help mitigate potential artifacts and ensures strong consistency across a wide range of scenarios. On top of that, our coupled denoising, coupled prompt guidance, and few-shot consistency fine-tuning strategy further enhance generation reliability. These components are extensively validated through comprehensive real-world benchmark experiments, especially in challenging scenarios with low overlap, occlusions, and cluttered environments.

Q2: About computational overhead.
A2: While image generation and feature fusion introduce additional computational cost, they substantially enhance the quality of geometric descriptors, leading to significantly improved registration robustness. To better balance performance and efficiency, our future work will explore single-step denoising techniques and knowledge distillation mechanisms to accelerate both image generation and feature fusion.

Q3: Few-shot fine-tuning requirement.
A3: Our method already demonstrates impressive performance in zero-shot scenarios (as shown in Fig. 8). The few-shot fine-tuning, requiring only a minimal amount of image pairs (~3K samples), further improves accuracy. We believe that this lightweight requirement is both practical and consistent with common industry practices.

Q4: Potential sensitivity to viewpoint selection?
A4: We thank the reviewer for raising this important point. In practice, our method exhibits strong robustness to viewpoint selection. Notably, our extensive experiments on the ScanNet and 3DMatch benchmarks already cover a wide range of diverse and challenging viewpoint configurations. Across these varying viewpoint conditions, our method consistently achieves significant improvements in registration performance, demonstrating its reliability under viewpoint selection. We attribute this robustness to the powerful generalization ability of Match-ControlNet, which benefits from the large-scale pretraining of foundation models, as well as our coupled conditional denoising design. We will incorporate this discussion into the revised manuscript.

Q5: About real-world applicability in autonomous driving or robotics.
A5: We appreciate the reviewer’s concern. (i) While 3DMatch and ScanNet are standard benchmarks, it is important to note that they are constructed from real-world RGB-D scans and have been widely adopted in robotics and embodied AI research, especially ScanNet, which is a common benchmark for indoor robotic perception; (ii) Furthermore, these datasets capture realistic and challenging conditions, such as low overlap, occlusions, and cluttered layouts, which are highly representative of deployment scenarios in indoor robotics; (iii) Regarding autonomous driving scenarios, as shown in the figure (https://anonymous.4open.science/r/rebuttal-688D/outdoor_vis.pdf), our Match-ControlNet can generate cross-view consistent images from partial LiDAR scan-based sparse depth maps, demonstrating promising generation effectiveness in outdoor self-driving scenes (please refer to A1@synA for more details); (iv) Moreover, we tested our method on in-the-wild, low-overlap data (from the author's room) captured by a mobile phone. The resulting generations (https://anonymous.4open.science/r/rebuttal-688D/wild_vis.pdf) further confirm the robustness of our approach in the real-world, unconstrained environment (please refer to A4@P29t for more details). We will further expand the discussion on applications to autonomous driving and robotics in our revision.

Q1: Discussion on applicability in outdoor LiDAR scenes.
A1: We sincerely appreciate this insightful comment. Our current Match-ControlNet indeed targets leveraging depth maps rather than LiDAR data for image generation. Compared to forward-facing depth maps, outdoor LiDAR point clouds provide omnidirectional (360-degree) scans that cannot be directly represented as conventional single-viewpoint depth maps due to their inherent multi-directional nature. In this rebuttal, we demonstrate the feasibility of partially projecting LiDAR points from predefined viewpoints to produce sparse depth maps for Match-ControlNet generation, yielding promising preliminary generation results as illustrated in the figures (https://anonymous.4open.science/r/rebuttal-688D/outdoor_vis.pdf). For future work, we plan to comprehensively address omnidirectional scans by employing multi-view depth maps or equirectangular images (a 2D image representation of LiDAR data) to adapt our Match-ControlNet to LiDAR point clouds. We will include the above discussion into our revised manuscript.

Q2: Consistency of texture information under low overlap?
A2: Our texture consistency under low overlap can be largely ensured by following four aspects: (i) Extensive pre-training knowledge: Our Match-ControlNet effectively unlocks the intrinsic zero-shot texture consistency capabilities of foundation models, which are derived from large-scale data pre-training. We believe this extensive training provides a strong foundation for ensuring texture consistency across diverse real-world challenges, including low-overlap scenarios. (ii) Joint image-level and prompt-level texture consistency enhancement: Our coupled conditional denoising mechanism and the coupled prompt guidance jointly prompt the texture consistency by incorporating both image-level texture consistency interaction and the prompt-level texture consistency guidance, significantly enhancing the texture coherence in low-overlap settings; (iii) Effective texture consistency fine-tuning mechanism: We introduce a few-shot consistency fine-tuning strategy (see Sec. 3.4) that requires only a small number of samples to further enhance the generation robustness of Match-ControlNet, particularly in challenging scenarios; (iv) Extensive validation under low-overlap conditions: To validate the robustness of our method under low-overlap conditions, we significantly increased the viewpoint separation (from 20 to 50 degrees on ScanNet and from 20 to 40 degrees on 3DMatch) as detailed in Line 311 (right column) and Line 362 (left column). Both quantitative and qualitative results (Fig. 7, first image; Fig. 8, third column) demonstrate reliable texture consistency. Additionally, we tested our method on in-the-wild, low-overlap data captured by a mobile phone. The resulting generations (https://anonymous.4open.science/r/rebuttal-688D/wild_vis.pdf) further confirm the robustness of our approach (please refer to A4@P29t for more details).

Q3: Reference Issue.
A3: We thank the reviewer for pointing out the missing FREEREG reference. We will include and thoroughly discuss this work in the revised manuscript.

Q4: Code availability.
A4: We will release our implementation publicly upon acceptance to facilitate reproducibility and enable broader verification by the community.