GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields

We greatly appreciate your positive feedback on our paper's presentation and the novelty of our method. Below are the responses to your concerns.

[Q1]: The issue of the algorithm's complexity and the role of the components(SR and geometric constraints).

1. Complexity of Our Algorithm. Our method combines LiDAR-NeRF with a Geo-optimizer, which constitutes the two main modules of our model. The Selective-Reweighting (SR) strategy (Sec 3.4) and the 3D geometric constraints (Sec 3.5) do not actually complicate the model structure and almost do not increase the optimization time (please kindly refer to global rebuttal for more details). Specifically, the SR strategy adjusts the learning rate for different frames during optimization to avoid overfitting frames with high loss during training. The geometric constraints introduce a geometry-based loss function to ensure that the synthetic point cloud possesses more accurate geometric features.

2. The Role of the Components. We appreciate your inquiry regarding the roles of these two strategies(SR and geometric constraints). Given that the Geo-optimizer is crucial for the success of registration, it significantly enhances performance. Without this core module, the remaining two optimization strategies struggle to exert their influence effectively. However, these strategies play pivotal roles in specific aspects and metrics. We will now provide detailed explanations:

2.1. SR Strategy. SR strategy aims to prevent overfitting of NeRF to individual frames and plays a crucial role in specific sequences. It primarily addresses extreme cases where most frame poses are optimized well, except for a few overfitted outliers. However, in most sequences, the significant improvement in registration success with NeRF+Geo-optimizer reduces extreme cases. Another factor to consider is that our ablation experiments are averaged over five Nuscenes sequences, which may average out its effects on certain frames. Considering its effectiveness becomes particularly evident in specific sequences, especially when outlier frames appear during the optimization process, we conducted an ablation study on the specific sequence (KITTI-360, Seq-3) to demonstrate its effectiveness, as shown in Fig. (2)(a) in the attached PDF and table below.

As shown in the table, in the absence of the SR strategy, the pose metrics exhibit a sharp decline due to the presence of outlier frames. Additionally, the negative impact of the outlier frame on NeRF optimization significantly contributes to a comprehensive degradation in NVS performance.

In summary, the SR strategy can enhance the overall performance of the model by preventing NeRF from overfitting high-loss frames in the early stages and enables our algorithm to be more robust across various sequences. Importantly, this strategy introduces no additional computational burden in optimization time and does not increase system complexity, thereby demonstrating practical utility. Furthermore, we believe that this simple yet effective strategy will also be beneficial for other pose-free NeRF works.

2.2. 3D Geometric Constraints. Regarding 3D geometric constraints, we acknowledge their marginal improvement in pose optimization accuracy and the 2D evaluation metrics of NVS. Their impact primarily manifests in optimizing the 3D geometric features of the synthesized point cloud, offering specific practical significance. Primarily, our geometric constraints supervise the 3D aspects of the synthesized point cloud through a dedicated loss function, aiming for enhanced geometric fidelity. Therefore, the improvements from geometric constraints are primarily reflected in 3D point cloud error metrics, notably reducing CD (a 17.8% reduction). This indicates that geometric constraints help align the distribution of our model-synthesized point clouds closer to the ground truth point clouds in 3D space.

Additionally, since our geometric constraints primarily affect the 3D domain, their impact on the evaluation metrics focusing on 2D range maps in NVS is not significantly pronounced. Hence, we illustrate the effectiveness of our geometric constraints from a 3D perspective.

As shown in Fig. (2)(b) in the attached PDF, since previous LiDARNeRF[52] is optimized based on 2D range images, it effectively performs optimization from the front view perspective. Using 2D-based loss without our geometric constraints can still yield satisfactory results from the front view (compare Fig. (2) b.1 and b.2). However, in the bird's-eye view, the reconstruction quality significantly deteriorates without our geometric constraints(compare Fig. (2) b.3 and b.4). This is because our geometric constraints provide supervision from a 3D perspective, addressing the limitations of 2D perspective supervision. As a result, our synthesized point cloud exhibits more realistic geometric features, such as smoother and more realistic surfaces.

[Q2]: Run-time Comparison and Analysis. Please refer to global rebuttal.

We greatly appreciate your careful reading and your questions provide valuable insights. We are pleased to address your concerns and engage in further discussion with you.

[Q1]: Initialization of Poses. We follow previous works BARF[32] and NeRFmm[59], which are classical in the pose-free NeRF domain, to set up experiments. This setting is practical in autonomous driving scenarios, where pose estimations are not always accurate. Existing datasets with pose sensors(GPS, IMU), require calibration via time-consuming registration methods. Without initial pose information or with sparse inputs, SLAM or registration errors are more significant. Thus, adding random perturbations to each frame is a reasonable test of algorithm performance.

Nevertheless, we also perform experiments on Nuscenes with all initial poses set to identity, and our algorithm remains effective, as shown in Fig. (1) and Tab. (1) of the attached PDF. Additionally, you may run the demo we provide by adding the flag '--no_gt_pose', which implies that all poses are initialized to identity. For practical applications, initialization can use either the identity matrix, coarse point cloud registration results, or GPS/IMU information, especially in outdoor scenes.

[Q2]: Reasons for Prohibiting the Joint Capture of Image and LiDAR Scans. Firstly, as noted in L21-23, pose-free NeRF in camera domain approaches aim to eliminate NeRF's reliance on SfM algorithms, which are time-consuming and often fail in homogeneous regions and fast-changing view-dependent appearances. Thus, the primary motivation for pose-free NeRFs is to enable reconstruction without relying on time-consuming SfM computations and avoiding the instability of SfM failures. Additionally, directly using point cloud registration algorithms might be a better choice. This is because 3D point cloud registration, in most cases, achieves more accurate poses compared to 2D-based registration. However, as shown in Fig.(1) of the main text and L33-38, pairwise and multiview point cloud registration may still be prone to local optima and error accumulation, making it difficult to achieve globally accurate poses.

In summary, poses obtained from GPS/IMU are often not sufficiently accurate(but can serve as a "bad initialization"), both image-based SfM and point cloud registration algorithms suffer from error accumulation and instability due to registration failures, and introduce extra computational overhead in reconstruction. Therefore, to achieve high-quality and efficient reconstruction, pose-free NeRFs are of practical significance.

Besides, We commit to a LiDAR-only framework, aiming at achieving efficient reconstruction and NVS of point clouds without relying on images and accurate poses. Therefore, it has stronger applicability in LiDAR NVS.

[Q3]: Motivation and Benefit of Using NeRF and Downstream Application. Traditional simulators and explicit reconstruct-then-simulate methods exhibit a large domain gap compared to real-world data. Moreover, traditional simulators like CARLA simulate within virtual environments, exhibiting diversity limitations and a reliance on costly 3D assets. Explicit reconstruction methods like LiDARsim[36] need to reconstruct mesh surfel representations, which are challenging for recovering precise surfaces in large-scale complex scenes. Additionally, NeRF inherently applies to the modeling of intensity and ray drop of LiDAR and does not require an intermediate modality for reconstruction, thereby eliminating the domain gap. Thus, some methods such as NFL[24] and LiDAR-NeRF[52], utilize NeRF to represent LiDAR data but need accurate poses.

Besides, we appreciate your attention to the applications of our method. (1) Autonomous Driving Simulation: As shown in Fig. (3) of the attached PDF, we can conduct autonomous driving simulations in real-world scenarios rather than in simulators like CARLA, which is a prominent research field in autonomous driving. (2) Simulating point clouds from different LiDAR sensors: Our method allows for the adjustment of LiDAR sensors during NVS process. This flexibility facilitates simulations with various LiDAR configurations.

Additionally, unlike previous methods that require accurate poses from multiple sensors, our approach circumvents this complex and time-intensive process, offering substantial practical benefits. Further applications can be referenced in LiDAR4D[71] and we look forward to discussing them with you in the future discussion.

[Q4]: Pose Registration metrics. Registration Recall(RR) can be computed on this data. However, inlier Ratio and Feature Matching Recall cannot be reported on this data because these metrics are only applicable to feature-based point cloud registration methods, and our method does not require the computation of point features. Moreover, ATE and RPE are widely adopted in many pose-free NeRF works, such as Nope-NeRF[6] for evaluating the registration accuracy of a sequence of image/point clouds. However, RR is typically applied in pairwise registration, and is challenging to measure the registration performance of a sequence, because error accumulation in a sequence can prevent it from accurately reflecting registration performance. We realized that it is meaningful to compute the RR of adjacent frames by comparing gt relative poses with predicted relative poses. The results are shown in Tab. (2) of the attached PDF.

[Weekness1(W1)]/ W[2]/ W[3]. Please refer to Q1/ Q2,Q3/ Q4.

[W4.]: Determining Outliers. We directly select the top k frames with the highest training loss in each epoch as outliers without any priors (L206). The reason and effectiveness can be referenced in L201.

[W5, W6]. Please refer to the global rebuttal.

[W7]: Translation Representation. We project the point cloud into a range image and each pixel on the image is represented as a ray, where the translation is the ray origin in the world coordinate system.

We greatly appreciate your positive feedback on our method's novelty and the significance of our new task to reduce the impact of noisy estimated LiDAR poses during LiDAR NVS. The issues you raised have been effectively addressed as follows. Regarding weaknesses 2,5 and your concern about the time-consuming limitation, please kindly refer to the global rebuttal.

[Q1] The Issue of Term J in Pose Representation Formula. Commonly used $\xi \in \mathfrak{se}(3)$ represents the transformation in a coupled manner, the translation component of $T$ will be influenced by the rotation parameter $\phi \in \mathfrak{so}(3)$, as shown in Eq. (5) and L153-154. This means the translation of the rigid body's centroid is affected by the rotation. Let $B$ and $A$ denote the transformed and original point clouds, where $\overline{B}$ and $\overline{A}$ represent the homogeneous coordinates. Then, $\overline{B} = T\overline{A}$. By omitting $J$ in Eq. (5), according to the principles of matrix multiplication, our method for rigid body pose transformation $\overline{B} = T_{**w/o** J}\overline{A}$ equates to $B = \boldsymbol{R}A + \rho = \sum_{n=0}^{\infty} \frac{1}{n!}\left(\phi^{\wedge}\right)^n A+\rho$, $\phi \in \mathfrak{so}(3)$ and $\rho \in \mathbb{R}^3$ are the pose parameters to be updated, and the translation is represented solely by $\rho \in \mathbb{R}^3$. In this way, the rotation parameter $\phi$ and the translation parameter $\rho$ represent the rigid body motion as rotation about the center of mass and translation of the center of mass. This method represents rotation and translation separately and independently, which is reasonable and physically meaningful.

Besides, in the context of “independently” in L157 of the main text, we refer to the fact that our approach optimizes rotation and translation simultaneously, but their updates are decoupled and do not interfere with each other. Thus, there is no conflict between the Appendix and the main text. We fully understand your concern and believe that changing “independently” to “in a decoupled manner” would be more appropriate.

[Q3] The Issue of Missing Descriptions in ICP Algorithms. Thanks for your comments. By describing “in line with the optimization objective of each step in the ICP algorithm"(L173), we intend to state that our CD-based optimization is inspired by the well-known classic ICP algorithm. We do not use ICP in our method, and the core optimization step is clearly explained in Eq. (7) and L190-192 (through gradient descent). Given the fluency of description, we briefly summarize the optimization steps of ICP in L163-L166 and Eq. (6) in our main text. We will include the detailed optimization steps of ICP in the supplementary material.

[Q4]: The Issue of Distance Representation ($d$) in Eq. (8). We apologize for not having explained our method more clearly. $d$ denotes the distance between a pair of matching nearest neighbor points. We intend to use the subscript “clipped" to indicate that this distance has been truncated to be greater than a predefined threshold. Therefore, $d_{clipped}$ represents the $d_i$ we actually used in the first formula. We will update $d_{clipped}$ to $d_i$. The revised formula is as follows:

$w_{i} = \frac{\mathrm{exp}(t/d_i)}{\sum_{i=1}^N \mathrm{exp}(t/d_i)}, t=\mathrm{scheduler}(t_0),d_i=\mathrm{max}(d_0,d)$

where $d$ denotes the distance between a pair of matching nearest neighbor points, $t$ is the temperature to sharpen the distribution of the $d_i$, and $d_0$ is a threshold we set.

[Q6]: The Issue of Caption Content of Fig. (3). Based on your suggestion, we will further revise the caption for better readability. The revised caption of Fig. (3) is “Graph-based RCD (left). We introduce control factor $t$ in CD to diminish the weighting of non-overlapping regions between point clouds. Geo-optimizer and its impact on pose optimization (right). Pose errors are rapidly reduced after each increase caused by NeRF's incorrect optimization direction. Comparison of (a) and (b) shows Geo-optimizer prevents incorrect pose optimization of NeRF.".

The content shown on the left side of Fig. (3) is explained in L185-190. As for the right side of Fig. (3), our primary intention is to provide readers with a clear and intuitive comparison through the figures. To facilitate this, we have carefully annotated the figures with arrows and text for better guidance. Reviewer fgnm and Reviewer TE2m also point out that this figure is informative and accurately demonstrates the effectiveness of our method. We hope the revised title will better explain the information in the figure.

[Q7]: The Issue of Organizational Structure in the Main Text. We fully understand your perspective on creating a more cohesive structure. However, within the domain of NeRF / pose-free NeRF, it is common practice to place the NeRF optimization loss (Sec 3.5) at the end to better present the overall pipeline, such as Nope-NeRF[6], NFL[24], LiDARNeRF[52].

Therefore, we adopt the following structure: Sec 3.2 discusses two main aspects of pose-free NeRF, $*i.e.*$, the representation of pose and the encoding method of NeRF. This serves as a foundational premise for the subsequent discussions. Specifically, the representation of pose is essential for pose optimization, as the $T \in \text{SE}(3)$ in original NeRF lacks gradient propagation capabilities ($T + \delta T \notin \text{SE}(3)$). And the encoding method is fundamental across all NeRF works. Then we proceed to discuss the optimization of pose (Sec. 3.3) and NeRF (Sec 3.4-3.5). These three sections (3.3-3.5) constitute our primary contributions. Based on your suggestion, we will integrate Sec 3.5 with the NeRF encoding content from Sec 3.2, thereby encapsulating the entirety of the NeRF optimization process within Sec 3.5, which will help to make the paper more cohesive.

Could you please detail how normals are being computed? I believe this is an important detail to include in the manuscript since the proposed normal loss requires gradients to be backpropagated through the normal estimation function.