GS-LiDAR: Generating Realistic LiDAR Point Clouds with Panoramic Gaussian Splatting

Thanks to the reviewer for the valuable feedback and constructive suggestions. We addressed the raised concerns in the revised manuscript and our response below.

Q1: Insufficient reference

Thanks for pointing this out! We have cited 2D Gaussian primitives and periodic vibration properties in the revised Introduction to clearly acknowledge their contributions and clarified our use of these concepts.

Q2: Ambiguous symbol

We apologize for the confusion. In Line 199 (previous 204) of the preliminary section, $\mathcal{G}^{\prime}$ represents the influence of the projected Gaussian primitive on the corresponding pixel in the rendering image. $\mathcal{G}(u, v)$ in Lines 240-250 (previous 245-255) refers to the influence of a 2D Gaussian within its local tangent space, where $(u, v)$ are the coordinates in that local tangent space. We have made these distinctions clearer in the revised manuscript.

Q3: Incorrect definition

Thanks. We compute the intensity map in Line 346 (previous 316) by aggregating the intensity values, $\lambda$, associated with each Gaussian. We compute the ray-drop probability map in Line 352 (previous 323), $P_\mathrm{gs}$, by aggregating the ray-drop probability values, $\rho$. We have clarified this distinction in the revised manuscript.

Q4: The presentation of lidar rendering is not clear enough to make it difficult to follow

Thanks. Our rendering approach is based on tile-based rasterization, similar to the method used in 3D Gaussian splatting (3DGS). While 3DGS evaluates the influence of a single primitive in 2D image space, it does not explicitly define the ray-splat intersection point, which can result in view inconsistencies. In contrast, our method leverages 2D Gaussian primitives, modeled as 2D disks in 3D space, allowing us to explicitly compute the ray-splat intersection point. By analytically deriving the ray-splat intersection during rasterization, our approach ensures view-consistent panoramic rendering.

Dear Reviewer K7sa,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our response solves the concerns. We have addressed all the thoughtful questions raised by the reviewer (eg, insufficient reference, ambiguous symbol and presentation of LiDAR rendering) and we hope that our work’s impact and results are better highlighted with our responses. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors

Thanks to the reviewer for the valuable feedback and constructive suggestions. We addressed the concerns in the revised manuscript and our response below.

Q1: ''Generating''

Yes, we are actually doing the novel view synthesis (NVS) task in this work. We use the term ''Generating'' as we aim to produce novel LiDAR point clouds from novel viewpoints. We are considering to change our title to ''Simulating Realistic Point Clouds with Panoramic Gaussian Splatting'' if it is allowed.

Q2: 2DGS instead of 3DGS

Great suggestion! Although 3DGS performs well in camera image rendering, it is not suitable for panoramic LiDAR rendering. This limitation arises from the complexity of transforming 3D spatial points into LiDAR range map, which makes it challenging for the Jacobian matrix of the transformation to accurately approximate the effects of such a transformation. Given a point $(x,y,z)$ in the camera space, its corresponding radian angles $(\phi, \theta)$ can be calculated as:

\begin{pmatrix} \phi \\\\ \theta \end{pmatrix}= \begin{pmatrix} atan2(x, z)\\\\ atan2(\sqrt{x^2+z^2}, -y) \end{pmatrix} $$ By combining this with **Equation 8** from the paper, we can derive the projection of this point onto the LiDAR range map, resulting in its pixel coordinates $(\xi, \eta)$ as:

\begin{pmatrix} \xi \\ \eta \end{pmatrix}= \begin{pmatrix} \frac{W}{2\pi}atan2(x, z)+\frac{W}{2}\\ H\frac{atan2(\sqrt{x^2+z^2}, -y) - \mathrm{VFOV}_{\min}}{ \mathrm{VFOV}_{\max} - \mathrm{VFOV}_{\min}} \end{pmatrix}

$$$$

\mathrm{J}=\begin{pmatrix} \tilde{W} \frac{z}{x^2 +z^2} & 0 & -\tilde{W} \frac{x}{x^2 +z^2} \\ -\tilde{H} \frac{xy}{\sqrt{x^2+z^2}(x^2+y^2+z^2)} & \tilde{H}\frac{\sqrt{x^2+z^2}}{(x^2+y^2+z^2)} & -\tilde{H} \frac{yz}{\sqrt{x^2+z^2}(x^2+y^2+z^2)} \end{pmatrix} $$ where $\tilde{W} = \frac{W}{2\pi}$ and $\tilde{H} = \frac{H}{\mathrm{VFOV}\_{\max}-\mathrm{VFOV}\_{\min}}$.

We tile the plane with 2D and 3D Gaussian primitives of different colors and render their color and depth. Please refer to the supplementary material 2dgs_vs_3dgs.pdf. We can observe that for 3D Gaussian primitives, due to the poor approximation of the Jacobian for the projection transformation, the originally tightly arranged points become scattered, which also leads to the discontinuity in depth shown in Figure 8 of the updated paper. In contrast, the 2D Gaussian primitives, being based on ray intersection without approximation, exhibit the desired effect.

Q3: Performance

In the field of autonomous driving, the most critical factor for LiDAR simulation is rendering speed. Efficient rendering is essential to meet the requirements of closed-loop training and evaluation for driving agents. With efficient rendering, driving agents can undergo closed-loop training on huge datasets, bridging the final gap between simulation and real-world driving. Compared to LiDAR4D's 0.3 fps, GS-LiDAR rendering speed reaches 11 fps, which is 37 times faster, marking a key advancement in the usability of LiDAR simulation. This improvement in rendering speed is attributed to the carefully designed ray-splat intersection technique for LiDAR range image rendering and the efficient CUDA implementation.

Furthermore, in comparison with LiDAR4D, our method shows improvements across various metrics, demonstrating the reliability of our approach. For static scenes on the KITTI-360 dataset, GS-LiDAR achieves a 5.3% reduction in Chamfer distance for the simulated point cloud, a 10.7% reduction in RMSE for simulated depth, and a 9.8% reduction in RMSE for simulated intensity compared to the leading LiDAR4D. For dynamic scenes on KITTI-360 and nuScenes, GS-LiDAR achieves 11.4% and 13.1% reduction in RMSE for simulated depth, respectively.

Additionally, existing mainstream method for LiDAR processing by driving agents is voxel occupancy representation. As shown in Figure 5 (previous Figure 4), by leveraging supervision on median depth, our method is able to better model the edges of objects without scattering, which is a significant improvement for obtaining realistic and reliable voxel occupancy, an advancement that might be overlooked as it is not directly reflected in the metrics.

Q4: Can the proposed method handle solid-state LiDAR?

Yes, our method is fully capable of handling solid-state LiDAR, as it shares similar properties with conventional mechanical LiDAR, such as intensity and ray-drop. Solid-state LiDAR can be treated in the same way as mechanical LiDAR within our framework.

Dear Reviewer t1nR

We sincerely appreciate the reviewer's time for reviewing and thanks again for the valuable comments and the positive score!

Regarding the technical contribution, we strongly advocat that using Gaussian splatting for LiDAR simulation is non-trival. Simply applying 3D Gaussian primitives for point cloud reconstruction revealed that the nonlinear transformation from camera-space coordinates to range image pixels is highly complex. A naive approximation using the Jacobian matrix as a linear transformation is inadequate, leading to geometric inconsistencies across multiple viewpoints.

To this end, we propose a novel panoramic Gaussian splatting technique. This involves parameterizing the rays corresponding to panoramic range image pixels and intersecting them with 2DGS. We implemented this process efficiently using CUDA, making it feasible to model LiDAR using Gaussian primitives and perform efficient rendering.

Compared to the previous NeRF-based LiDAR4D approach, our method not only achieves improvements in metrics but also demonstrates a 37× increase in rendering speed. This significant advancement is attributed to our advanced panoramic rendering technique and its highly efficient implementation. This represents a significant step toward LiDAR simulation for autonomous driving. In a nutshell, we believe our work lays the foundation for LiDAR simulation in autonomous driving.

Hope our responses clarify above thoughtful question. It is very much appreciated if the reviewer can kindly check our response, and could increase the score of our paper and champion it for publication. Thank you!

Best wishes

Authors

Thanks to the reviewer for the valuable feedback and constructive suggestions. We addressed the raised concerns in the revised manuscript and our response below.

Q1: Novelty

Closed-loop simulation for autonomous driving urgently requires efficient rendering of LiDAR point clouds from novel viewpoints to facilitate multimodal driving agent training and evaluation. However, existing LiDAR reconstruction and novel view synthesis methods are predominantly based on NeRF, whose training and inference speeds are prohibitively slow, severely limiting progress in fields such as autonomous driving.

Motivated by this limitation, using 3D Gaussian primitives for point cloud reconstruction naturally emerges as a promising idea. However, our experiments revealed that the nonlinear transformation from camera-space coordinates to range image pixels is highly complex. A naive approximation using the Jacobian matrix as a linear transformation is inadequate, leading to geometric inconsistencies across multiple viewpoints.

To address this issue, we shifted our focus to 2D Gaussian primitives, which avoid approximations by leveraging ray-splat intersections. To enable 2DGS for LiDAR range image rendering, we proposed a novel panoramic Gaussian splatting technique. This involves parameterizing the rays corresponding to panoramic range image pixels and intersecting them with 2DGS. We implemented this process efficiently using CUDA, making it feasible to model LiDAR using Gaussian primitives and perform efficient rendering. To this end, utilizing 2D Gaussian as the representation for LiDAR simulation is non-trival.

Compared to the previous NeRF-based LiDAR4D approach, our method not only achieves improvements in metrics but also demonstrates a 37× increase in rendering speed. This significant advancement is attributed to our advanced panoramic rendering technique and its highly efficient implementation. This represents a significant step toward LiDAR simulation for autonomous driving.

Furthermore, through extensive experiments, we enhanced each Gaussian primitive with additional attributes, including intensity and ray-drop probability. We also incorporated periodic viberation property, enabling the Gaussian primitives to model dynamic scenarios. In a nutshell, we believe our work per se is novel and our contribution lays the foundation for LiDAR simulation in autonomous driving.

Q2: Ray-splat intersection

Thanks. $(u, v)$ are the coordinates on the local tangent plane of the 2D Gaussian primitive, with the coordinate axes aligned with the directions $t_u$ and $t_v$ in the world coordinate system. To transform this coordinate into the world coordinate system, the following steps are performed:

(i) Apply scaling by $s_u$ and $s_v$ along the corresponding axes, resulting in the relative position with respect to the 2DGS center: $s_u t_u u + s_v t_v v$.

(ii) Translate this relative position by the 2DGS center $\tilde{\mu}$, yielding the world coordinates: $\tilde{\mu} + s_u t_u u + s_v t_v v$

To transform the coordinates into the camera coordinate system, the world-to-camera transformation matrix $𝑊$ should be applied. The final transformation can be expressed as: $W(\tilde{\mu} + s_u t_u u + s_v t_v)$, which is consistent with Equation 6 in the paper.

We have also included an illustration in the updated paper; please refer to Figure 4.

Dear Reviewer YP8o,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our response solves the concerns. We have addressed all the thoughtful questions raised by the reviewer (eg, novelty, ablation of ray-splat intersection and experiments on Waymo dataset) and we hope that our work’s impact and results are better highlighted with our responses. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors

Dear Reviewer YP8o,

We genuinely appreciate the reviewer’s recognition of our rebuttal, as well as reviewer's valuable time and efforts for reviewing our work.

Honestly speaking, the quality of a piece of work should not be judged solely on how new is the method, but more on what problem it solved and to what extend it solved the problem.

In our work, we are the first to introduce Gaussian splatting into LiDAR NVS, utilizing panoramic ray-splat intersection in LiDAR range image rendering and addressing the geometric inconsistencies across multiple viewpoints that arise from the naive use of 3D Gaussians. Our approach successfully handles static and dynamic scenes, while enabling fast rendering capabilities.

Neither the existing techniques, nor a combination of existing methods, could solve this problem. The concept of 3DGS first appeared on arXiv on August 8, 2023, while the most recent LiDAR NVS method, LiDAR4D (CVPR, March 6, 2024), still relies on NeRF. This highlights the non-triviality and challenges of proposing GS-based method into the LiDAR NVS. In other words, we firmly believe our approach per se is novel, and hope our panoramic rendering method could set a new paradigm for future GS-based LiDAR NVS.

As Reviewer aQ3K noted, "This paper still makes significant contributions to LiDAR NVS."

Hope our responses clarify on the point of “novelty”. It is very much appreciated if the reviewer could champion our work for publication. Thank you!

Best wishes,

Authors

Q4: Normal consistency

For the rendered depth map, we first back-project it into 3D space to obtain a 3D point cloud. The spatial gradients of the 3D point cloud are then used as pseudo ground-truth normals. Since the depth map and ray-drop are rendered separately, the resulting depth map is dense and unaffected by ray-drop. While object edges may affect the accuracy of this loss, their sparsity and the relatively low weight assigned to this loss ensure that it optimizes the geometric consistency of surfaces without disrupting the overall structure of the scene. We additionally add the ablation experiment of normal consistency loss below and in Table 4 of the updated paper.

We can observe that the normal consistency loss, by enhancing the geometric consistency in 3D space, significantly reduces the chamfer distance (CD) between the rendered point cloud and the ground truth point cloud in 3D space.

Q5: Initialize Gaussian points

For dynamic scenes, we initialize the positions of Gaussians by sampling from the LiDAR point clouds of each view. The life peak property $\tau$ of each Gaussian is initialized based on the timestamp of the view from which it was sampled. As our Gaussian primitives incorporate time-dependent opacity (Eq. 4), which peaks at the life peak $\tau$ and decays over time, this ensures a reasonable initialization where Gaussians sampled at a given timestamp primarily influence adjacent time periods. We initialize all scenes with $1 \times 10^6$ Gaussians. After optimization, the final number of Gaussians typically ranges from approximately $5 \times 10^6$ to $1 \times 10^7$, depending on the scene's complexity.

Q6: 3DGS in Figure 1

Great suggestion! Although 3DGS performs well in camera image rendering, it is not suitable for panoramic LiDAR rendering. This limitation arises from the complexity of transforming 3D spatial points into LiDAR range map, which makes it challenging for the Jacobian matrix of the transformation to accurately approximate the effects of such a transformation. Given a point $(x,y,z)$ in the camera space, its corresponding radian angles $(\phi, \theta)$ can be calculated as:

\begin{pmatrix} \phi \\\\ \theta \end{pmatrix}= \begin{pmatrix} atan2(x, z)\\\\ atan2(\sqrt{x^2+z^2}, -y) \end{pmatrix} $$ By combining this with **Equation 8** from the paper, we can derive the projection of this point onto the LiDAR range map, resulting in its pixel coordinates $(\xi, \eta)$ as:

\begin{pmatrix} \xi \\ \eta \end{pmatrix}= \begin{pmatrix} \frac{W}{2\pi}atan2(x, z)+\frac{W}{2}\\ H\frac{atan2(\sqrt{x^2+z^2}, -y) - \mathrm{VFOV}_{\min}}{(\mathrm{VFOV}_{\max}-\mathrm{VFOV}_{\min})} \end{pmatrix}

$$$$

\mathrm{J}=\begin{pmatrix} \tilde{W} \frac{z}{x^2 +z^2} & 0 & -\tilde{W} \frac{x}{x^2 +z^2} \\ -\tilde{H} \frac{xy}{\sqrt{x^2+z^2}(x^2+y^2+z^2)} & \tilde{H}\frac{\sqrt{x^2+z^2}}{(x^2+y^2+z^2)} & -\tilde{H} \frac{yz}{\sqrt{x^2+z^2}(x^2+y^2+z^2)} \end{pmatrix} $$ where $\tilde{W} = \frac{W}{2\pi}$ and $\tilde{H} = \frac{H}{\mathrm{VFOV}\_{\max}-\mathrm{VFOV}\_{\min}}$.

We tile the plane with 2D and 3D Gaussian primitives of different colors and render their color and depth. Please refer to the supplementary material 2dgs_vs_3dgs.pdf. We can observe that for 3D Gaussian primitives, due to the poor approximation of the Jacobian for the projection transformation, the originally tightly arranged points become scattered, which also leads to the discontinuity in depth shown in Figure 8 of the updated paper. In contrast, the 2D Gaussian primitives, being based on ray intersection without approximation, exhibit the desired effect.

Q7: Motion trajectories

Our method is able to render the velocity of the Gaussian points. We additionally provided the visualization video pvg_trajectory.mp4 in the supplementary materials. As shown in the video, moving objects in the scene have higher velocities (blue), while static objects exhibit velocities close to zero (red). This demonstrates that multiple high-speed, short-period periodic vibration Gaussian points can collectively represent long trajectory motion effectively.

Thanks for the valuable feedback and constructive suggestions. We addressed the raised concerns in the revised manuscript and our response below.

Q1: Contribution

Closed-loop simulation for autonomous driving urgently requires efficient rendering of LiDAR point clouds from novel viewpoints to facilitate multimodal driving agent training and evaluation. However, existing LiDAR reconstruction and novel view synthesis methods are predominantly based on NeRF, whose training and inference speeds are prohibitively slow, severely limiting progress in fields such as autonomous driving.

Motivated by this limitation, using 3D Gaussian primitives for point cloud reconstruction naturally emerges as a promising idea. However, our experiments revealed that the nonlinear transformation from camera-space coordinates to range image pixels is highly complex. A naive approximation using the Jacobian matrix as a linear transformation is inadequate, leading to geometric inconsistencies across multiple viewpoints.

To address this issue, we shifted our focus to 2D Gaussian primitives, which avoid approximations by leveraging ray-splat intersections. To enable 2DGS for LiDAR range image rendering, we proposed a novel panoramic Gaussian splatting technique. This involves parameterizing the rays corresponding to panoramic range image pixels and intersecting them with 2DGS. We implemented this process efficiently using CUDA, making it feasible to model LiDAR using Gaussian primitives and perform efficient rendering. To this end, utilizing 2D Gaussian as the representation for LiDAR simulation is non-trival.

Compared to the previous NeRF-based LiDAR4D approach, our method not only achieves improvements in metrics but also demonstrates a 37× increase in rendering speed. This significant advancement is attributed to our advanced panoramic rendering technique and its highly efficient implementation. This represents a significant step toward LiDAR simulation for autonomous driving.

Furthermore, through extensive experiments, we enhanced each Gaussian primitive with additional attributes, including intensity and ray-drop probability. We also incorporated periodic viberation property, enabling the Gaussian primitives to model dynamic scenarios. In a nutshell, we believe our work per se is novel and our contribution lays the foundation for LiDAR simulation in autonomous driving.

Q2: Ray-splat intersection

Although we calculate the intersection of each ray with the Gaussian primitives, we still adopt the tile-based rendering method proposed in 3D Gaussian splatting to achieve efficient rendering.

As illustrated in Figure 4 in the updated paper, we first transform the epipolar coordinates into pixel coordinates and sort the Gaussian primitives within each tile. For pixel rendering, the $\alpha$ and depth are determined based on the intersection results.

Q3: Performance improvement

In the field of autonomous driving, the most critical factor for LiDAR simulation is rendering speed. Efficient rendering is essential to meet the requirements of closed-loop training and evaluation for driving agents. With efficient rendering, driving agents can undergo closed-loop training on huge datasets, bridging the final gap between simulation and real-world driving. Compared to LiDAR4D's 0.3 fps, GS-LiDAR rendering speed reaches 11 fps, which is 37 times faster, marking a key advancement in the usability of LiDAR simulation. This improvement in rendering speed is attributed to the carefully designed ray-splat intersection technique for LiDAR range image rendering and the efficient CUDA implementation.

Furthermore, in comparison with LiDAR4D, our method shows improvements across various metrics, demonstrating the reliability of our approach. For static scenes on the KITTI-360 dataset, GS-LiDAR achieves a 5.3% reduction in Chamfer distance for the simulated point cloud, a 10.7% reduction in RMSE for simulated depth, and a 9.8% reduction in RMSE for simulated intensity compared to the leading LiDAR4D. For dynamic scenes on KITTI-360 and nuScenes, GS-LiDAR achieves 11.4% and 13.1% reduction in RMSE for simulated depth, respectively.

Additionally, existing mainstream method for LiDAR processing by driving agents is voxel occupancy representation. As shown in Figure 5 (previous Figure 4), by leveraging supervision on median depth, our method is able to better model the edges of objects without scattering, which is a significant improvement for obtaining realistic and reliable voxel occupancy, an advancement that might be overlooked as it is not directly reflected in the metrics.

Dear Reviewer aQ3K,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our response solves the concerns. We have addressed all the thoughtful questions raised by the reviewer (eg, contribution, normal consistency and 2DGS v.s. 3DGS) and we hope that our work’s impact and results are better highlighted with our responses. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors