Is Your LiDAR Placement Optimized for 3D Scene Understanding?

Dear Reviewer iXBb,

Thanks for devoting time to this review and providing valuable comments.

---

Q: The representation may be a little confusing. The M-SOG and the optimization method, under my understanding, are task-independent. However, there are two optimized LiDAR placements related to Det and Seg tasks, which is a little confusing.

A: Sorry for the confusion and thanks for the comment. To clarify, the M-SOG and placement optimization strategy are task-independent, but the optimization result depends on the semantic classes involved.

For object detection (Det) and semantic segmentation (Seg), the semantic classes differ. Det considers three object categories: Car, Pedestrian, and Bicycle, while Seg includes 21 object and environment classes, such as Building and Fence. Consequently, the placement optimization result varies between Det and Seg due to the different semantic class definitions.

Q: To demonstrate the effectiveness of M-SOG, it's recommended to draw a scatter plot of M-SOG and model performance in Tab 2 and Tab 3, which can be more intuitive.

A: Thanks for your suggestion. We include the plot in the attached single-page rebuttal PDF.

As illustrated, the results demonstrate a clear correlation, where the performance generally improves for both tasks as the M-SOG increases. While there might be fluctuations in some placements with specific algorithms, the overall relationship follows a linear correlation, highlighting the effectiveness of our M-SOG for computationally efficient sensor optimization purposes.

Q: I am curious about the influence of LiDAR numbers. It is better to add an experiment to explore the most cost-effective LiDAR number.

A: Thanks for your suggestion. We appreciate the reviewer's comment highlighting the importance of determining the most cost-effective number of LiDARs. In our study, we investigate the strategic placement of 4 LiDARs, as preliminary experiments indicated that a perception system utilizing 4 LiDARs is more cost-effective compared to other configurations. We test the detection and segmentation performance using 1 to 5 LiDARs, respectively. The results of experiments are as follows:

We observe that the detection performance of BEVFusion-L improved as the number of LiDARs increased. The most cost-effective LiDAR number can be 4 LiDARs as 5 LiDARs only have slight improvement. We will include more experiments on this point using other detectors in the revision.

Additionally, we also supplement the segmentation result of SPVCNN using 1 to 5 LiDARs, respectively.

Similar to detection, we observe improved segmentation performance when the use of more LiDARs. Again, using 4 LiDARs tends to achieve the optimal trade-off between the segmentation accuracy and computational cost. To further consolidate this finding, more results on other LiDAR segmentation methods will be supplemented during this revision.

Q: M-SOG seems to be a global metric. Can it also reflect the scene coverage of a specific local area?

A: Thanks for your question. Yes, the M-SOG can be adapted to reflect the scene coverage of specific local areas by segmenting the scene into smaller regions.

The score of M-SOG is connected to the voxels traversed by LiDAR rays in the scene. In our study, the LiDAR is set to sweep [-24.6, 2.0] vertically and [-180.0, 180.0] horizontally in degree. To compute the M-SOG of local areas, we can reset the sweep range of LiDAR rays according to the specific local areas we are interested in, and then get the local traversed voxels. By performing the same process of global M-SOG to local traversed voxels, we can obtain the M-SOG scores which reflect the scene coverage of specific areas.

Dear Reviewer PfQe,

Thanks for devoting time to this review and providing valuable comments.

---

Q: In the experiments, although the baseline LiDAR placement methods are based on the existing configuration of current autonomous vehicle systems (Fig 7), it is better to compare the proposed method with some randomly sampled LiDAR placements. This can further justify the effectiveness of CMA-ES compared to a wider range of scenarios.

A: Thanks for the insightful comments. We acknowledge that the theoretical upper bound for the performance of the placements optimized by our method cannot be exceeded by random LiDAR placements.

Although enumerating all possible LiDAR placements for random selection is computationally infeasible, we add experiments on evaluating the performance of three random placement strategies as follows:

As can be seen from the above study, randomly sampled placements tend to be sub-par compared to the existing and our optimized strategies. This verifies the effectiveness of the proposed CMA-ES under more general scenarios.

Q: It is not clear how the density of the occupancy grid (Sec 3.3) will influence the surrogate metric, as the estimation of semantic distribution depends on the number of samples.

A: Thanks for your question.

1. Density of samples: As the number of samples used to merge and synthesize dense semantic occupancy grids increases, the accuracy of M-SOG metrics is enhanced. This improvement is attributed to the smoother and more continuous semantic representation of the scene resulting from the increased density of semantic LiDAR points. A higher sample density provides a more detailed and granular depiction of the environment, reducing gaps and ambiguities.

2. Density of voxels: As the grids become denser, the M-SOG scores for all placements will slightly increase, and the linear consistency between our metric and perception performance improves. However, with increasing density, linear consistency of M-SOG may converge as the voxel size decreases to very small dimensions. In the experiments, we used voxel sizes matching those in the detection or segmentation training parameters to compute M-SOG, aiming for better linear consistency. For example, for segmentation, we used voxel sizes of [0.10, 0.10, 0.10]. We computed the M-SOG of other voxel sizes for comparison as follows: |Voxel Grid Size|Line-roll|Trapezoid|Line|Pyramid|Square|Pyramid-roll|Center|Ours| |-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| |[0.20, 0.20, 0.20]|-5.64|-6.72|-6.13|-5.88|-4.30|-4.24|-4.52|-4.70| |[0.10, 0.15, 0.20]|-3.80|-3.65|-3.22|-3.29|-2.67|-2.39|-2.47|-2.28| |[0.10, 0.10, 0.10]|-3.13|-2.89|-2.62|-2.56|-2.35|-1.63|-1.58|-1.29| |[0.05, 0.10, 0.10]|-2.97|-2.75|-2.40|-2.33|-2.12|-1.37|-1.24|-1.13|

Q: The experiments are conducted in the simulation. Although CARLA is a rather realistic simulator and has been widely adopted in academia, it is unknown how well the proposed method works in a real-world setting. Perhaps it is not feasible to conduct on-site experiments, but more justifications regarding this aspect are needed for further discussion.

A: Thanks for your insightful comments. We appreciate the reviewer's concern regarding the applicability of our proposed method in real-world settings.

The main challenge of conducting real-world experiments is the inability to ensure identical scenarios for testing different placements. Given multiple LiDAR placements, we need to collect data multiple times in the same area. Even if we ensure that the ego vehicle follows the exact same route and the data is collected at the same time each day, we cannot guarantee the activities of traffic participants within the scene will be identical. This variability in scenarios can lead to difficulties in making fair comparisons. Real-world experiments are also time-consuming and resource-intensive, which can limit the extent and frequency of testing. We will include more discussion of extensions to real-world settings in the revised version.

Q: The optimization process and data generation might require substantial computational resources. Either the main body or the appendix omitted such details. It is highly recommended to supplement such details as they are critical for follow-up work.

A: Thanks for your suggestion. For data generation in CARLA, we recommend using at least 8GB GPUs. For the optimization process, the hardware requirements primarily depend on the resolution of the voxel representation used for the scene.

A CPU with a substantial number of cores and at least 64GB of RAM is necessary for efficient optimization through parallel processing. The amount of RAM needed scales with the resolution of the voxel representation. We will include details in the revision.

Q: Can the proposed benchmarks and optimization strategies be extended to other 3D scene understanding tasks, such as 3D object tracking, semantic occupancy prediction, etc?

A: Thanks for your question. Yes, the proposed benchmarks and optimization strategies can indeed be extended to other 3D scene understanding tasks, such as 3D object tracking and semantic occupancy prediction, provided we can obtain the semantic prior knowledge from the training dataset.

In fact, the M-SOG is task-agnostic as it only evaluates the overall semantic coverage quality provided by the LiDAR placements. The differences in optimization results only arise from the varying semantic settings defined in different tasks.

Dear Reviewer DpmT,

Thanks for devoting time to this review and providing valuable comments.

---

Q: Writing can be improved.

A: Thanks for your comment. Due to limited space for response, for minor modifications (e.g. notations), please refer to the above Author Rebuttal section. For specific questions, please refer to the following.

Q: The voxels are not independent of each other.

A: Sorry for the confusion. The voxels in a 3D space are not independent of each other due to spatial correlations in a single frame. We intended to convey that the probability distribution of occupancy for each voxel can be modeled independently across a large number of frames.

Q: Is there an invalid setup where the metric would not capture properly?

A: Thanks for asking. In our optimization, we consider LiDAR placement within a limited hyper-rectangle space on the vehicle roof. This ensures practical and relevant placements, avoiding physically or operationally infeasible configurations.

Q: The optimization in Sec 3.3 is not well explained.

A: In Sec 3.3, we introduce an optimization approach using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to find optimal LiDAR placements. This method iteratively searches and refines LiDAR configurations in hyper-rectangle solution space to maximize the M-SOG based objective function. We will include more in the revision.

Q: How significant and practical is theoretical analysis?

A: Thanks for the question. The theoretical analysis gives a guarantee for the optimality of the local optima from heuristic CMA-ES under the assumption of the Lipschitz constant $k_G$ and the distance $C_M$ between the maximal and minimal of objective function G. As we have shown on Line 201-204, Lipschitz constant $k_G$ and the difference between maximum and minimum of G $C_M$ over S can be approximated through calculating $G(u^k_i )$ of Algorithm 1 for each sampled $u^k_i$ over the δ density Grids subset. More specifically, in each iteration $k$, $k_G \sim = \max_i (G(u^k_{i+1}) - G(u^k_i))/(u^k_{i+1} - u^k_i)$, $C_M \sim= \max_i G(u^k_i) - \min_i G(u^k_i)$. As a special case, if input space $U$ is bounded as hyper-rectangle $U$ (as in our experiments), the calculation $C_M$ can be omitted as shown in Corollary 1. We humbly argue that the theoretical analysis is significant in giving theoretical insight into the optimization of NN performance from semantic transformation input (e.g. LiDAR placement in Euclidean space), which may inspire robustness analysis from other semantic transformation perturbations, e.g., rotation, translation, brightness change, etc.

Q: Line 238, Tabs 2 and 3 don't show the correlation between M-SOG and performance.

A: Thanks for your comment. Tabs 2 & 3 show the performance of different LiDAR placements, and Tab 1 shows the M-SOG of LiDAR placements. We add a plot of M-SOG and performance in the rebuttal PDF to illustrate it better.

Q: The detectors used in these experiments are not clearly discussed.

A: Thanks for your comment. We used 4 detectors including PointPillars, BEV method CenterPoint, and BEVFusion-L, spatio-temporal representation method FSTR, as discussed in Appendix B.3. We will include more discussion in the revision.

Q: The authors tried to stuff too many things in one paper.

A: Thanks for your comment. We intend to provide a comprehensive view of our contributions, including the introduction of M-SOG metric, the optimization of LiDAR placements, and the evaluation of placements under both clean and adverse weather conditions. Covering all these aspects is essential to fully demonstrate the value and impact of our method.

Q: What are the points in Fig 4?

A: Thanks for asking. Fig 4 illustrates comparisons to existing sensor placement methods, as discussed in Sec 4.2 (Lines 244 to 251). We compare S-MIG [34] and our M-SOG in terms of the linear consistency between metrics and performance, with M-SOG doing better. Greater linear consistency indicates that metrics can effectively predict perception performance, enhancing downstream optimization tasks.

Q: The authors do not spend a lot of time/space on the adverse aspect.

A: Thanks for the comment. In Sec 4.1, we detailed the types of adverse conditions considered, including severe weather, external disturbances, and sensor failures. In Sec 4.2, we validated the performance of optimized LiDAR placements under adverse conditions, showing consistent outperformance compared to baselines. In Sec 4.3, we investigated how our method informs LiDAR placements to improve robustness against corruptions. As suggested, we will further explain and analyze the adverse aspects during revision.

Q: Why is the placement good for adverse weather?

A: Thanks for asking. The M-SOG utilizes semantic occupancy as prior knowledge, which is invariant to changing conditions, thereby enabling robustness of optimized placement in adverse weather. We also design an ablation study to explore the interplay between the optimization method and perception performance under corruptions in Sec 4.3.

Q: How generalizable are these placements?

A: To clarify, our method is data-driven, and the optimized LiDAR placement aligns with the deployment scenarios. This is common industry practice that deploy specific vehicle configurations for specific country or region. However, our method itself is generalizable for any scenario.

Q: More discussions and insights about the found placement. Any interesting findings?

A: Our experiments revealed several findings:

• LiDAR heights: Increasing the average height of the 4 LiDARs improves performance, likely due to an expanded field of view.
• Variation in heights: A greater difference in LiDAR heights enhances perception, as varied height distributions capture richer features from the sides of objects.
• Uniform distribution: For Seg, the Pyramid placement performs better, as a more spread-out and uniform distribution of 4 LiDARs captures continuous surface.

Dear Reviewer 8eRR,

Thanks for devoting time to this review and providing valuable comments.

---

Q: Although the proposed method investigates LiDAR placement for more tasks under more weather, the core formulation is similar to that in [34] as cited in the paper.

A: Thank you for the insightful comment. We acknowledge the reviewer's observation regarding the similarities between our core formulation and [34]. However, our method M-SOG addresses critical limitations present in S-MIG [34] and makes significant improvements as follows:

• While S-MIG primarily focuses on object binomial occupancy, our M-SOG incorporates semantic differentiation to form a multinomial occupancy distribution, providing a richer and more detailed understanding of the scene. This allows our metric to better capture the true nature of the environment.
• S-MIG constructs occupancy grids with 3D bounding boxes, overlooking the occlusion relationships between objects and the environment. Our M-SOG addresses this critical limitation by leveraging semantic distribution, which only incorporates the reachable surface of objects and environments for LiDAR rays.
• We compare S-MIG and our M-SOG in terms of the linear consistency between metrics and perception performance (Lines 244 to 251) and illustrate our superiority in Fig 4. Greater linear consistency enables the surrogate metric to better predict perception performance. Thus M-SOG enhances applicability to downstream optimization tasks.

Q: Only one type of 16-beam LiDAR sensor is investigated. Sensors with more laser beams (64 or even 128) are becoming low-cost and popular. It is important to evaluate the effect of placement as the point cloud data gets dense.

A: Thanks for your insightful comment. We appreciate the reviewer's comment on the using denser beam LiDARs. In our work, we specifically chose to utilize 16-beam LiDARs to perform experiments, totaling 64 beams with 4 LiDARs, based on the following goals:

• Using low-resolution LiDARs increases the challenge in both object detection and semantic segmentation, making it easier to observe the impact of LiDAR placements on perception.
• LiDAR point clouds become sparse at long distances and the use of multiple LiDARs is specifically to enhance the perception of sparse point clouds. To this end, we use low-resolution LiDARs to simulate the conditions of sparse point clouds from high-resolution LiDARs at long distances. This highlights the role of multiple LiDARs in improving the perception of sparse point clouds, especially for distant objects.

We added experiments on 64-beam LiDARs and found that placement remains important with higher-resolution LiDARs. The results are as follows.

Q: During the companions in Tabs 2 and 3, different refined placements are applied for different tasks. How is the performance if applied a Seg-optimized placement to the Det task? It is impractical to change the placement for different tasks. Is it possible to find an optimal placement for both tasks?

A: Thanks for asking. We show the performance of applying Seg-optimized placement to Det tasks and list the performance of Det-optimized layouts as a comparison. While the performance is close, the Seg-optimized placement performance still lags slightly behind the Det-optimized placement performance.

To find optimal placement for both tasks, a possible solution is to define a weighted objective function for optimizing, such as $F(L_j) = M_{SOG|Seg}(L_j) + \lambda M_{SOG|Det}(L_j)$, where $\lambda$ is the weight which should be designed carefully.

Q: In the dataset comparison part, it is written that there is no adverse weather for nuScenes and Waymo. They may include rainy weather and even more (e.g., Snow). Although in the appendix part, please also have a check on that.

A: Sorry for the oversight and thanks for your comment. We will address this in the revision.

While nuScenes and Waymo include adverse weather, our dataset provides categorized weather data under identical scenarios. This improvement allows for systematic study and fair evaluation of the degradation of perception models in adverse conditions, compared to the naturally distributed weather data in nuScenes and Waymo.

Q: How are the solution spaces for the LiDAR placement? Are the displacements along the Z-axis or even rotation considered during the optimization?

A: Thanks for asking. In our work, Z-axis and rotation are exactly considered in the optimization. The solution space for LiDAR placement in our optimization framework is defined as a 3D cuboid space above the vehicle roof. This space includes both the location coordinates (X, Y, Z) and roll angles for each LiDAR unit. Specific placements and a detailed explanation are discussed in Appendix B.1.

Q: Is there any intuitive interpretation of why refined sensor placement is beneficial to different tasks?

A: Thanks for your question. There are several intuitive reasons why refined sensor placement is beneficial.

• LiDAR heights: Increasing the average height of the 4 LiDARs improves performance, likely due to an expanded field of view.
• Variation in heights: A greater difference in LiDAR heights enhances perception, as varied height distributions capture richer features from the sides of objects.
• Uniform distribution: The pyramid placement performs better in segmentation, as a more spread-out and uniform distribution of 4 LiDARs captures richer surface features.

As discussed, our optimized placements are anticipated to align with these findings in the experiments, leading to better performance.

Dear Reviewer 8eRR,

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

---

In response to your insightful comments, we have made the following revisions to our manuscript:

• We have clarified the core formulation and highlighted the improvements over the work cited in [34], specifically addressing critical limitations in S-MIG.
• We have expanded our experiments to include higher-resolution LiDARs, demonstrating that placement remains important even with denser point clouds.
• We have provided intuitive explanations of how refined sensor placements enhance performance in various tasks, such as improving the field of view and capturing more detailed features.
• We have verified the presence of adverse weather conditions in the Waymo and nuScenes datasets and detailed the key differences between our dataset and others in terms of adverse weather scenarios.

---

We hope these revisions adequately address your concerns.

We look forward to actively participating in the Author-Reviewer Discussion session and welcome any additional feedback you might have on the manuscript or the changes we have made.

---

Once again, we sincerely thank you for your contributions to improving the quality of this work.

Best regards,

The Authors of Submission 1401