OmniRe: Omni Urban Scene Reconstruction

Q2. GS-based methods generally perform well in scenarios with static environments or low vehicle speeds, as demonstrated by most of the demos on the project page. However, I am curious about the reconstruction performance of this approach in situations where the ego vehicle is moving at higher speeds.

Thank you for raising this concern. OmniRe does perform equally well for high-speed scenarios. We have included some demos where the ego vehicle is moving at higher speeds (e.g., Pandaset scene-012, Argoverse scene-037 in the anonymous page).

To fully address this concern, we conducted additional tests of OmniRe, StreetGS, PVG, and DeformGS, on three scenes (seg119252…, seg122514…, seg152664…) with moderately high vehicle speeds. The results are detailed below, showcasing OmniRe's superior performance in full scene reconstruction and human modeling, while performing on par with StreetGS in vehicle modeling.

Furthermore, we tested them on three extremely high-speed driving scenes (seg109239…, seg113924…, seg118396) from Waymo. Highways typically lack non-rigid classes (e.g., humans), making human reconstruction metrics inapplicable. Since non-rigid modeling is not required in these scenarios, OmniRe and StreetGS demonstrate comparable performance. We provide additional videos in our anonymous page (Section: OmniRe on Challenging Scenes). We will greatly appreciate it if reviewers get a chance to check these additional results of highly challenging scenes.

Q3. I wonder about the computational cost of reconstructing a complete segment in the Waymo dataset, as the entire pipeline seems a bit complex.

Our pipeline is cleanly organized as shown in the supplement material. We have dedicated significant engineering efforts to ensure reconstruction efficiency. Reconstructing a complete Waymo segment with three front-facing cameras takes 1-1.5 hours, depending on the scene's dynamic complexity. The GPU memory usage is 6GB when sensor data is cached on the CPU and 15GB when cached on the GPU. Tests are performed on a single 4090 GPU.

Q4. Why does it seem that the reconstruction quality of NuPlan is significantly worse than that of other datasets?

The lower quality on NuPlan is due to severe image distortions of NuPlan’s raw data. Unlike NeRF-based (ray-based rendering) methods, which can account for distortion parameters to correct ray directions, the whole-image rasterization approach of 3DGS does not natively support such undistortion corrections. To the best of our knowledge, no GS-based method currently incorporates distortion parameters for whole-image rasterization.

Fortunately, recent works like 3D Gaussian Ray-Tracing [a] adapt Gaussian to ray-based rendering, making distortion-aware rendering possible. We will integrate this into our framework as a future work, to enhance distortion handling across datasets and improve the system's overall robustness.

[a] 3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes. SIGGRAPH2024

We sincerely thank the reviewer for the time and effort invested in reviewing our manuscript. We also appreciate the recognition of our work in advancing driving reconstruction and simulation systems, as well as the engineering effort involved. Your comments are very helpful, our responses are detailed below.

Q1. As mentioned by the authors in the limitations section, there are still two key shortcomings: 1. The lack of lighting modeling results in unnatural object insertions. 2. The synthesis of new viewpoints is constrained to the original trajectory, limiting the approach from achieving fully free-trajectory digital reconstruction.

Thank you for your thoughtful comments on the limitations discussed in our paper. While addressing these challenges is non-trivial and slightly beyond the scope of our current work, we would like to propose a few potential directions to address them.

For light modeling, potential directions include: 1) incorporate BRDF model to learn reflectance, material properties, and utilize environmental maps for illumination modeling; 2) post-processing rendered images with harmonization techniques, such as rendering feature maps and training a network to convert these maps into photorealistic RGB images. 3) using video diffusion models to refine the outputs, e.g. localized editing with diffusion models [a].

For novel view rendering, the current quality meets the requirements for typical driving simulations (e.g., lane shifts or varying vehicle heights). However, limitations arise during free navigation with viewpoints far outside the training space. For future work, we believe methods using generative models like [b] could enhance reconstruction by inferring missing details and completing unseen regions.

[a] Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets [b] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion. SIGGRAPH2024

Q2. Performance in Specific Urban Scenes: For specialized scenarios, such as highways (with fast-moving vehicles), nighttime environments, and adverse weather conditions, does OmniRe maintain high reconstruction quality under these challenging conditions?

The answer is yes-OmniRe maintains high reconstruction quality under these challenging conditions. We tested OmniRe, StreetGS, PVG, and DeformGS on various specialized scenes, with detailed results provided below (videos available in the 'OmniRe on Challenging Scenes' of the anonymous page).

a. Nighttime environments: We tested three nighttime scenes (seg129008…, seg102261…, seg128560…) from Waymo and observed that OmniRe outperformed the compared methods.

b. Adverse weather conditions: We tested seven scenes under varying weather conditions: 1) rainy: seg113555…, seg109277…, seg141339…; 2) foggy: seg161022…, seg172163…; 3) cloudy: seg144275…, seg157956…. OmniRe handled these challenging scenarios robustly, maintaining high reconstruction fidelity across all weather conditions.

c. Highway scenes: We tested three highway scenes from the Waymo dataset (seg109239…, seg113924…, seg118396). Highways typically lack non-rigid classes (e.g., humans), making human reconstruction metrics inapplicable. As non-rigid modeling was not required in these scenes, OmniRe and StreetGS demonstrated similar performance.

We sincerely thank the reviewer for their time and effort in reviewing our paper. We greatly appreciate the positive feedback and are eager to address the points raised. The reviewer's insights are valuable for improving our work. Below, we provide our responses.

Q1. Handling Occlusions and Complex Dynamics: OmniRe addresses in-the-wild challenges, yet the performance might be limited by severe occlusions and overlapping actors in complex urban scenes. Further refinement or integration of advanced occlusion handling techniques could enhance reconstruction fidelity.

Reconstructing scenes with extreme dynamic occlusions pose significant challenges. However, OmniRe is designed to tackle these challenges and performs well even in scenes with extreme occlusions. We tested OmniRe, StreetGS, PVG, and DeformGS on three extremely occluded scenes from Waymo (seg112520…, seg104859…, seg152664…), including scenarios with a large crowd of people simultaneously crossing a street. OmniRe demonstrated strong performance:

OmniRe achieved a +2.14 PSNR improvement on scene reconstruction and a +2.96 PSNR improvement on human reconstruction. We provide additional videos in our anonymous page (Section: OmniRe on Challenging Scenes). We will greatly appreciate it if reviewers get a chance to check these additional results of highly challenging scenes.

Q3. Computational Complexity

& Q4. Challenges with Real-Time Adaptability

Training Efficiency: Although our system integrates multiple Gaussian representations for full category reconstruction and jointly optimizes the appearance and poses of all objects (dozens or even over a hundred) along with the background, we made substantial engineering efforts for the whole system to ensure its efficiency and robustness, including:

• Parallelizing the deformation and transformation of each object's Gaussians using category-specific Gaussian containers. These containers handle all objects' Gaussians and compute deformations in parallel. For example, the vehicle class container processes all vehicles’ Gaussians simultaneously.
• Sharing the weights and gradients of the deformation network for non-rigid deformable nodes and using a shared network with multiple instance embeddings to distinguish them.

Rendering Efficiency: The rendering of OmniRe operates in real-time (>24 fps), meeting most simulation requirements. To address the concern regarding its real-time adaptability, we conducted tests under various settings with increasing dynamic complexity:

• Background only: 170 fps

• Background + Vehicle (Rigid Nodes): 108 fps

• Background + Vehicle + SMPL-GS (Non-Rigid Nodes): 57 fps

• Background + Vehicle + Deformable-GS (Non-Rigid Nodes): 68 fps

• Background + Vehicle + Deformable-GS + SMPL-GS: 45 fps

  (Experiments conducted on a single 4090 GPU)

Q5. I wonder for items like causality and new synthesis if an approach more configurable could take place now that they have separated the pedestrians from the road. Thinking of something like CityLifeSim [a], where the data is later attached to an engine like Carla or airsim.

The idea behind CityLifeSim [a] is highly inspiring and crucial for an advancing simulation system, especially in aspects like modeling causality and designing configurable actor behaviors. We will explore ways to apply its ideas to our 3DGS-based framework.

The waypoint-based POI (points of interest) for actor interaction rule design presented in CityLifeSim [a] is adaptable to our framework with some additional work, such as labeling valid activity areas for each object. However, the primary challenge lies in the animation of humans, since we use a different modeling approach (SMPL), animating humans in our framework requires additional attention to motion naturalness. Ensuring realistic motion through effective SMPL parameter control is a non-trivial task. Techniques like text-driven motion [b] models could enhance motion naturalness and contribute to the design of more robust interaction rules.

Thank you again for highlighting this important work. We have included a discussion of CityLifeSim in the revised manuscript line 511-513.

[a] CityLifeSim: A High-Fidelity Pedestrian and Vehicle Simulation with Complex Behaviors. ICIR2022 [b] Generating Diverse and Natural 3D Human Motions from Text. CVPR2022

We sincerely thank the reviewer for the time and effort reviewing. We are encouraged by the reviewer's recognition of the value our method brings to dynamic scene modeling, human-level reconstruction and simulation, and SOTA performance. Our replies are stated below.

There are several limitations in OmniRe approach, which are correctly identified in the paper too.

Q1. Lighting Effects: OmniRe doesn’t model lighting variations explicitly. This can lead to visual inconsistencies when combining scene elements with differing lighting conditions, which may reduce realism in certain simulations. Addressing this would require additional modeling of lighting dynamics.

Thanks for pointing out its importance and for recognizing that we have acknowledged it in the limitation section. Modeling lighting effects to enhance simulation realism is crucial for achieving more convincing and harmonious results. Solving it is non-trivial and requires dedicated efforts. For example, LightSim [a] is a standalone paper that focuses specifically on this issue.

While this is slightly beyond the scope of our current work, we would like to discuss a few potential directions for light effect modeling based on our current framework: 1) incorporate BRDF model to learn reflectance, material properties, and utilize environmental maps for illumination modeling; 2) post-processing rendered images with harmonization techniques, such as rendering feature maps and training a network to convert these maps into photorealistic RGB images. 3) using video diffusion models to refine the outputs, e.g. localized editing with diffusion models [b].

[a] LightSim: Neural lighting simulation for urban scenes. NeurIPS2023. [b] Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets.

Q2. Novel View Synthesis Limitations: OmniRe’s per-scene optimization approach struggles to generate satisfactory results when the camera view deviates significantly from the training trajectories. This could be a limitation for scenarios requiring a wide range of viewing angles, such as free navigation through the reconstructed scenes. The authors suggest incorporating data-driven priors or generative models as future work to address this.

Driving scenes, which typically follow a forward-moving trajectory, often involve sparser observations compared to object-centric scenes. For driving simulation, simulated viewpoints are typically constrained to those on the road, with acceptable deviation. As a result, novel view rendering maintains high quality, as shown in the top-right novel-view navigation demo on the anonymous page.

While the current novel view quality meets the requirements for typical driving simulations (e.g., lane shifts or various vehicle heights), limitations arise for free navigation with viewpoints far outside the training space. For future work, we believe methods using generative models like [c] could enhance reconstruction by inferring missing details and completing unseen regions.

[c] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion. SIGGRAPH2024

Dear reviewer,

We greatly appreciate the time and effort you have invested in reviewing our work! We hope the additional experiments on OmniRe’s real-time rendering and simulation capabilities address concerns regarding computational complexity and real-time performance. This work involved substantial engineering efforts to ensure the system's efficiency and robustness.

We are also greatly inspired by your suggestion regarding CityLifeSim to further explore the possible extension of our work. Based on our current reconstruction framework, we will incorporate elements like configurable actor behaviors and interaction rules to develop a more comprehensive simulator in future work.

We understand how busy you must be, especially as we approach the end of the discussion period. We will greatly appreciate an increased score if we successfully address your concerns. Otherwise, we are committed to provide additional results to answer any further questions.

Dear Reviewer,

Thank you again for helping us improve the paper. As the rebuttal period is coming to an end, we would like to check if our responses have addressed your concerns. If so, we would greatly appreciate it if you could consider raising the score accordingly. Otherwise, we would be grateful for any additional comments or questions, and we would be happy to include further experiments and results. Thanks!