Memorize What Matters: Emergent Scene Decomposition from Multitraverse

We sincerely appreciate your constructive feedback and helpful suggestions. Please find our detailed response below.

Q1. Multiple rounds

We have added one additional round of segmentation and mapping by incorporating the emerged masks into COLMAP to remove transient objects during Gaussian initialization. This resulted in improved rendering scores, with PSNR increasing from 23.11 to 23.26 and SSIM from 0.8299 to 0.8372 in 1000 images at one location. We will include more experiments using multiple rounds. We appreciate your insightful comments.

Q2. EmerNeRF

We would like to emphasize that our method is fundamentally different from EmerNeRF, not just in using multitraversal data and 3DGS. Although both our method and EmerNeRF address self-supervised scene decomposition, the key ideas are entirely different: EmerNeRF learns static-dynamic separation by effective parameterization of corresponding fields and leverages scene flow as an inductive bias, whereas our method leverages multitraversal consensus as a self-supervised signal to decompose transient objects from the permanent environment. As a result, our method can decompose not only dynamic objects but also static yet movable objects, e.g., parked cars. In contrast, EmerNeRF can only decompose moving objects. Besides, our method only needs camera input while EmerNeRF requires LiDAR. Our insight is to leverage more camera observations to boost the reconstruction performance. In summary, compared to EmerNeRF, our method has three differences:

• Our method introduces the conceptually novel idea of multitraversal consensus-based scene decomposition.
• Our method provides a more powerful scene decomposition approach, capable of decomposing both moving and static yet movable objects.
• Our method adopts a more cost-effective sensor setup.

We have added quantitative and qualitative results of EmerNeRF. We found that EmerNeRF cannot remove static cars because it requires motion to decompose dynamic parts. Additionally, the decomposition performance is not ideal, likely due to the monocular camera input. The segmentation IoU is 7.3%, which is significantly lower than our method (>40%). Visualizations are shown in Figures 6 and 7 in the PDF under the global rebuttal.

We hope our clarifications address your concerns regarding the novelty of our work. As agreed upon by the other two reviewers, our method is an innovative approach with key insights.

Q3. Object discovery

We agree that adding comparisons to object discovery methods would make our experiments more comprehensive. We have included results from FOUND [1] which is a SOTA unsupervised object discovery method and found them to be much worse than our method, with an IoU of only 12.77%. Some qualitative examples are shown in Figure 8 in the PDF under the global rebuttal. Another notable advantage of our method compared to object discovery methods is that it not only discovers objects in 2D but also obtains a 3D representation of the static environment. We will add more results and an expanded literature survey.

[1] Unsupervised object localization: Observing the background to discover objects. CVPR 2023.

Q4. Segmentation baselines

We agree that adding more baselines using multiple traversals would make our experiments more comprehensive. We have added another baseline method: we directly matched DINO features across traversals and identified patches with few matching counterparts in other traversals as transient. The results were less convicing, with an IoU of 10.17%. Visualizations are shown in Figure 9 in the PDF under the global rebuttal.

We would like to emphasize that the key difference between our method and existing methods is that we lift 2D to 3D, which facilitates the identification of 2D objects through 3D representation learning. As a result, our method not only segments objects in 2D with high quality but also obtains a 3D environment map. We will include more baseline results and discussions.

Q5. Multiview stereo and depth metrics

Our method is a 3DGS-based approach that integrates both geometric and photometric information. It can be used not only for depth estimation but also for various tasks such as rendering and segmentation. Therefore, our method is more versatile compared to monocular or multiview depth estimation methods.

Additionally, our method is a camera-only solution, whereas learning-based depth estimation methods require LiDAR-supervised training. Hence, our method requires a more cost-effective sensor setup compared to depth estimation methods.

We have also added a SOTA multiview stereo baseline, DUSt3R [1]. We agree that including depth metrics makes the evaluations more comprehensive. The mean depth RMSE of our method is 7.376m, while DepthAnything is 13.562m, and DUSt3R is 13.37m (background regions). Some qualitative examples of DUSt3R are shown in Figure 10 in the PDF under the global rebuttal. These results demonstrate that our method remains the best without the need for LiDAR sensors. We will add the mutliview stereo results and depth metrics in the final version of our paper. Note that our method and depth estimation can be complementary: our method can provide pseudo ground truth for depth estimation, while depth estimation can aid in the initialization stage of our method. Future work includes leveraging multitraversal to enhance depth estimation methods and utilizing depth estimation for efficient 3DGS initialization.

[1] DUSt3R: Geometric 3D Vision Made Easy. CVPR 2024

Q6. Table 3

Table 3 does not evaluate pixels corresponding to transient objects, as we do not have ground truth background pixels in regions occluded by transient objects. Therefore, the observed gap is not too large. However, our method performs significantly better in occluded regions, as shown in Figures 6 and XIV.

We are grateful for your insightful feedback and suggestions. Please review our detailed response below.

Q1. The end-to-end process for 3DGM appears to be quite complex, with multiple stages involved. However, the paper fails to provide a lucid explanation of the nitty-gritty details involved in implementing the algorithms. Without the availability of the accompanying source code, it would be an uphill task for others to replicate the results or build upon this work effectively.

We would like to emphasize that, although our method has multiple stages, each module is easy to deploy and computationally efficient. As discussed in the paper, using a feature map with a resolution of 110×180 requires only 2,000 iterations to achieve an IoU score exceeding 40%, taking approximately 8 minutes on a single NVIDIA RTX 3090 GPU for 1,000 images from 10 traversals of a location. Future work will include investigating more efficient initialization methods. A detailed workflow of the overall method is provided in Appendix A. To support reproducibility and further research, we will release our source code and dataset upon acceptance of our work.

Q2. Firstly, although this is a self-supervised approach, in practical applications, multiple passes along the same road are still required for reconstruction. Could some prior information be incorporated to enable the model to infer the background and complete the task with fewer traversals?

We agree that incorporating prior information can further enhance the efficiency of the proposed method. For example, if past traversals with both LiDAR and camera data are available, the 3D reconstruction could benefit significantly from such prior information. In fact, our method could serve as a strong camera-only baseline for constructing such a scene prior, facilitating scene decomposition and reconstruction in future traversals of the same location. We leave the exploration of prior information for future work, as it requires a complete reformulation of the problem, which is non-trivial and beyond the scope of this work. Future research opportunities include developing algorithms that seamlessly integrate prior data, exploring the impact of different types of prior information on reconstruction quality, and creating frameworks that can dynamically adapt to changes in new traversals. We will include more discussions on this in the final version of our paper.

In addition, we would like to emphasize that the setup of multiple passes of the same location is quite feasible, as vehicles typically operate within the same spatial region. As shown in a recent publication at CVPR 2024 [1], it is possible to obtain hundreds of traversal data for the same location within several days.

Q3. Secondly, the methods based on 3D reconstruction have high data quality requirements, and the 3DGS-based methods need a reasonably accurate initial pose estimation (by COLMAP), which further exacerbates the demand for high-quality data.

Our method only requires monocular RGB images with a resolution of around 900x600, which can be easily obtained and is the most cost-effective data collection method. No other sensor data, such as LiDAR point clouds, GPS, or IMU data, are needed. Additionally, data collection by multiple traversals of the same location is highly feasible, as demonstrated by existing datasets collected by either academic labs or industry companies, such as Ithaca365, nuPlan, and MARS [1].

In addition, we have found that multitraversal RGB images can facilitate COLMAP initialization, producing very accurate camera poses. Based on our empirical studies, COLMAP initialization requires only 2 or 3 traversals with only monocular RGB images to significantly improve the success rate compared to single-traversal scenarios.

[1] Li, Y., Li, Z., Chen, N., Gong, M., Lyu, Z., Wang, Z., Jiang, P. and Feng, C., 2024. Multiagent Multitraversal Multimodal Self-Driving: Open MARS Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22041-22051).

We appreciate your valuable feedback and insightful suggestions. Below is our detailed response.

Q1: Assumptions on Environmental Stability: The method strictly assumes a stable environment without major geometry change under consistent illumination and weather.

We agree with you that relaxing the assumption of environmental stability can further enhance the impact and generality of our method. We have made two efforts to address your concern:

• We have tested our method under more challenging illumination and weather conditions, such as night, foggy, and rainy scenarios. Specifically, we added images from one additional traversal under adverse conditions to the multitraversal collection. Surprisingly, our method still performed well in unsupervised segmentation, even under night, foggy, and rainy conditions, as shown in Figure 1-3 in the PDF under the global rebuttal. This demonstrates that our method is robust against inconsistent illumination and weather conditions, including night, fog, and rain.
• We have incorporated a learnable traversal embedding into the 3DGS. This effectively models the appearance changes across traversals. Due to the limited time for the rebuttal, we present preliminary results in Figure 4 in the PDF under the global rebuttal. From the top left to the bottom right, it shows the rendering results by interpolation between two traversal embeddings.

We will include these experiments in the final version of our paper. Additionally, we would like to emphasize that 3DGS for self-driving with multiple traversals is an unexplored research topic, especially considering significant environmental changes (adverse weather). There are many open questions in this domain, and we believe our dataset and method can serve as a benchmark and baseline to inspire further research on this topic.

Q2: Lack of Failure Case Study: Even though the paper discusses some failure cases regarding shadow, occlusion, reflection, etc. It's still interesting to see the influence of weather and large illumination changes on this approach and if increasing the number of traversals can tackle the issue.

We agree with you that discussing the influence of weather and large illumination changes would make our experiments more comprehensive. We have conducted additional experiments with traversals collected under challenging weather and illumination conditions. As discussed above, we find that our method demonstrates good robustness against diverse conditions. Note that these experiments use 11 traversals, and decreasing the number of traversals will degrade segmentation performance, similar to the findings in Ablation Study C.4 in our paper (Ablation Study on Number of Traversals: Visualization and Discussion).

In addition, we have also added one traversal with snow on the road. We observed that snow on the road can also be segmented, as it appears only in a single traversal, as shown in Figure 5 in the PDF under the global rebuttal. We do not consider this a "failure case" since snow is transient. We will include these results in the final version of our paper.

Q3: Lack of Comparative Baselines Regarding Env Reconstruction and Rendering: Although the paper provides a solid evaluation of all tasks, it could benefit from more comparative analysis with state-of-the-art urban scene reconstruction methods.

We agree that adding additional comparisons to urban scene reconstruction methods would enhance our experiments. There are several fundamental differences between our method and existing ones such as EmerNeRF, HUGS, and NeuRad. We will include these comparisons in the final version of our paper to provide a broader discussion on the effect of multitraversals and sensor types.

• Input and Output: These works take a single-traversal video as input, which limits their ability to reconstruct static environments occluded by static yet transient objects, such as parked cars. In contrast, our method, based on multi-traversal input, can reconstruct a static environment without any movable objects (including both dynamic and static vehicles).

• Sensor Requirements: These methods require LiDAR point cloud data as input. In contrast, our method only uses RGB images, using a more cost-effective and portable sensor setup.

• Supervision: HUGS and NeuRad require 3D bounding boxes as input, making them not fully self-supervised methods. Our method, however, does not rely on such external supervision.

We have added quantitative and qualitative results of EmerNeRF, a self-supervised method, for a fair comparison. We ran the original codebase of EmerNeRF with both LiDAR and camera inputs on each traversal in our dataset.

• Actor Removal: EmerNeRF cannot remove static cars because it models static cars as backgrounds and only segments objects with motion. Besides, the decomposition performance is not ideal, likely due to the monocular camera input. The segmentation IoU is 7.3%, which is significantly lower than our method.
• Adverse Weather: The performance under adverse weather conditions is unsatisfactory, as shown in the rainy day example in Figure 4 in the PDF under the “global” response.
• Rendering Quality: The rendering quality is also inferior to our method, as shown in Figure 6 and Figure 7 in the PDF under the global rebuttal. We will include more quantitative results in the final version of our paper.

Q4: More Details on Training and Rendering Procedure.

We agree that providing more details can be valuable for further work in this field. Most of our method follows the hyperparameters of 3DGS. Empirically, we found that the feature rendering loss function plays a crucial role in scene decomposition: the KL divergence loss function performs much better than the L1 loss. We will report more ablation studies in the final version of our paper and release our code to support further research.