We respectfully acknowledge the reviewer SLKs’s valuable comments on our work to improve this paper. We’d be delighted if your concerns could be solved by our reply.

Reply to Strengths:

We appreciate the reviewer recognition of our new setting and various experiments.

Reply to Weaknesses & Questions:

1. “While the authors discuss the differences in problem formulation between StreetSurf and StreetUnveiler, a quantitative comparison of mesh reconstruction is still lacking. StreetSurf uses point clouds as ground truth to calculate Chamfer Distance as a metric. Could a point cloud derived from Structure from Motion (SfM) be used as a ground truth for evaluating StreetUnveiler’s performance? Even relative values across different baselines could provide meaningful insight.”

We really appreciate the reviewer’s concern. This is really helpful to the enhancement of our paper. As mentioned in Section.A.1 of our supplementary PDF. Both SfM points and Lidar points are both used in our reconstruction stage. These points all model the original “before unveiled” scene. Here we call them “before unveiled points”(BUPoints) for simplicity.

The reconstructed mesh of StreetSurf should be strictly aligned with BUPoints. The StreetSurf doesn’t recover the unobserved geometry. In contrast, our “time-reversal inpainting” based StreetUnveiler wishes to recover the unobserved geometry occluded by the removed objects. In this case, since BUPoints don’t include the unobserved geometry, evaluation with Chamfer Distance from BUPoints may not be proper. Consequently, StreetSurf may not be a proper baseline to be compared with, because it doesn’t contain the result of our targeting unobserved geometry.

We respectfully welcome the reviewer to share more existing concerns about this part. We are very willing and eager to solve your concerns further.

2. “Current semantic supervision seems limited to common objects like vehicles, yet street environments also include pedestrians and less common obstacles. Could expanding the range of semantic (instance) labels improve the model’s robustness across diverse urban scenes?”

We appreciate the reviewer for sharing the concern. We provided an example of removing the pedestrian in Figure.18 of our supplementary PDF.

For the expansion of the range of semantic labels, as long as the semantic prediction from a pretrained semantic segmentation method is correct, a finer granularity of semantics may potentially facilitate the removal with finer granularity. As the dataset we used is an autonomous driving-oriented dataset, vehicles and pedestrians are more common; thus, we mainly remove them from our current method. In the future, it would be an interesting direction to study our method on more diverse datasets.

3. “However, real-world street environments include more than just vehicles, and vehicles can often obscure parts of other objects, such as the lower half of a road sign. Would it be more practical to first apply 2DGS to separate scenes based on specific requirements, then proceed with targeted inpainting? This approach could provide more utility by selectively reconstructing essential scene elements.”

We appreciate the reviewer’s valuable suggestions. This is a possible solution for a special targeted inpainting. From the technical side, our method can remove other objects on the road, if a pretrained semantic segmentation model performs well. For example, if we focus on recovering a road sign as an example, we can specially separate the specific vehicle that obscures the sign, and then remove that specific vehicle to proceed with targeted inpainting. To identify this specific vehicle, we can cluster the 2DGS with vehicle semantics into separate clusters. Then, the user will be able to select the specific vehicle cluster that is required for targeted inpainting. As we mentioned, the main issue depends on the performance of the semantic segmentation method.

4. “Current methods, such as Street Gaussians[1] (using 3D box-based scene decomposition) and S3Gaussian[2] (self-supervised scene decomposition), aim to separate scenes into dynamic and static components. Could integrating these approaches enhance StreetUnveiler’s ability to more effectively distinguish between removable objects and essential scene elements? This could potentially improve the precision in identifying and preserving critical components.”

We appreciate the reviewer’s question. We believe the separation of dynamic and static would help the removal of dynamic objects. However, it may not work all the time to distinguish between removable objects and essential scene elements. Examples would be stopping vehicles in Scene 1 of the supplementary video we provided. They are static in videos, but they are not essential elements in the scene. We will cite [1] and [2] and add a discussion about this to further enhance our paper in our revision.

We thanks again for your valuable questions.

Dear Reviewer,

We sincerely thank you once again for the valuable time and effort you have devoted to reviewing our work. Your insightful feedback has been instrumental in helping us improve the quality and clarity of our paper.

In response to your suggestions, we have made the following revisions:

1. We have included both quantitative and qualitative experiments on the geometry of the "before unveiled" scene in Section C.1 of the supplementary PDF. We believe these additions will enrich the depth and comprehensiveness of the paper.

2. We have added a detailed discussion on semantic label supervision in Section D of the supplementary PDF, supported by the newly introduced Figure 13 for better illustration.

Again, we deeply appreciate your constructive comments in order to make this paper better.

Update:

3. We additionally extend the discussion about the static/dynamic decomposition in Section E of the supplementary PDF. (Corresponding to the Reviewer's feedback in Question 2 of the initial review)

We respectfully acknowledge the reviewer orfH’s valuable contribution to our work to improve this paper. We’d be delighted if your concerns could be solved by our reply.

Reply to Strengths:

We appreciate the reviewer’s recognition of semantic-aware design and the enhancement of temporal consistency brought by the time-reversal framework.

Reply to Weaknesses & Questions:

1. “The 2DGS reconstruction and optimization processes might be computationally complex and costly, especially for large-scale street scenes, leading to long training and inference times. The author may consider other faster reconstruction baselines.”

We appreciate the reviewer’s comment. We utilize 2DGS for its accurate geometry property, which is essential for the application of mesh extraction. Meanwhile, we provide a computational analysis in Section B.4 and Table.4 of our supplementary PDF, and it takes 2-3 hours for reconstruction, which may be not unacceptably slow. The technique in some recent work, like InstantSplat, has the potential to further accelerate the reconstruction process into 30 minutes. However, its acceleration requires the accurate initialization of the scene, which would not be so easy for large-scale street scenes.

Technically, our manuscript code was conducted before this 2DGS acceleration upgrade, which improves performance by 30–40% through optimized global memory access in the kernel function. Enhancements like this continuously improve the efficiency of the reconstruction stage in our pipeline.

2. “The method has limited ability to handle dynamic objects. Although some approaches for simple dynamic cases are proposed, more improvements are needed for complex dynamic scenes.”

We appreciate the reviewer’s concerns. As the first to investigate 3D inpainting with single-pass data in long street scenes, rather than simpler small-scale scenes, we focus on static scenarios. While this is already a challenging problem that existing baseline methods cannot address effectively, we provide a “Discussion on Dynamic Object Removal” in Section D of our supplementary PDF. Some explicit dynamic modeling may be introduced in future work to achieve better results in more challenging dynamic cases. However, our current inpainting strategy remains effective for static objects, which are not addressed by dynamic modeling approaches.

3. “In fact, there are many methods that directly predict the masks of dynamic objects, and these are usually more reliable than the masks obtained by optical flow.”

We appreciate the reviewer’s comment. We agree that dynamic object detection would be a powerful tool for handling dynamic object removal if the segmentation results are robust enough. And would potentially be an important component for future work about handling more challenging dynamic cases.

4. “Additionally, in many complex situations (such as vehicles driving sideways at an intersection), can the prior advantage of time-reversal inpainting still be maintained?”

We appreciate the reviewer’s question. In the case of “Driving sideways at an intersection”, the relative positional relationship between the video-capturing vehicle and the driving sideway vehicle would be similar to the video-capturing vehicle crossing the static vehicle. This positional relationship would be similar to the case in our experiment. (Firstly come near to each other and then get far away from each other) However, the actual effect may be affected by the dynamic challenges mentioned above and in Section. D of our supplementary PDF.

We thanks again for your valuable questions.

Dear Reviewer,

We want to kindly follow up regarding our earlier response to your review. We would like to know whether our reply adequately addressed your concerns.

We have revised our paper to include the potential solutions for dynamic object removal through dynamic object detection, provided in Section E of our supplementary PDF. (Corresponding to the feedback from the Questions of the initial review)

Thank you again for your time and valuable insights, which have greatly contributed to improving our work. We deeply appreciate your guidance.

We respectfully acknowledge the reviewer roan’s valuable comments on our work to improve this paper. We’d be delighted if your concerns could be solved by our reply.

Reply to Strengths: We appreciate the reviewer’s recognition of our paper’s novelty and effectiveness in the methodology’s designs.

Reply to Weaknesses & Questions:

1. “The contribution seems somewhat limited. While the two major contributions are effective, they appear to be incremental rather than groundbreaking. The work could serve as a baseline, but not much more that.”

We appreciate the reviewer’s comment. To the best of our knowledge, we are the first work that extends the 3D inpainting task from simple scenes to large-scale street environments with one-pass data, successfully reconstructing an empty street, which provides temporal consistency videos, good novel-view synthesis results, and good geometry. Our approach also provides a verified solution to temporal consistency over a long sequence, which is relatively understudied, as highlighted by reviewer SpDA. The results of our method can be readily integrated into a simulator for autonomous driving or a game engine.

2. “The performance metrics are not entirely convincing. Table 1 doesn't demonstrate significant advantages over other baselines. The video in the supplementary materials appears noticeably inferior to other NVS methods on Waymo (e.g., EmerNeRF, StreetSurf, StreetGaussian).”

We appreciate the reviewer’s comment. In Table 1, our FID is only lower than the baseline SDXL, yet SDXL doesn’t maintain temporal consistency between different video frames. Further, SDXL-based per image inpainting solution cannot achieve good geometry and does not support novel view synthesis.

The task settings of the other NVS methods mentioned by reviewer(EmerNeRF, StreetSurf, StreetGaussian) are not the same as those in our task. Our paper focuses on a challenging unsupervised problem with one-pass data of a street scene, where no ground truth data exists for the "after unveiled" street. It’s not the same between “reconstructing the scene with ground truth data” and “reconstructing the background that’s completely blocked by static objects”. The works the reviewer mentioned here are mainly focusing on reconstructing the original fully-observed scene. Thus, these methods cannot be applied to our current settings. That's why we can not compare our method with them.

3. “The benchmark methodology raises some questions (please correct me if I'm mistaken, as I haven't encountered this benchmark setting before). What are the ground truth images used for LPIPS?”

We appreciate the reviewer’s comment. Our task setting doesn’t have ground truth images, and that’s why this task is very challenging.

Following well-established baselines(SpIN-NeRF, and [1,2,3]), we evaluate with LPIPS and FID. Each frame of the output video is paired with the corresponding frame from the original training video to compute the LPIPS. We use the image collections of the output video and original training video to compute FID. LPIPS directly compares the visual similarity between two images, focusing on perceptual quality differences. FID measures the similarity between the distributions of generated images and real images.

4. “If all rendered images lack vehicles while the ground truth images typically include them, is this an appropriate measurement for LPIPS/FID?”

Actually, there are also some other long-standing tasks without ground truth for evaluation, like image/video generation, and some representative works in this direction, including StyleGan and DDPM, also use FID for evaluation. Their experiment is also an appreciated measurement when the ground truth images are lacking.

[1] Silvan Weder, Guillermo Garcia-Hernando, Aron Monszpart, Marc Pollefeys, Gabriel Brostow, Michael Firman, and Sara Vicente. Removing objects from neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

[2] Zhiheng Liu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue, Yu Liu, Yujun Shen, and Yang Cao. Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior. arXiv preprint arXiv:2404.11613, 2024.

[3]Chieh Hubert, Changil Kim, Jia-Bin Huang, Qinbo Li, Chih-Yao Ma, Johannes Kopf, Ming-Hsuan Yang, and Hung-Yu Tseng. Taming latent diffusion model for neural radiance field inpainting. In European Conference on Computer Vision (ECCV), 2024a.

We respectfully acknowledge the reviewer SpDA’s valuable contribution to our work to improve this paper. We’d be delighted if your concerns could be solved by our reply.

Reply to Strengths:

We appreciate the reviewer’s recognition of our paper’s methodology tailored for driving videos and our results.

Reply to Weaknesses & Questions:

1. “I wonder if a simpler pipeline could achieve similar goals - for example, establishing object correspondences across the video frames and then applying LeftRefill”

We appreciate the reviewer’s question, while the 3D representation obtained by our method has some more applications. For example, an empty street’s geometry can be extracted by our final 3D representation, as placed as an example in Figure 16 and Figure 17 of our supplementary PDF, and such an empty scene can be integrated into a simulator for autonomous driving or a game engine. Further, Novel view synthesis is also a function that a simpler 2D video pipeline won’t achieve.

2. “I also note that LeftRefill is missing from the quantitative comparisons.”

We thank the reviewer for pointing this out. We additionally conduct a quantitative comparison of LeftRefill. Since LeftRefill requires an image as a reference, LeftRefill can’t be naturally run as unconditional inpainting methods like LAMA and SDXL in Table 1.

We adapt LeftRefill with the 10th future frame as a condition and use the mask obtained after our reconstruction stage.

We observe that the evaluated LPIPS and FID of LeftRefill are all lower than ours on both Waymo and Pandaset.

This naive reverse inpainting with LeftRefill will cause the red mask region in Figure 4 of our paper to be inpainted with the future frame as a reference, but some regions in the red mask are not visible in the future frame. This will lead to the wrong content generated by LeftRefill. We will revise this paper with an additional discussion about this with both qualitative and quantitative comparisons, which we believe will further enhance this paper.

We thanks again for your valuable questions.

We thank the reviewer roan for sharing the concerns.

We hope our reply can address your questions:

1. Regarding Contribution: The key idea of building 3D scene representations to render unoccluded views by leveraging multi-view observations has been well explored in prior work. Dozens of methods have demonstrated similar capabilities in scene decomposition and object removal.

We appreciate the reviewer for sharing the concern. First of all, we would like to point out that the reviewer misunderstood our work. The goals of your mentioned paper and that of our papers are also different. We elaborate on each work about its setting you mentioned here:

(1) “Neural Scene Graphs for Dynamic Scenes”: This paper decomposes dynamic scenes with neural rendering. Such work cannot remove the static contents (like static cars in the scene)

(2) “UniSim: A Neural Closed-Loop Sensor Simulator”: This paper composites the static background and dynamic actors in the scene to simulate data from new viewpoints. It is worth noting that the non-essential static elements, like stopping vehicles, are also part of the static background in their setting. This paper cannot remove these non-essential static elements.

(3) “NeuRAD: Neural Rendering for Autonomous Driving”: This paper aims to design a robust novel view synthesis method tailored to dynamic AD data. It cannot recover a street without the stopping cars/pedestrians if stopping cars/pedestrians exist in the scene.

(4) “EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision”: This paper decomposes dynamic scenes into static and dynamic components. (Note: The non-essential elements we wish to remove are not disentangled from its static components.) It still cannot remove those non-essential static elements.

(5) “Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting”: This paper models the dynamic urban streets for autonomous driving scenes. It is worth noting that the non-essential elements we wish to remove are not disentangled from their static components in this paper. Consequently, the stopping cars/pedestrians cannot be removed in this paper.

In summary, the goals of the papers you mentioned are totally different from ours. The works mentioned above mainly focus on how to decompose the dynamic and static scenes, while we aim to remove non-essential static elements from the scene with single-pass data.

We would like to restate that, to the best of our knowledge, we are the first to scale up the static object removal tasks to the long sequences data.

2. “Regarding quality: for visible regions: The re-rendered results show noticeably lower quality compared to optimization-based approaches, exhibiting more blur than the methods mentioned above (e.g streetgaussian). For occluded regions: There are clear artifacts in the inpainting results, For example, in scene2/data_vs_ours.mp4 at timestamp 00:02, where the removal of the SUV on the left leaves visible distortions. Similar issues are present throughout other example videos.”

Thank you for sharing your concern. We have elaborated on the difference in terms of the problem settings with methods like StreetGaussian.

The task setting of optimization-based NVS methods (e.g streetgaussian) focuses on modeling both static and dynamic elements of the scene with ground-truth data. While our method aims to recover the completely occluded regions(like the ground under the vehicle), it lacks the corresponding ground-truth data. The task settings are different for our method and the other methods you mentioned. Thus, it’s unfair and improper to compare methods in two different settings. Thus, it is improper to mention that “The re-rendered results show noticeably lower quality compared to optimization-based approaches, exhibiting more blur than the methods mentioned above (e.g streetgaussian)” in your comments. Further, our method is the first work in the single-pass-based object removal in long-trajectory videos, and there is still some room for our method to improve in the future.

our core contribution is the time-reversal inpainting pipeline

This claim could kill the paper

The regions that can indeed benefit from "time-reversal inpainting" are the regions that are visible in other frames, which can be inpainted in streetgaussian (and any other methods I mentioned)

Streetgaussian, without any of these heuristics tricks, produce way better results in these regions.

I would say the contribution lies in incorporating 2D inpainting techniques to handle non-essential object removal in 3D scene representations. But both the methodology and results appear too preliminary.

The orthogonal combination would be “combining StreetGaussian with our time-reversal inpainting pipeline

It is the authors' responsibility to implement solid baselines for comparison and demonstrate competitive results by building upon the best available methods. While minor differences would be acceptable, the visual quality gap in the current results appears quite significant compared to the state-of-the-art.