FreeVS: Generative View Synthesis on Free Driving Trajectory

W2. About the temporal consistency of generated results and performance on dynamic objects

Our proposed generation pipeline can naturally leverage temporal information and generate temporally consistent results, and handle dynamic objects effectively.

Ensuring temporal consistency. Being generated from accurate 3D points, pseudo-images are always consistent with real scenes, which ensures the fundamental temporal consistency performance of FreeVS. Moreover, the pseudo-image representation can naturally offer 3D priors with high temporal consistency through accumulating 3D points across data frames (as described in lines 171-173 and lines 344-345). Finally, we formulate the proposed pipeline as a video generation pipeline, which also strengthens the temporal consistency of generated videos.

Handling dynamic objects. Dynamic objects will not cause trouble in generating pseudo-images. For each dynamic object, we accumulate its 3D points along its moving trajectory and project its 3D points onto each pseudo-image based on its current position. Therefore nearly no artifacts will be introduced by dynamic objects into pseudo-images. We think the easy and effective handling of dynamic objects is a unique advantage of the proposed pseudo-image representation.

With the demo video submitted as supplementary material and Figure A in the appendix, we already show that FreeVS has strong performance on both temporal consistency and dynamic object handling.

To further prove this, we provide several long video sequences generated by FreeVS on an anonymized page, as posted at the beginning of this comment (https://anonymize289.github.io/anonymized289.github.io/). These videos showcase scenes with dynamic objects, clearly demonstrating FreeVS's ability to maintain temporal consistency and handle dynamic objects even with changes in trajectory and scene content. FreeVS outperforms previous NVS methods and even many video generation works on temporal consistency and dynamic object handling.

W3: About generating views far from captured views.

Yes, the pseudo-image representation surely has some limitations. For example, the pseudo-image representation cannot handle non-rigid dynamic objects (like pedestrians and cyclists) as well as rigid dynamic objects since their 3D points are hard to be correctly accumulated across frames.

However, thanks to the generalization ability of video generation models, views of pedestrians and cyclists synthesized by FreeVS are still satisfying as shown in the posted videos in response to W2, and Figure 5 in the paper.

FreeVS also cannot perfectly recover scene contents when too few observations are provided. However, FreeVS can still outperform previous NVS methods in such situations, as shown in Figure 4. We have added discussions on the limitation of the pseudo image representation in Sec. A.8 in the revised version of our paper.

As for synthesizing views far from captured views, as reported in Table 2, FreeVS has a greater performance advantage on views far from captured views. Therefore synthesizing far views is clearly not a relative weakness of the proposed FreeVS. Indeed, pseudo-images can not be perfectly generated on views too far from the captured views when most scene contents in the novel views are not observed. However, such views are usually not considered in the NVS task since the NVS task focuses on generating novel views of the observed scene contents.

As shown in the demo videos, FreeVS can synthesize high-quality images when laterally moving the ego vehicle for up to 5 meters, which is enough for simulating normal driving behaviors like lane changing. Although FreeVS cannot synthesize viewpoints at arbitrary distances from the captured views, its capable range of synthesizing high-quality views still far exceeds that of current approaches.

Thank you for your valuable concerns and suggestions.

First of all, we would like to provide several long video sequences generated by FreeVS on an anonymized page as supplements to the submitted video demo: https://anonymize289.github.io/anonymized289.github.io/

Below are our responses to your comments:

W1: About the proposed method seems too normal

As discussed and acknowledged by many previous works [1-9], novel view synthesis (NVS) in driving scenes requires solutions significantly different from NVS methods for object-centric scenes.

Since no specific works are referenced in the review comment, here we generally list the major differences between NVS for object-centric scenes and NVS for driving scenes:

1. Narrow, static scenes vs. open, dynamic scenes. Object-centric scenes usually only involve a single, static object in a narrow scene, while driving scenes are unbounded scenes with rich dynamic objects.
2. Dense, rich viewpoints vs. sparse, trajectory-bounded viewpoints. In object-centric scenarios, the camera views are distributed surrounding an object. In driving scenes, camera views are sparsely and homogeneously distributed along the recorded trajectory. This makes training a model to synthesize views beyond the recorded trajectories a key challenge. Tackling this challenge is one of the main goals of our paper.
3. Synthesizing monocular images vs. synthesizing multi-view videos. Views in driving scenes are usually required to be synthesized as multi-view videos, while views in object-centric scenes are usually monocular images with no temporal relationship. This places higher demands on the temporal and spatial consistency of view synthesis in driving scenes.

The proposed generation pipeline, with pseudo-image as 3D priors, is specially designed to address the above-listed challenges in driving scenes：

1. Dynamic scene handling. Under the pseudo-image representation of 3D priors, we can easily handle dynamic objects by accumulating 3D points of moving objects along their driving trajectory. This is a simple yet effective solution for handling dynamic objects, as we will demonstrate in response to Weakness #2 below.
2. Training model on limited data. To train the model synthesizing views beyond recorded trajectories without the respective ground truth data, we propose to formulate the generation process in an inpainting fashion and simulate camera movements with augmentation (as described in Sec. 3.1, lines 185-198) on pseudo-images. Both of these designs are effective in alleviating the shortage of training data in driving scenes.
3. Cross-frame and cross-view information fusion. By representing 3D scenes with sparse, colored 3D points, we can naturally aggregate information across frames and multi-views through point cloud accumulation, therefore ensuring the temporal and spatial consistency of generated results. We will also demonstrate the strong temporal consistency of our proposed method later in response to weakness #2.

As far as we know, FreeVS is the first NVS method to be able to render high-quality views far from captured views (e.g., when laterally moving the frontal camera for 4m), which we think is a major progress in driving scene NVS and cannot be trivially achieved.

Reviewer 6Zxi recognized the novelty of our pseudo-image representation of 3D priors, while Reviewer YUkf acknowledged that the design and implementation of our generation pipeline are nontrivial and non-conventional.

References:

[1] Guo, J., Deng, N., Li, X., Bai, Y., Shi, B., Wang, C., ... & Li, Y. (2023). Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint arXiv:2306.04988.

[2] Wu, Z., Liu, T., Luo, L., Zhong, Z., Chen, J., Xiao, H., ... & Zhao, H. (2023, July). Mars: An instance-aware, modular and realistic simulator for autonomous driving. In CAAI International Conference on Artificial Intelligence (pp. 3-15). Singapore: Springer Nature Singapore.

[3] Xie, Z., Zhang, J., Li, W., Zhang, F., & Zhang, L. (2023). S-nerf: Neural radiance fields for street views. arXiv preprint arXiv:2303.00749.

[4] Turki, H., Zhang, J. Y., Ferroni, F., & Ramanan, D. (2023). Suds: Scalable urban dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12375-12385).

[5] Yang, J., Ivanovic, B., Litany, O., Weng, X., Kim, S. W., Li, B., ... & Wang, Y. (2023). Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arXiv preprint arXiv:2311.02077.

[6] Chen, Y., Gu, C., Jiang, J., Zhu, X., & Zhang, L. (2023). Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv preprint arXiv:2311.18561.

[7] Tonderski, A., Lindström, C., Hess, G., Ljungbergh, W., Svensson, L., & Petersson, C. (2024). Neurad: Neural rendering for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14895-14904).

[8] Zhou, X., Lin, Z., Shan, X., Wang, Y., Sun, D., & Yang, M. H. (2024). Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21634-21643).

[9] Yan, Y., Lin, H., Zhou, C., Wang, W., Sun, H., Zhan, K., ... & Peng, S. (2024). Street gaussians for modeling dynamic urban scenes. arXiv preprint arXiv:2401.01339.

W3. Zero-shoot generalization to other datasets / can FreeVS benefit the downstream reconstruction process

FreeVS models trained on the Waymo Open dataset (WOD) can be applied to any unseen sequences in the WOD dataset and generate high-quality results.

However, like most current image or video generation models, FreeVS has limited adaptability to changes in the image domain like imaging styles and lighting conditions. The current FreeVS models are also indeed sensitive to the number of LiDAR beams.

We showcase two examples of applying the FreeVS model trained on the WOD dataset straight to the XLD or nuScenes dataset in a zero-shot manner on the anonymized page (https://anonymize289no2.github.io/).

Like the WOD dataset, XLD dataset also has a high number of LiDAR beams, which leads to pseudo-images with dense valid pixels. Therefore FreeVS generalized to the XLD dataset can generate images with mostly correct scene contents. The generated result is mainly different from GT images in their image style.

The nuScenes dataset has significantly fewer LiDAR beams compared with the WOD dataset. FreeVS applied on the nuScenes dataset has difficulties in recovering the scene contents. Training-time augmentations on LiDAR beams might alleviate this problem. However, we find it difficult to verify this during the rebuttal period.

Finally, yes, we think the ability to generate out-of-trajectory novel views of FreeVS can benefit scene construction by providing extra pseudo observations. We plan to work on this in the future.

W4. About performance on dynamic scenes:

FreeVS has a very strong performance on dynamic scenes. In fact, none of our validation segments are static scenes and more than half of them riches dynamic objects ( such as segment 1191788760630624072_3880_000_3900_000, segment 10444454289801298640_4360_000_4380_000, segment 10625026498155904401_200_000_220_000, segment 10963653239323173269_1924_000_1944_000, segment 12161824480686739258_1813_380_1833_380, and segment 18111897798871103675_320_000_340_000. )

As described by lines 171-173 in the paper, for each dynamic object, we accumulate its 3D points along its moving trajectory and project its 3D points onto each pseudo-image based on its current position. Therefore nearly no artifacts will be introduced by dynamic objects into pseudo images. We think the easy and effective handling of dynamic objects is a unique advantage of the proposed pseudo-image representation.

Here we provide several long video sequences generated by FreeVS on another anonymized page: https://anonymize289.github.io/anonymized289.github.io/.

These videos showcase scenes with rich dynamic objects, demonstrating FreeVS's ability to handle dynamic objects well even with appearance or movement modifications on dynamic objects.

W2. Experiment on the XLD dataset:

Thanks for your constructive suggestion. We have managed to apply FreeVS on the XLD benchmark. We will quantitatively and qualitatively compare FreeVS with reconstruction-based methods (based on their performance reported in the XLD dataset paper[1]) on the XLD dataset below:

First, we report the quantitative performance of FreeVS on the XLD dataset with different offsets, following the experiment setting of Table 1 in [1]:

Experiments show that FreeVS outperforms all previous methods on all metrics on trajectories with 2m or 4m offsets. On trajectories with 1m offsets, the performance of FreeVS is still better than most previous methods. On trajectories with no offset, reconstruction-based methods have better PSNR and SSIM performance. Still, we think the high performance of reconstruction-based methods on the original trajectories comes from their overfitting on the training views, considering the huge performance gap between their performance on trajectories with no offset and trajectories with a slight 1m offset (-5.88 PSNR for UC-NeRF,-10.94 PSNR for PVG).

It is also worth emphasizing that FreeVS has a significantly better LPIPS performance on all validation trajectories, even including trajectories with no offsets. As a perceptual metric, LPIPS is more aligned with human perception compared with pixel-error metrics such as PSNR/SSIM. As the performance degradation of the reconstruction-based methods mainly comes from artifacts in the image when facing out-of-domain test views, the performance degradation of FreeVS mainly comes from the loss of high-frequency details such as the brick patterns on a wall or the number of leaves on a tree. We believe this is why FreeVS exhibits significantly better visual appeal in visualized results.

We also present a qualitative comparison between FreeVS and reconstruction-based methods through an anonymized page: https://anonymize289no2.github.io/. We compare the generation results of FreeVS with all visualization results shown in Figure 3 in the XLD dataset paper[1].

[1] XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis, arxiv, 2024.

We sincerely thank you for your valuable suggestions. Here are our responses to your comments:

W1. Comparing with methods with virtual warping:

Here we compare the performance of FreeVS and UC-NeRF on the validation sequences.

Following the official implementation (https://github.com/kcheng1021/UC-NeRF) of UC-NeRF, we only consider the three frontal cameras in the following supplementary experiments, due to UC-NeRF’s mostly hard-coded implementations and hyper-parameters optimized for the 3-camera setting.

We compare FreeVS with UC-NeRF under the novel-frame synthesis setting on the recorded trajectories and under the novel-trajectory synthesis setting. Considering dropping the side cameras will have some impact on the overall performance of each method, here we also provide the performance of EmerNerf under the 3-camera setting as a reference.

For evaluating the perceptual robustness performance of each method, we only consider ground truth labels that are visible in the three frontal cameras.

On the recorded trajectories, UC-NeRF and FreeVS have similar performance (UC-NeRF has a slightly better SSIM/PSNR performance, while FreeVS has a significantly better LPIPS performance.) On the new trajectories, FreeVS significantly outperforms UC-NeRF.

As for HO-Gaussian, we did not find its open-sourced implementation, making it very challenging to finish the respective experiments with HO-Gaussian before the rebuttal period ends.

[1] XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis, arxiv, 2024.

We sincerely thank you for your valuable suggestions. Here are our responses to your comments:

Q1: About missing citations / compare with Free View Synthesis

Thank you for reminding us of those missing citations. We have added citations to those works in the related works section (Sec. 2.2, lines 135, lines 146-147) in the revised version of our paper.

As for comparing with Free View Synthesis (FVS), we have attempted to apply FVS to the WOD dataset but did not obtain satisfactory results. We have shown one typical failure case of FVS when synthesizing views of open, dynamic scenes in the appendix of our paper (Sec A.11 and Figure E, page 20).

FVS is an early generation-based novel view synthesis method specially designed for narrow, static scenes, which are mostly object-centric. We found applying FVS on open, dynamic scenes in the WOD dataset usually can not generate high-quality views, as shown in Figure E. We believe this is due to FVS's optimization process relying on scene modeling with the COLMAP algorithm, which can not handle dynamic objects. FVS also does not consider dynamic objects in its pipeline. Additionally, the significantly more dispersed and sparse viewpoints in driving scenes also contribute to FVS's poor performance.

In such a case, we think that FVS performs too weakly as a baseline in driving scenes. We have added citations to it as above-mentioned.

Q3. About the selection of training data.

FreeVS is trained on the whole WOD training set, except for the selected validation sequences (i.e. all the 798-12 = 786 remaining sequences). We did not specifically choose sequences for model training.

We did not test the impact of the scale of training data on FreeVS before, and here we report the performance of FreeVS model trained with 25% training data (we randomly choose 197 training sequences from the above-mentioned 786 sequences):

We report the performance of models under the multi-view novel frame synthesis setting and under the novel trajectory synthesis setting (y±2m). As the experiment result shows, FreeVS is not demanding for the scale of training data.

Q4. About the performance of vanilla Stable Video Diffusion (SVD)

The vanilla stable video diffusion model is unable to handle the novel view synthesis(NVS) task.

Generation models used for novel view synthesis (NVS) need to possess two fundamental capabilities: generating images of a real scene and generating images from the specific given viewpoints. Vanilla SVD models lack the ability to precisely control the viewpoints of generated images since they do not take viewpoint priors as input.

For the NVS task, the target camera viewpoint prior is essential since we have to tell the diffusion model to generate images in which viewpoint. We think the most trivial way to encode the target camera viewpoint prior is to provide the model with a camera pose transformation matrix from the reference viewpoint to the target viewpoint.

Therefore we think the most trivial baseline in our experiment will be an SVD model with reference images and camera pose transformation matrices as priors, as the experiment setting (c) in Table 4. Dropping more priors based on experiment setting (c) in Table 4 would make the SVD model completely incapable of performing NVS.

Still, here we provide the experiment result of an extra experiment setting (f) as a supplement to Table 4, in which only the reference images are fed as priors to the generative model (i.e. a vanilla SVD model):

As reported, a vanilla SVD model with only reference images as prior performs similarly to a model trained under ablation setting (c). This is because as discussed in the current Sec. A.5 of the appendix, the generative model trained under ablation setting (c) nearly cannot follow the given viewpoint instruction. Therefore the performance of the model trained under ablation setting (c) is close to a vanilla SVD model. These models will only generate videos following the data distribution of training data. Specifically, they will only generate videos in which the ego vehicle moves along the recorded trajectory, as shown in the current Figure C. These models can not satisfy our requirements to generate views out of the recorded trajectories.

Q1. About model performance in the ablations table.

We train the SVD model for 20,000 iterations for ablation experiments in Table 4 to save training costs, as explained in line 417.

The model is trained for 40,000 iterations when compared with SOTA methods in Table 1/2/3 (as explained in line 349).

Q2. About the impact of ablation study / pretraining / model architecture

Firstly, FID alone cannot reliably reflect the model's performance in the novel view synthesis (NVS) task.

The FID metric can compare the overall image distribution between synthesized images and ground truth images, but it can not assess the 3D geometric accuracy of the synthesized images at all. Theoretically, the model can achieve high FID performance as long as it generates realistic images, no matter whether it follows the viewpoint instruction or not, or even no matter whether it generates images of the given scene or not.

To fix this, we propose the perceptual robustness metric to evaluate the geometric accuracy of generated views. The ablation experiments show great impacts on the geometric accuracy performance of models. Such as under the ablation experiment setting (d) in Table 4, the model has a very poor geometric accuracy since it can not follow the given viewpoint instructions, as shown in Figure D in the appendix.

As for the model architecture, we did not specifically optimize the model design of the Stable Video Diffusion model. In fact, our pipeline is decoupled with the design of the generation model. We initialize the model from a Stable Diffusion model checkpoint, which is a common practice. The only unusual module in our pipeline is the pseudo-image encoder, and as reported in Table 4 ((a) vs (e)), our pipeline is not sensitive to the design of the pseudo-image encoder as well. We think the key design of our work is the pseudo-image representation of 3D priors, the formulation of recovering images from pseudo-images, and the camera movement simulation technique as a training time augmentation.

We will open-source our code if the paper is accepted and welcome anyone to evaluate the robustness of our proposed pipeline regarding the design of the generative model.

We sincerely thank you for your valuable suggestions. Here are our responses to your questions:

Q1. Generation with no priors.

The vanilla stable video diffusion model with no prior inputs is unable to handle the novel view synthesis(NVS) task.

Generation models used for novel view synthesis (NVS) need to possess two fundamental capabilities: generating images of a real scene and generating images from the specific given viewpoints.

We regard reference images as a default input prior, like most Stable Video Diffusion (SVD) models for driving scenes.

The target camera viewpoint is also an essential condition since we have to tell the diffusion model to generate images in which viewpoint. We think the most trivial way to encode the target camera viewpoint prior is to provide the model with a camera pose transformation matrix from the reference viewpoint to the target viewpoint.

Therefore we think the most trivial baseline in our experiment will be a video generation model with reference images and camera pose transformation matrices as priors, as the experiment setting (c) in Table 4.

Dropping more priors based on experiment setting (c) in Table 4 would make the video diffusion model completely incapable of performing NVS. Still, here we provide the experiment result of an extra experiment setting (f) as a supplement to Table 4, in which only the reference images are fed as priors to the generative model (i.e. a vanilla SVD model):

As reported, a vanilla SVD model with only reference images as prior performs similarly to a model trained under ablation setting (c). This is because as discussed in the current Sec. A.5 of the appendix, the generative model trained under ablation setting (c) nearly cannot follow the given viewpoint instruction. The performance of the model trained under ablation setting (c) is close to a vanilla SVD model. These models can not precisely control the viewpoint of generated views, and therefore can not handle the NVS task well.

Q2. Generation without ground truth LiDAR points.

Yes, the LiDAR inputs for FreeVS can be replaced by pseudo-LiDAR points generated from estimated depths.

We experimented with applying an off-the-shelf Depth-Anything-V2[1] model on reference images and sample pseudo LiDAR points from the obtained depth map. Here we show a success case as well as a failure case of generation with pseudo LiDAR points through an anonymized page: https://anonymize289no3.github.io/. We also provide a video comparison between results generated with GT LiDAR points or pseudo LiDAR points.

According to our observation, the proposed FreeVS pipeline can be applied to pseudo LiDAR points and recover most scene contents correctly. However, we found the depth prediction model sometime has problems in predicting the depth of objects very near to the camera, as shown in the failure case, circled with red. We believe this issue is caused by directly applying the Depth-Anything-V2 model to the WOD dataset. A depth predictor that is fully converged on the WOD dataset should be able to avoid this problem. As for FreeVS, the visualized results show that our method has satisfying performance on most scene contents with about the right depth.

Note that we generate the above-provided results by directly applying the FreeVS model trained with GT LiDAR points on the pseudo LiDAR points. The different valid pixel patterns and pixel density of pseudo image generated from pseudo LiDAR points lead to some slight imaging style changes of the generated result, which is totally avoidable by training FreeVS on pseudo image generated from pseudo LiDAR points. However, we do not have enough time to validate this during the rebuttal period. We believe the above-provided results can prove that the proposed FreeVS does not heavily rely on GT depth. We have added the above-reported results to Sec. A.8 in the appendix.

[1] Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., & Zhao, H. (2024). Depth Anything V2. arXiv preprint arXiv:2406.09414.