DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving

（W4）The Necessity of a 4D Feedforward Network

While we acknowledge that the rendering quality of the feedforward network in driving scenes may currently be slightly lower than that of optimization methods, we are confident that feedforward networks will surpass optimization methods in the near future, especially with advancements in generative techniques. We would like to highlight several key points regarding the necessity of feedforward networks:

1. General Driving Pre-Training Model: The 4D feedforward network serves as a general pre-training model for driving tasks. By learning 4D reconstruction, the network extracts geometric information from the entire scene while simultaneously predicting motion information for moving objects. This pre-training approach captures more geometric and temporal features than existing methods, as verified in Table 6 of our paper. Our model can serve as a pre-training tool to enhance the performance of driving algorithms in multiple tasks such as perception and planning without relying on labels like segmentation or 3D bounding boxes.
2. Possibility of a 4D World Model: Current driving world models often exist in video form without explicit geometric representation, leading to limitations in perspective and temporal consistency. Upgrading our reconstruction model to support predictable 4D generation could address these issues. A 4D world model could provide explicit geometry to constrain end-to-end planning, representing a significant advancement for the field.
3. Scaling Capabilities: The 4D feedforward network has the potential for scalability. It can be trained using an abundance of driving videos available on the Internet. This extensive training could greatly enhance reconstruction capabilities, which can then be leveraged for tasks such as perception and planning, either through pre-training or as part of the world model.
4. Importance of Generalization and Efficiency: Building a 4D scene for a new city using optimization methods is very time-consuming, particularly with long videos capturing thousands of scenarios. For example, a 200-frame video (10 seconds) might take around 6 hours for an optimization-based reconstruction method, which is not feasible in practice.

（Q）Minor Typos

We would like to thank the reviewer for catching the typo regarding “DA-Block” in line 202; it should indeed be “PD-Block.” We will proofread the paper thoroughly to enhance its writing, presentation, and layout.

Thank you for your attention to our responses. If you have any further questions or if anything remains unclear, please don’t hesitate to let us know. We would be more than happy to discuss your concerns in greater detail.

---

（W1）The Need for Depth Map Ground Truth

We appreciate the reviewer's insights on the necessity of depth supervision. While depth supervision is not strictly required in our framework, we acknowledge that it can enhance the speed of model convergence and lead to improved final results.

To further clarify our findings, we conducted a comparative experiment to evaluate the impact of depth supervision under the experimental conditions detailed in Table 1. The results are presented below:

The table indicates that our algorithm can still learn geometric information through temporal consistency, even in the absence of depth constraints. Furthermore, in driving scenarios, depth information from LiDAR is typically readily available, as discussed in other works (e.g., paper [1]).

（W2）The Limitations of Semantic Segmentation

Although we now only segment cars, two-wheelers and pedestrians as dynamic objects. Here are some simple ideas for segmenting all dynamic objects. For instance, calculating optical flow from videos could help identify points in dynamic regions, which could then serve as prompts for a segmentation algorithm like SAM. This would enable us to segment a broader range of dynamic objects.

We believe that this limitation should not overshadow the value of our paper, as many advanced and well-regarded driving reconstruction algorithms share this same challenge [1, 2, 3, 4]. Addressing this issue is a minor aspect of our framework and does not reflect the core innovations presented in our work.

[1] Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. CVPR2024.

[2] Holistic urban 3d scene understanding via gaussian splatting. CVPR2024

[3] OmniRe: Omni Urban Scene Reconstruction." arXiv preprint arXiv:2408.16760 (2024).

[4] Street gaussians for modeling dynamic urban scenes. ECCV2025

（W3）Lack of Efficiency Analysis

We appreciate the reviewer pointing out the need for an efficiency analysis. We will include an analysis of the efficiency of our method in the paper. According to the evaluation protocol in Table 1, we compared the speed and PSNR of our method against traditional optimization methods:

As shown in the table, our algorithm performs comparably to traditional optimization methods in terms of PSNR, while significantly reducing time cost. The experiment was updated to part C of supplementary materials. To prove the effectiveness of our reconstruction, we have provided some reconstruction videos where you can observe our method at the following link: https://anonymize58426.github.io/Drive-Recon/. In the videos, you can see the lane lines being translated between 3 to 12 seconds, and the viewpoint being rotated between 17 to 26 seconds. This demonstrates that the scenes reconstructed by our algorithm maintain geometric consistency.

Dear Reviewer EbAv,

I would like to express my sincere gratitude to you for your constructive comments. As the ICLR discussion phase is almost over, I wanted to kindly ask if there are any remaining questions or clarifications needed regarding our responses. Please feel free to reach out at any time.

We would be truly grateful if you could consider raising our rating, as your support is crucial for the potential acceptance of our work.

Best wishes,

Authors

Dear Reviewer EbAv,

Thank you for carefully reading our response and providing further comments. I hope our further discussion will allow you to change your opinion.

Most importantly, multiple supervision and complex modules are not a weakness to reject a paper: (1) optimization-based driving reconstruction algorithm approaches will also have multiple supervision (point cloud initialization and segmentation of dynamic and static objects) [1,2,3,4]. When reasoning, our models do not need depth and segmentation labels. Besides, our method can indeed be trained without requiring semantic segmentation or 3D boxes. As shown in Table 4(a), we can learn geometry through perspective consistency and point cloud depth information without the need for Dynamic and Static Rendering (DS-R). In the W1 response we also explained that our approach can be done without depth supervision. (2) For learnable driving tasks, such as perception task, they also have multiple modules including temporal fusion model, multi-view fusion model, image encoder model, image decoder and detection [5, 6, 7].

[1] Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. CVPR 2024

[2] OmniRe: Omni Urban Scene Reconstruction. arXiv preprint arXiv:2408.16760 (2024).

[3] Street gaussians for modeling dynamic urban scenes. ECCV2025

[4] Unisim: A neural closed-loop sensor simulator. CVPR2023.

[5] Exploring object-centric temporal modeling for efficient multi-view 3d object detection. CVPR2023

[6] Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. ECCV2023

[7] Panoocc: Unified occupancy representation for camera-based 3d panoptic segmentation. CVPR2024

Compare the existing FeedForword method

Thanks to the reviewer's reminder, we should indeed evaluate the efficiency of other SOTA forward generalizable models.

As shown in the table, our method is significantly optimal in reasoning speed and memory usage. For the automatic driving scene, the efficiency of our method is due to: (1) Multi-view fusion better integrates multiple views with small overlap through the form of range view. (2) Temporal fusion is the fusion of highly compressed implicit features, which greatly reduces memory and inference delay. (3) Image encoder and decoder are shared for different views and can be inferred in parallel.

Disadvantages of other methods: (1) For the input of multiple views, MVSplat needs to calculate the cost volume between any two image pairs, which greatly increases the computational memory and inference delay. (2) LGM and L4GM cat all the images into a multi-view attention fusion network. The uncompressed image sent to the view fusion network consumes memory and increases inference delay. In addition, the small overlap of different perspectives in the driving scene does not require such redundant attention mechanisms. (3) pixelSplat uses the polar coordinate attention fusion mechanism to integrate different perspectives. The small overlap of different perspectives in the driving scene does not require such redundant attention mechanisms. Specifically, a large number of queries are empty.

Limitations of the Original 3DGS

In our efficiency comparison experiments, the original 3DGS heavily also relies on point cloud supervision to initialize the Guassian points. At each time step $t$, a 3DGS model needs to be trained. This is why it takes a significant amount of time, approximately 5.5 hours, to reconstruct a 200-frame video using 3DGS. However, the reconstruction performance tends to overfit, and its ability to synthesize novel view is bad, as shown in the table below.

As shown in the table, 3DGS significantly deteriorated in new view synthesis. At each time step, there are only a few observed viewpoints in driving scenes (such as 6 views in nuScenes and 5 in Waymo) to supervise 3DGS. Besides, the overlap between these surround views is minimal, making it challenging for 3DGS to accurately learn the geometry. To address this, DrivingGaussian and ours use segmentation to identify the static background and perform cross-temporal supervision.

If you have any further questions or if anything remains unclear, please don’t hesitate to let us know.

Thank you for your thoughtful comments and for taking the time to review our work. Your feedback is genuinely appreciated, and I hope to clarify and enhance our responses to your concerns.

(W1 and W4) Generalization to Different Viewpoints

To demonstrate the effectiveness of our method in generating new perspectives, we have provided a series of reconstruction videos available at the following link: Drive-Recon Videos. In these videos, you can observe the translation of lane lines between 3 to 12 seconds and the rotation of viewpoints between 17 to 26 seconds. This effectively illustrates that the scenes reconstructed by our algorithm maintain geometric consistency across varying perspectives.

Additionally, I appreciate your reference to the Freevs method; it is indeed an intriguing approach. I believe we can adapt some of its useful components to enhance our model's ability to generate new viewpoints.

(W2) Limitations of Resolution

Currently, it seems that feedforward networks yield slightly lower rendering quality in driving scenarios compared to traditional optimization methods. However, I am hopeful that advancements in network architectures, training strategies, and generative techniques will soon enable feedforward networks to exceed the capabilities of optimization methods. Our paper serves as a preliminary study in the driving domain and provides a codebase that may accelerate progress in this area.

Moreover, existing state-of-the-art generalizable Gaussian splatting algorithms tend to operate at lower resolutions and realism, particularly in indoor scenes [1, 2, 3]. Therefore, we believe our work is still at the forefront of the field of generalizable Gaussian splatting.

References:

1. Mvsplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images. ECCV 2025.
2. Large Spatial Model: End-to-End Unposed Images to Semantic 3D. NeurIPS 2024.
3. FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scene Reconstruction. NeurIPS 2024.

(W3) Lack of Efficiency Analysis

Thank you for highlighting the need for an efficiency analysis. We will include a thorough evaluation of our method's efficiency in the revised paper. Based on the evaluation protocol outlined in Table 1, we compared the speed and PSNR of our method against traditional optimization methods:

As indicated in the table, our algorithm performs comparably to traditional optimization methods in terms of PSNR while significantly reducing time costs. This efficiency makes our method more suitable for data-driven applications, such as driving simulators. The experiment was updated to part C of supplementary materials.

(Q) Details of Scene Editing

To edit scenes, we can utilize existing 3D generation models to create Gaussian representations of arbitrary objects, such as LGM [1]. We then modify the Gaussian representation of the x, y, and z positions (in world coordinates) to place them appropriately within the driving scenario. Importantly, the Gaussians predicted by our method are represented within the world coordinate system.

Reference:

1. LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation.

Thank you for your attention to our responses. If you have any further questions or if anything remains unclear, please don’t hesitate to let us know. We would be more than happy to discuss your concerns in greater detail.

Dear Reviewer 3kKi,

I would like to express my sincere gratitude to you for your constructive comments. As the ICLR discussion phase is almost over, I wanted to kindly ask if there are any remaining questions or clarifications needed regarding our responses. Please feel free to reach out at any time.

We would be truly grateful if you could consider raising our rating, as your support is crucial for the potential acceptance of our work.

Best wishes,

Authors

Dear 3kKi

Thank you for carefully reading our response and providing further comments. Here is our further response:

Compare the existing FeedForword method

We apologize for our misunderstanding. Our method in the table is inference latency, and the other optimization method is optimization time. The inference latency of rendering part is only about 0.04 seconds. In addition to optimization-based methods, we further evaluated the efficiency of other SOTA forward generalizable models.

As shown in the table, our method is significantly optimal in reasoning speed and memory usage. For the automatic driving scene, the efficiency of our method is due to: (1) Multi-view fusion better integrates multiple views with small overlap through the form of range view. (2) Timing fusion is the fusion of highly compressed implicit features, which greatly reduces memory and inference delay. (3) Image encoder and decoder are shared for different perspectives and can be inferred in parallel.

Disadvantages of other methods: (1) For the input of multiple graphs, MVSplat needs to calculate the cost volume between any two images, which greatly increases the computational memory and inference delay. (2) LGM and L4GM cat all the images into a multi-view attention fusion network. The uncompressed image sent to the view fusion network consumes memory and increases inference delay. In addition, the small overlap of different perspectives in the driving scene does not require such redundant attention mechanisms. (3) pixelSplat uses the polar coordinate attention fusion mechanism to integrate different perspectives. The small overlap of different perspectives in the driving scene does not require such redundant attention mechanisms. Specifically, a large number of queries are empty.

Some issues of novel views

For driving scenes, synthesizing new views is still a very challenging problem. Even well-known optimization methods, such as DrivingGuassian, do not render new perspectives well. The Freevs you mentioned is a good new view synthesis solution, even if it is only the latest paper appearing on arxiv. In addition, Freevs is only a generative method for synthesizing new view images, rather than a 3D/4D reconstruction method for predicting Gaussians. These are the two routes. DriveRecon makes it very easy to incorporate these new method into a synthetic solution. I believe our approach has great potential.

Thank you for your attention to our responses. If you have any further questions or if anything remains unclear, please don’t hesitate to let us know. We would be more than happy to discuss your concerns in greater detail.

(W1) Overcomplicated Design

Thank you for your thoughtful comments regarding the complexity of our model. We believe that the design elements are essential for effective driving reconstruction for several reasons:

1. Necessity of Different Modules: The 4D feedforward model requires the efficient integration of multiple perspectives and varying time intervals. To achieve this, we utilize temporal cross-attention to merge temporal information and implement the PD-Block for effective multi-view image fusion. Additionally, the GaussianAdapter facilitates the transfer of image features into a Gaussian representation.
2. Importance of Regularization Terms: At any given moment $t$, the rendering supervision of the scene is limited by the sparse number of views. Moreover, the presence of multiple dynamic objects complicates monitoring at time $t$. To address this, we decouple dynamic objects from static ones, allowing for better perspective utilization during the rendering process. Although this decoupling introduces numerous regularization terms, we regard them as necessary for achieving the desired outcomes.
3. Significance of Pretrained Models: Besides leveraging the SAM and DeepLab models, we acknowledge that more efficient methods exist for decoupling dynamic and static objects, which we discuss in detail in response to Q3. Importantly, DepthNet is not a pretrained network; it constitutes a part of our model that can be trained.

Our findings in Table 4(a) demonstrate that these components are both valid and necessary. For downstream tasks, we utilize only the image encoder as the pretrained model, omitting the others to prevent any additional burden on training and inference. The model’s reconstruction inference speed is elaborated upon in (Q1).

Furthermore, even widely recognized optimization-based reconstruction algorithms exhibit significant complexity in driving scenes [1, 2, 3, 4, 5]. They often integrate multiple annotations, regularizers, and neural network architectures.

[1] Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. CVPR 2024

[2] Hugs: Holistic urban 3d scene understanding via gaussian splatting. CVPR2024

[3] OmniRe: Omni Urban Scene Reconstruction. arXiv preprint arXiv:2408.16760 (2024).

[4] Street gaussians for modeling dynamic urban scenes. ECCV2025

[5] Unisim: A neural closed-loop sensor simulator. CVPR2023.

(W2) The limation of realism

Currently, it appears that feedforward networks exhibit slightly lower rendering quality in driving scenarios compared to optimization methods. We have provided some reconstruction videos where you can observe our method at the following link: https://anonymize58426.github.io/Drive-Recon/. In the videos, you can see the lane lines being translated between 3 to 12 seconds, and the viewpoint being rotated between 17 to 26 seconds. This demonstrates that the scenes reconstructed by our algorithm maintain geometric consistency. I am confident that, in the near future, advancements in network architectures, training strategies, and generative techniques will enable feedforward networks to surpass optimization methods. This paper is a preliminary study at the field of driving and give a code base, which will accelerate the development of the field.

Furthermore, existing state-of-the-art generalizable Gaussian splatting algorithms often operate at relatively low resolutions and realism in indoor scenes [1, 2, 3]. Therefore, our paper is still at an advanced level in the field of generalizable Gaussian splatting.

[1] Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. ECCV 2025.

[2] Large Spatial Model: End-to-end Unposed Images to Semantic 3D. NeurIPS 2024

[3] FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes Reconstruction. NeurIPS 2024

(Q1) Comparison to Optimization-Based Approaches

We appreciate your inquiry regarding efficiency. We require only 20 hours to complete 50,000 iterations on 24 A100 GPUs. According to the protocol outlined in Table 1, we have compared the time cost and PSNR of our method with that of traditional optimization methods.

As illustrated in the table above, our algorithm is nearly comparable to traditional optimization methods, while inference takes only 1.21 seconds, highlighting our method's substantial speed advantage. The experiment was updated to part C of supplementary materials.

---

(Q2) The Need for 3D Boxes

I am grateful for your question about the use of 3D boxes. Our method can indeed be trained without requiring semantic segmentation or 3D boxes. As shown in Table 4(a), we can learn geometry through perspective consistency and point cloud depth information without the need for Dynamic and Static Rendering (DS-R).

Additionally, we can use simple techniques for segmentation without utilizing 3D boxes. For example, we can compute optical flow from videos to identify points in dynamic regions, which can be used as prompts for the SAM to segment dynamic objects, enabling us to segment any type of dynamic object without 3D boxes.

---

(Q3) Downstream Applications

We train DriveRecon on the nuScenes training set without relying on 3D boxes or segmentation ($\lambda_{sr} =0$ and $\lambda_{seg}=0$). The pretrained model is then employed as an image encoder for UniAD. Then, UniAD is fine-tuned entirely using the original UniAD's training parameters. The results for UniAD in Table 6 are sourced from the original paper, and our pretrained model significantly enhances performance. I will provide clearer details about the experimental setup in the the paper.

---

(Q4) Miscellaneous

Thank you for your attention to our experimental setup. We utilized three frames of images as input for all experiments. In Table 4(b), "Training Num" refers to the mean number of scenes. Tables 1, 2, 3, and 4(a) use only 64 scenes (NOTA-DS64). Specifically, we trained with 64 scenes (NOTA-DS64) and tested with 54 new scenes (Diversity-54), achieving satisfactory results. This indicates that our algorithm demonstrates good generalization performance even when trained on a small dataset. We will reiterate these details in the appropriate sections of the paper.

Thank you for your attention to our responses. If you have any further questions or if anything remains unclear, please don’t hesitate to let us know. We would be more than happy to discuss your concerns in greater detail.

Dear Reviewer conn,

I would like to express my sincere gratitude to you for your constructive comments. As the ICLR discussion phase is almost over, I wanted to kindly ask if there are any remaining questions or clarifications needed regarding our responses. Please feel free to reach out at any time.

We would be truly grateful if you could consider raising our rating, as your support is crucial for the potential acceptance of our work.

Best wishes,

Authors

Dear Reviewer conn,

Thank you for carefully reading our response and providing further comments. This is our further reply:

(P1) Complicated Design

We acknowledge that our approach is complex. But for autonomous driving, it's hard to avoid. For learnable driving tasks, such as perception task, they also have multiple modules including temporal fusion model, multi-view fusion model, image encoder model, image decoder and detection head [1, 2]. These methods, which were considered too complex, have already been deployed to run on actual vehicles. So, this is not a shortcoming to reject our paper. Moreover, the two algorithms you mentioned are also complex [3,4].

[1] Exploring object-centric temporal modeling for efficient multi-view 3d object detection. CVPR2023

[2] Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. ECCV2023

[3] G3r: Gradient guided generalizable reconstruction.ECCV 2025.

[4] SCube: Instant Large-Scale Scene Reconstruction using VoxSplats." arXiv preprint arXiv:2410.20030 (2024).

(P2) Other generalizable approaches

Thank you for mentioning the two new related papers. The quality of the two papers produced is not significantly better than ours. If you zoom in on Fig.4 both of Paper 3 and Paper 4, you find that their rendering quality is not good enough. Paper 3 and 4 are difficult to compare because of its lack of open source and complex design. Paper 4 appeared after we submitted the paper. In addition, we discuss them further:

Paper [3] also lacks realism. In the presentation of his paper, his images (Fig.4) are very small. If you zoom in these pictures are very blurry. And he reported reasoning speeds of 31s and 123s for a single scence. His model could not even predict the 3D Gaussian representation directly, and he needed multiple iterations of the network to predict the 3D Gaussian representation. Besides, his method is not open source and very difficult to reproduce with his complex algorithm.

Method [4] appears on arxiv after we commit ICLR. The visualization in Figure 4 in his paper is not much better than ours. Because we are working with multiple views of multiple times, the resolution of the image we render is lower due to GPU memory. His approach was to take only three images as input and split the model into two stages. This allows him to render more high-resolution images. His method can learn geometric features entirely by relying on dense point cloud supervision, which uses a set of point-cloud dense methods. I believe that our approach has more potential with small improvements: (1) Splitting different time images onto different GPUs and then merging timing features across GPUS, which is already widely used in video generation. In this way we can render more high-resolution images. (2) Our approach further makes good use of pre-trained video models. However, his approach relies on Voxel-based models, which severely limits his upper boundary. (3) The visual encoder of our method can be used as a pre-training model, while his model cannot be used at all.

I believe that in the near future, advances in network architecture, training strategies, and generation techniques will enable feedforward networks to transcend optimization methods. For example: (1) Use pre-trained video encoders and decoders to improve the rendering quality. (2) Assign different time images to different GPUs and then perform feature fusion across GPUs, which is a common operation in the field of video generation. (3) Use more driving datasets to train stronger models.

In addition, the most advanced generalizable Gaussian spatter algorithms available generally operate with relatively low resolution and realism in indoor scenes [5,6,7]. Therefore, this paper is still at an advanced level in the field of generalized Gaussian sputtering.

[5] Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. ECCV 2025.

[6] Large Spatial Model: End-to-end Unposed Images to Semantic 3D. NeurIPS 2024

[7] FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes Reconstruction. NeurIPS 2024

UniAD experimental

I would like to express my sincere apologies for the inaccuracy of my reply. I copied the original data of this table from Table 10 from VIDAR[8] ranther than UniAD. The experimental parameters were completely carried out in accordance with page https://github.com/OpenDriveLab/ViDAR. Specifically, we just replaced the image encoder and everything else was exactly the same.

[8] Visual Point Cloud Forecasting enables Scalable Autonomous Driving. CVPR 2024

Thank you for your attention to our responses. If you have any further questions or if anything remains unclear, please don’t hesitate to let us know. We would be more than happy to discuss your concerns in greater detail.