GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Driving

Q1. Additionally experiments about V2-99 vision backbone.

Thank you for your helpful suggestion. To explore the scalability of our framework, we extended GaussianFusion by applying a more powerful image backbone (V2-99) and increasing the dimension of Gaussian feature from 128 to 256 to adapt to V2-99. The results are shown in Table 1 below.

As we can see, our enhanced version (GaussianFusion*), equipped with V2-99, further improves performance across all metrics—achieving state-of-the-art results on the Navtest benchmark. This demonstrates the scalability of our framework with stronger backbones.

We note that, due to the tight rebuttal timeline, we were only able to perform a single-seed training run for this extended version. Nevertheless, the results already show clear performance gains. The results will be updated to Table 1 of the paper for comprehensive comparison.

Table 1. Performance on the Navtest Benchmark. ‘*’ denotes an extended version of our method.

Q2. Why the EPDMS metric is used with NavSim v1?

Thank you for the question. We adopt EPDMS because it is a more challenging and comprehensive evaluation metric than PDMS. Originally proposed by Hydra-MDP++ [1], EPDMS extends the standard evaluation protocol by introducing several additional criteria:

• Lane Keeping (LK)
• Extended Comfort (EC)
• Driving Direction Compliance (DDC)
• Traffic Light Compliance (TLC)
• False-Positive Penalty Filtering

These components enable a more holistic assessment of driving behavior. Importantly, while EPDMS is employed by NavSim v2, it is also compatible with NavSim v1 for single-stage evaluation.

We agree with the reviewer that only a limited number of existing methods have been evaluated under EPDMS, which may affect direct comparability. Therefore, following your suggestion, we will move the EPDMS results to the Appendix in the final version.

Q3. Training end-to-end driving methods can exhibit quite some variance. Both Tables 1 and 2 would benefit from evaluating multiple (3 is typical) training seeds and reporting the mean and standard deviation.

Thanks for your helpful suggestion. We have trained models under 3 training seeds on both the NAVSIM and Bench2Drive benchmarks. As the evaluation on the CARLA-based Bench2Drive benchmark is very time-consuming, one trained model is only evaluated once. Due to the limitation of character count, please see the results in Tables 1 and 2 of the response to the Reviewer v3h9.

On the NAVSIM benchmark, our method exhibits stable performance with minimal variance, reflecting its robustness to random initialization and consistent behavior across multiple runs.

For the closed-loop Bench2Drive benchmark, while slightly higher variance is observed—as expected due to the complexity and stochastic nature of interactive scenarios—our method consistently surpasses other baselines in average performance. These results will be updated to the final version of the paper.

Q4. Reporting FLOPs to better show the computational efficiency.

According to your suggestion, we have calculated the FLOPs of different fusion methods. The results are shown in Table 2 below.

Table 2. Comparison of different fusion methods on accuracy and efficiency.

Compared with the BEV fusion method, our Gaussian fusion achieves better performance (+0.04 mIoU and +0.7 EPDMS), while also significantly reducing computational cost: the FLOPs are 38.9% lower (58.4G → 35.7G) and latency is reduced by 25.6% (43ms → 32ms), showing clear advantages in both accuracy and efficiency.

Compared with the Flatten (TransFuser-style) fusion, which is a very lightweight approach, our method exhibits only a 30.3% increase in FLOPs (27.4G → 35.7G), while achieving much stronger performance (+0.05 mIoU and +2.7 EPDMS). However, our method's latency is 77.8% higher (18ms → 32ms), which we believe is largely due to the differences in operator-level optimization. This could be further mitigated through engineering improvements in deployment.

Q5. The examples shown in Figure 4 do not seem particularly interesting. The first example is at the end of a left turn, where the car just has to follow the lane. The second and third examples seem to be basic straight driving. Figure 4 could be moved to the appendix if more space is needed in the main paper.

Thanks for your suggestion. We will move Figure 4 to the appendix and include additional visualizations of more complex scenarios and failure cases in the final version of the paper.

Q6. The 2 sentence limitations in the conclusion seem to be too brief. There are probably more limitations to this study. These can perhaps be expanded in the appendix if space is a concern.

Thank you for pointing this out. We agree that the limitations section in the conclusion is overly concise and does not sufficiently reflect the boundaries of our study. In the final version, we will include a dedicated Limitations section in the appendix to provide a more comprehensive discussion.

Based on the reviewer’s suggestion and further reflection, one important limitation is that we have not evaluated the robustness of our fusion method under sensor occlusion or noise, which is a critical aspect for real-world deployment.

Another limitation is that our current design relies on 3D bounding boxes and road topology annotations to optimize 2D Gaussians. While effective, such supervision can be expensive or unavailable in real-world deployment scenarios.

We appreciate the reviewer’s input in helping us better contextualize the scope of our work.

Q7. Citing a BEV method MILE [1] for E2E driving.

Thank you for pointing this out. MILE [1] is a pioneering work in BEV-based end-to-end driving. It lifts image features into 3D space by leveraging a predicted depth probability distribution and then aggregates the resulting voxel features into the BEV space via a predefined grid-based sum-pooling strategy. We appreciate the reviewer’s suggestion and will include a citation and discussion of MILE in the related work section of the final version.

Q8. About the image resolution setting.

Thank you for pointing this out. We acknowledge that the resolution setting was not clearly described in the manuscript. The resolution of 448 × 250 (W×H) is used on the NAVSIM benchmark, where multiple camera views are concatenated. For the Bench2Drive benchmark, we adopt only the front-view camera, which already provides a wide field of view. In this case, we use the resolution of 1024 × 384 (W×H), which is consistent with the setting used in TransFuser++.

We will clarify this resolution setting and correct the description in the final version of the paper.

Q9. About training of the GaussianFusion framework.

In our framework, the perception head (used for map construction) and the planning head (used for trajectory prediction) are designed to be independent branches. The perception head does not contribute directly to the driving actions; rather, it serves as an auxiliary supervision signal to optimize 2D Gaussians and build geometry inductive biases.

As for the training strategy, our model is trained in a single-stage end-to-end manner, without any separate pretraining of the perception head.

Q10. About the training settings on the Bench2Drive benchmark.

We believe the lower performance of TF++ in our experiments can indeed be attributed to the factors: the smaller model capacity and the reduced training data.

Specifically, the original TF++ paper utilizes RegNetY-32 as the backbone and is trained on a dataset that is roughly three times larger than what we used in our experiments. In contrast, our reproduced version of TF++ adopts a lighter-weight backbone (for fair comparison) and is trained from scratch with a smaller training split (due to the computing power limitation).

According to your suggestion, we will explicitly state the differences about the training datasets in the caption of Table 2 of the paper.

References

[1] Hu A, Corrado G, Griffiths N, et al. “Model-Based Imitation Learning for Urban Driving”. NeurIPS, 2022.

Dear Reviewer,

There is an author-reviewer discussion stage before Aug. 6th. Please take a look at of the authors' rebuttal as soon as possible, and leave a message to authors in case you still require additional clarity, or if you want to post points of disagreement for discussion with the authors.

AC

Q1. Concerns about the necessity and advantage of 2D Gaussian representations over existing methods.

Thank you for the thoughtful comments. Our proposed 2D Gaussian framework offers several key advantages over existing methods.

First, compared to GaussianAD [1], our method significantly reduces the number of Gaussians required to model a driving scene by compressing 3D Gaussians into 2D representations (25600 Gaussians in GaussianAD vs. 512 in ours). This leads to a substantially lower computational burden. Moreover, GaussianAD requires expensive 3D occupancy labels for optimization, while our method only uses 2D semantic maps—projected from road topology and bounding boxes onto the ground plane. These labels are widely available in existing driving datasets and can be more easily generated through automatic labeling pipelines.

Second, compared to BEV-based fusion methods [2], our 2D Gaussian framework improves both efficiency and accuracy. Specifically, our method reduces inference latency by 25.6%, while achieving better performance in perception and planning, as shown in Table 3 of the paper. In a typical BEV setup covering a range of [-32m, 32m] on both the x and y axes, a voxel size of 0.5m results in a [128, 128] query map—i.e., 16,384 BEV queries. In contrast, our approach uses only 512 deformable and position-adaptive 2D Gaussians, which are far more compact and flexible in representing the scene compared to dense BEV grids.

Q2. Authors can provide some visualization results with more complex motion modes and some failure cases.

Thanks for your suggestion. Due to the limitations of the current rebuttal submission, we are unable to include visualization results here. We will include the relevant visualizations—covering both complex motion scenarios and failure cases—in the appendix of the final version.

References

[1] Zheng W, Wu J, Zheng Y, et al. "Gaussianad: Gaussian-centric end-to-end autonomous driving". arXiv preprint arXiv:2412.10371, 2024.

[2] Li Z, Wang W, Li H, et al. "Bevformer: learning bird's-eye-view representation from lidar-camera via spatiotemporal transformers". IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

Q1. Can the current 2D Gaussian framework handle scenarios with vertical structures (e.g., viaducts, tunnels)? Are there plans to incorporate 3D Gaussians or hybrid representations?

Our current 2D Gaussian framework primarily focuses on modeling foreground objects and road topology, which are critical for safe driving decisions. Since the output of the end-to-end planner is a 2D trajectory on the ground plane, we simplify the scene representation by assuming the ego vehicle operates on a relatively flat plane in its local coordinate system.

Although extending our framework to support full 3D Gaussians or hybrid representations is technically feasible, such extensions would incur prohibitive computational costs that would undermine the system's real-time performance. Therefore, we currently do not plan to extend our framework with 3D Gaussian representations.

Q2. The paper lacks detailed comparisons with BEV methods in terms of computational efficiency, especially in real-time driving scenarios.

Thank you for highlighting this important aspect. We fully agree that computational efficiency is critical for real-time deployment.

We compare the inference latency of a representative BEV-based method and our approach on an NVIDIA RTX 3090 GPU, as reported in the Table 4 of the paper. Our method achieves superior perception and planning performance while maintaining a lower inference latency of approximately 32 ms.

Given the unavailability of an embedded platform such as the NVIDIA Jetson AGX Orin, we conduct latency measurements on a desktop GPU (GTX 1080 Ti) as a proxy. This choice is justified by the fact that both hardware platforms exhibit similar computational capabilities for inference. Using the PyTorch implementation with float32 precision and without inference-specific optimizations, our model achieved an average inference time of 65 ms, which demonstrates the real-time applicability of our approach.

We acknowledge the importance of direct on-device benchmarking and plan to include such evaluations when the necessary hardware becomes available.

Q3. Could you include ablation results for the dual-branch structure and cascade planning head to quantify their contributions to EPDMS/PDMS metrics?

Thanks for the valuable suggestion. We conducted ablation studies to quantify the individual contributions of the dual-branch structure and the cascade planning head to the EPDMS metrics. The results are summarized in the table below:

The results show that adding the Gaussian Explicit Fusion branch alone (second row) significantly improves perception performance, with DAC increasing from 94.1 to 96.5 and EPDMS from 81.8 to 84.2 compared to the baseline without either branch. Adding the Gaussian Implicit Fusion branch independently (third row) further enhances these metrics slightly, raising DAC to 96.6 and EPDMS to 84.5.

Separately, incorporating the cascade planning head with only the explicit fusion (fourth row) also improves trajectory prediction, increasing LK and EPDMS compared to using explicit fusion alone. The best overall performance is achieved when combining dual-branch fusion and the cascade head (fifth row), reaching an EPDMS of 85.0. We will update these ablation results to Table 3 of the paper.

Q4. Do NAVSIM and Bench2Drive cover extreme weather conditions? How robust is the framework under low visibility?

Thank you for this insightful question. The Bench2Drive benchmark includes scenarios with extreme weather conditions such as heavy rain and dense fog, offering a realistic testbed to evaluate robustness under challenging environments.

Our method demonstrates strong robustness in low-visibility conditions. As shown in the supplementary materials, a demo video illustrates our framework performing effectively in a scenario with dense fog and heavy rain. These results confirm that our approach can maintain reliable perception and planning performance despite adverse weather, underscoring its potential for real-world deployment in extreme conditions.

Dear Reviewer,

There is an author-reviewer discussion stage before Aug. 6th. While you are positive to this paper, please also take a look at of the authors' rebuttal as soon as possible to see if you need further clarity. Please do not forget to leave a message to authors in case you still require additional clarity, or if you want to post points of disagreement for discussion with the authors.

AC

Q1. Although the framework is well thought out / developed, the core ideas build upon existing methods.

We sincerely appreciate the reviewer's recognition of our overall framework design. Our approach is inspired by prior great works, including 3D Gaussian Splatting [1] and GaussianFormer [2]. Our key contribution lies in introducing 2D Gaussian scene representations as a unified bridge for multi-sensor feature fusion in end-to-end planning. Notably, we are the first (to our knowledge) to apply 2D Gaussians for driving scene representation—a design that simultaneously achieves three critical advantages:

1. Flexible: Positions and shapes of 2D Gaussians can be dynamically adjusted during the feed-forward process.
2. Efficient: A few hundred 2D Gaussians are sufficient to model an entire driving scene.
3. Intuitive: 2D Gaussians can be directly visualized during the refinement process.

We believe this design provides a different and practical perspective for scene representation in autonomous driving.

Q2. The evaluation is based on single-seed results without any variance estimates, error bars, or confidence intervals.

Thank you for your valuable comment. Following your suggestion, we conducted multi-seed experiments (three times) on both the NAVSIM and Bench2Drive benchmarks. As shown in the tables below, we report the mean and standard deviation.

Table 1. Performance on the Navtest Benchmark.

Table 2. Performance on the Bench2Drive benchmark. '*' denotes the results are reproduced by ourselves.

On the NAVSIM dataset, our method demonstrates low variance, indicating strong robustness to random initialization. On the closed-loop Bench2Drive benchmark, although the variance is relatively higher, our method still consistently outperforms baseline methods on average, highlighting its stability and effectiveness across diverse conditions.

We will include these aggregated results and variance estimates in the final version of the paper.

Q3. The performance of Gaussian sparse representation on small and moving objects?

The proposed 2D Gaussian sparse representation is capable of effectively capturing objects of various sizes, including small and dynamic ones, due to its deformable and adaptive nature. Unlike BEV-based dense representations, which may suffer from resolution constraints or fixed grid structures, our 2D Gaussians allow for flexible allocation of representation capacity to fine-grained details.

As shown in Table 4, our method achieves higher semantic map quality compared to BEV-based baselines. Furthermore, as demonstrated in the supplementary demo video, the 2D Gaussians successfully capture moving objects such as pedestrians and vehicles, indicating their effectiveness in dynamic driving scenes.

Q4. How does or would the GaussianFusion perform under real-world challenges such as fog, occlusions, or sensor noise? Any plans to test beyond simulation?

We appreciate the reviewer’s thoughtful question. The robustness of GaussianFusion under adverse weather conditions has been partially validated using the CARLA-based Bench2Drive benchmark, which includes diverse environmental settings such as fog, rain, and cloudy skies. As shown in the supplementary demo video, our method performs robustly in a foggy and rainy scenario, indicating strong adaptability to challenging visibility conditions.

However, we have not explicitly evaluated the method under severe occlusions or sensor noise. We agree that assessing performance under these conditions is crucial for real-world deployment. We sincerely thank the reviewer for this insightful suggestion and will explore these factors in future work.

Regarding real-world testing, due to the high cost and complexity of such experiments, we currently do not have plans to move beyond simulation. Nevertheless, we believe that the insights gained from simulation benchmarks like Bench2Drive provide a valuable proxy for many real-world challenges.

Q5. Do the implicit and explicit branches ever conflict or interfere? How is their interaction managed during training?

We did not observe any conflicts or interference between the implicit and explicit branches in our experiments. On the contrary, as shown in the Table 3, incorporating the implicit branch consistently improves performance, indicating that the two branches provide complementary information rather than competing signals.

In our current implementation, their interaction is managed by simply concatenating their outputs in the trajectory prediction module. While this approach proves effective in our setting, it is not necessarily optimal. We plan to explore more effective interaction mechanisms in future work.

Q6. Table and figure captions lack clarity and exhibit inconsistent bolding.

Thank you for your helpful comment. We will carefully revise all table and figure captions to improve clarity, ensure consistency in formatting, and enhance overall readability in the final version of the paper.

Q7. The claim that GaussianFusion is more interpretable is kind of intuitive, but no formal interpretability analysis (e.g., user study, attribution) is provided, which could be beneficial for the contribution framing.

We agree with the reviewer that our paper lacks a formal interpretability analysis. In this work, our primary focus is on validating the effectiveness of the GaussianFusion framework in the planning task.

We argue that the proposed 2D Gaussian representation is inherently more intuitive than flattened feature vectors or dense BEV grids. It allows for visual inspection of spatial changes during the feed-forward process, which provides a more direct understanding of what the model attends to in a scene.

We acknowledge that incorporating formal interpretability evaluations (e.g., attribution maps or user studies) could strengthen our contribution, and we plan to explore this direction in future work.

Q8. Incorrect citation formatting: References should be ordered numerically and separated by commas.

Thank you for your kind suggestion. We will carefully revise all in-text citations to ensure they are numerically ordered and properly formatted, following the required style in the final version of the paper.

References

[1] Kerbl B, Kopanas G, Leimkühler T, et al. "3D Gaussian splatting for real-time radiance field rendering". ACM Transactions on Graphics, 2023.

[2] Huang Y, Zheng W, Zhang Y, et al. "Gaussianformer: Scene as gaussians for vision-based 3d semantic occupancy prediction". ECCV, 2024.