SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving

We thank you for the careful evaluation of our work and the valuable feedback. We address each comment in detail below:

1. The Core Technical Novelty

Thank you for your feedback and for highlighting the scope of our contributions. We acknowledge that 3D Gaussian Splatting and query-based architectures are established techniques. However, as the reviewers DKKC and uHvD concluded, our primary focus is on the novel integration of these components within a self-supervised pre-training framework specifically tailored for Sparse Perception Models (SPMs) in autonomous driving. The plug-and-play Gaussian queries can be conveniently integrated within SPMs using the proposed Query Interaction mechanism, leading to improved downstream performance without reliance on extensive labeled data.

We hope to clarify the novelty of our approach in the revised manuscript and appreciate your constructive comments that help us improve the presentation of our work.

2. Cross-dataset Generalization Analysis

We appreciate your insightful comment regarding the cross-dataset evaluation of our method. While our current study primarily focuses on the nuScenes dataset, we selected it due to its comprehensive sensor setup and rich annotations. We agree that evaluating across datasets would show the generalization capabilities of our approach.

To further evaluate, we conduct experiments on the KITTI-360[1] dataset for 3D semantic occupancy prediction with monocular camera, which differs significantly in sensor setup compared with nuScenes. GaussianFormer is selected as the Sparse Perception Model (SPM) to be improved with our SQS. We use the dense semantic occupancy annotations from SSCBench-KITTI-360[2] for supervision and evaluation. We utilize all training data for pre-training and only half data for fine-tuning due to the time limitation.

After being integrated with our method, the GaussianFormer obtains 0.48 mIoU and 1.74 IoU improvements, achieving 7.70% mIoU and 29.70% IoU, when compared to the 7.22% mIoU and 27.96% IoU without SQS. The results successfully demonstrate the generalization of our SQS. Please note that the results could be further improved through hyperparameter tuning, provided that sufficient time is available for experimentation. We will update these results in our revised version of the paper.

3. Compare with SOTA Methods

Thanks for your insightful comment. We agree that comprehensive comparisons are important. For the 3D semantic occupancy prediction, we evaluate SQS using the SurroundOcc benchmark, and all possible recent baselines in this dataset are included for comparison. For the 3D object detection, we primarily choose SparseBEV as the baseline, given its balanced performance and model complexity. In the revised version, we plan to include results from Sparse4Dv3 and other more recent 3D detection methods in the table.

Importantly, through comparison, we aim to demonstrate that using SQS can significantly improve the performance of current popular SOTA SPMs, rather than solely striving for SOTA metrics in 3D Occupancy and Detection tasks.

4. More Accessible Explanation for The Query Interaction Mechanism

Thank you for your valuable feedback. We acknowledge that the description of the query interaction mechanism is complex and may benefit from additional clarification. To address this, we will include a diagram illustrating the process flow and step-by-step explanations to enhance clarity in the revised version. The step-by-step explanations are listed as follows:

1. Infer the pre-trained Gaussian queries paired with anchors using the fixed pre-trained model;

2. Obtain Gaussian queries from step 1 by filtering out corresponding anchors with low opacity;

3. In the downstream SPMs, each task queries only aggregate features from $k$ closest 3D Gaussians selected from step 2 by applying $k$-nearest neighbor algorithm.

[1] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292–3310, 2022.

[2] Yiming Li, Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang, Zhiheng Li, Tao Jiang, Fisher Yu, et al. Sscbench: A large-scale 3d semantic scene completion benchmark for autonomous driving. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13333–13340. IEEE, 2024.

We thank you for the careful evaluation of our work and the valuable feedback. We address each comment in detail below:

1. Clarity on Query Training and Fine-tuning Procedure with Figure 2

We appreciate the reviewer’s concern regarding the clarity of our description about sparse query training and their role during fine-tuning. We recognize that additional textual explanation could complement Figure 2 and improve understanding.

To clarify, our proposed SQS introduces a plug-in module that predicts 3D Gaussian representations from sparse queries via a Gaussian Transformer Decoder in the top row of Figure 2 during pre-training, leveraging self-supervised splatting to learn fine-grained contextual features through the reconstruction of multi-view images and depth maps. In the downstream fine-tuning stage, we could share the pre-trained image encoder weights for downstream model initialization, and the pre-trained Gaussian queries are seamlessly integrated into downstream networks via query interaction in the Transformer Decoder of sparse perception models (SPMs) to enhance the performance.

In the revised version, we will further elaborate on the query interaction pipeline, supplement Figure 2 with additional explanatory notes, and provide a step-by-step procedure in the main text for greater clarity.

2. Generality of SQS Across Backbone Models

Thank you for pointing out the importance of thoroughly demonstrating the model-agnostic claim of SQS. As you observed, our current experiments focus on integrating SQS with GaussianFormer for occupancy prediction and SparseBEV for the 3D object detection task. And in Table 2, we have conducted experiments on two different image backbones of SparseBEV to validate the effectiveness of our method.

To further evaluate the generality of SQS, we have performed additional experiments, integrating SQS into another sparse perception model e.g., Sparse4Dv3, as pointed by Reviewer uHvD. On the nuScenes validation set, under limited training (using only 1/4 of the training data to accelerate training), Sparse4Dv3 achieves 29.5 mAP and 36.1 NDS. Incorporating our SQS pre-training module leads to further improvements, reaching 30.4 mAP and 37.3 NDS. These new results combined with the results of SparseBEV could further validate the generality of SQS across different backbone models. The final full dataset results will be detailed in our final version.

These results validate the model-agnostic property of the SQS framework and demonstrate its stable performance enhancement across architectures. We greatly appreciate the reviewer’s suggestion and agree that this substantially increases the solidity and generalizability of our method.

Thank you for the positive comments and the update! We’re pleased to know that your main concerns have been resolved and we will include the additional analyses in the final version.

We sincerely thank the reviewer for the detailed feedback and the opportunity to clarify and strengthen our work. We address each point below:

1. Comparison with Sparse4Dv3 Performance

We appreciate the reviewer’s concern about the lack of comparison to Sparse4Dv3. Our initial choice of SparseBEV as the primary baseline was motivated by its balanced design in terms of detection accuracy and model complexity, and because sparse detection frameworks generally share similar architectural paradigms. While the Sparse4D series has indeed introduced innovations in temporal fusion, many core mechanisms of sparse detection remain comparable across these approaches, making SparseBEV a widely recognized benchmark.

To further enhance the rigor of our work and address this omission, we have conducted additional experiments incorporating Sparse4Dv3. On the nuScenes validation set, under limited training (using only 1/4 of the training data), Sparse4Dv3 achieves 29.5 mAP and 36.1 NDS, while incorporating our SQS pre-training module leads to improvements, reaching 30.4 mAP and 37.3 NDS. These results demonstrate that our pre-training framework also benefits more advanced architectures like Sparse4Dv3, and thus our findings are not specific to a single backbone or detection framework. We will update the final version of our paper with full-scale results.

2. Latency and Memory Analysis

Thank you for drawing attention to the question of real-world applicability. We recognize the importance of benchmarking computation and memory efficiency. Our SQS framework is designed with plug-and-play considerations: the extra computational overhead is predominantly introduced during pre-training via the query interaction module. During downstream fine-tuning and inference, it is possible to selectively activate or entirely omit the query interaction module. For most deployment scenarios, loading only the pre-trained image encoder introduces negligible additional latency or memory consumption compared to vanilla baselines, while still reaping the representational benefits of pretraining.

We have conducted preliminary latency and memory profiling with different variants of the SQS modules across the occupancy prediction baseline (i.e., GaussianFormer) as responded to Reviewer DKKC, and observed that: The SQS-pretrained image encoder adds no extra inference latency when the query interaction module is omitted at deployment. When using the partial module sharing strategy, the latency increases by approximately 30%, and GPU memory consumption increases by 35% compared to baseline, with a notable performance boost on downstream tasks. These findings suggest that our method is not only flexible, but also practical for scalable real-world applications. We will provide detailed results and hardware configurations in the final version of our paper.

3. Study and Visualization of Query Reuse Across Tasks

We appreciate the reviewer’s insightful question regarding the quantitative analysis and visualization of query reuse across tasks, as well as how such reuse impacts downstream performance.

In our current implementation, the query interaction mechanism adopts a purely adaptive self-attention framework, where all pre-trained queries are dynamically weighted and selected according to the learned attention scores. This design enables flexible transfer of information but does not explicitly label or fix certain queries for reuse in particular downstream tasks. As such, it is inherently challenging to unambiguously trace and quantify “reused queries” or directly visualize their task-level roles, since query participation is emergent and distributed rather than discrete or statically assigned.

We acknowledge this limitation in our analysis. At the same time, we believe this adaptivity is a strength: self-attention enables each downstream task module to discover and leverage the most relevant subset of features, rather than hard-coding query-task assignments during pre-training. Consequently, improvements in downstream performance—in both detection and occupancy prediction—serve as indirect evidence that beneficial query reuse and knowledge transfer are indeed taking place.

For future work, we plan to conduct more comprehensive interpretability studies. For instance, techniques such as analyzing the learned attention distributions, performing query ablation, and visualizing per-query activation maps in downstream heads will likely yield more direct insights into how queries are reused or specialized for specific tasks. We see this as a promising direction for ongoing and follow-up research, and we are grateful to the reviewer for this suggestion.

We sincerely thank you for your careful evaluation and constructive feedback. We address each specific concern below:

1. Table and Figure Readability (Table 1, Figure 4):

Thank you for highlighting the readability issues. In the final version, we will enlarge Table 1 and reformat it to improve its clarity and readability. For Figure 4, we will adjust the color mapping and background of the top-right figure, to enhance visual contrast and ensure details are more easily discernible.

2. Fully Utilization of Pre-training Queries and Challenge Discussion:

We mentioned that the current design does not fully exploit the pre-training queries for different downstream tasks in the limitation section. As described in Sec. 3.3 of our main paper, SQS leverages a self-supervised splatting module to extract contextual 3D Gaussian representations from sparse queries for reconstructing the RGB images and depth maps of the whole scenario in the pre-training stage. However, even though our SQS is a generic plug-in method for sparse perception models (SPMs), the role of these queries diverges depending on the downstream applications. For example, in occupancy prediction, queries represent both foreground and background, while in object detection, only foreground queries are learned. While in SQS, the query interaction module is utilized to learn from pre-trained queries adaptively without considering this discrepancy, achieving promising performance improvements.

A potential direction to address this is to introduce explicit semantic information (e.g., pseudo-labels or clustering-based semantic grouping) during pre-training to annotate queries according to their semantic categories. This would enable more adaptive and targeted query interaction in downstream tasks, allowing the selection or weighting of queries based on semantic relevance. Such approaches could facilitate more effective knowledge transfer and potentially enhance performance, particularly in tasks requiring fine-grained discrimination.

3. Incorporating Semantic Information in Pre-training:

Following the above, incorporating semantic information during pre-training can enhance the query representation’s semantic awareness. This could be achieved by associating each query with semantic pseudo-labels, derived from clustering on features or through auxiliary losses related to semantic segmentation or object proposals. Introducing this semantic distinction during pre-training would allow queries to encode more discriminative and task-relevant features, boosting their transferability and downstream task adaptability. For instance, in the object detection head, only queries pre-trained with ‘foreground’ semantics would participate in the detection decoder, leading to improved sample efficiency and interpretability.

4. Computational Overhead and Possible Solutions:

We acknowledge the concern about additional computational overhead and memory consumption, mainly due to the plug-in mechanism introduced by SQS, as illustrated in the table below for the occupancy prediction task with the quarter of training data during the pre-training and fine-tune stage. To address this, we propose several feasible strategies:

• Tight Computation Application: In scenarios with tight computational constraints, users can opt to load only the pre-trained image encoder (backbone and neck) and omit the query interaction module during fine-tuning (SQS* in table below). This setup incurs no extra overhead compared to conventional models while still benefiting from the representational improvements gained through pre-training.

• Partial Module Sharing: It is also feasible to share the image encoder between the Gaussian queries (for query interaction) and downstream task queries, only enabling the Gaussian Transformer Decoder during fine-tuning. As detailed in the table below for SQS**, this results in a modest overhead but provides a favorable accuracy-resource trade-off.

• Future Optimization: We are actively exploring model compression and distillation techniques that can further reduce memory consumption and computation, without sacrificing performance.

We hope these clarifications directly address your questions and demonstrate our commitment to maximizing both scientific contribution and practical accessibility.