3D Perception with Differentiable Map Priors

We thank the reviewer for the detailed review and thoughtful questions about the technical implementation of our approach.

Presentation

We greatly appreciate the reviewer's constructive feedback and suggestions. While both Figure 1 and Figure 5 show examples of traversal overlap, they serve distinct purposes - Figure 1 provides a high-level motivating example while Figure 5 demonstrates the exact data distribution used in the official split and our experiments. For Figure 2, we thank the reviewer for pointing out the ambiguity of "global map" and have updated the caption to clarify that it refers to the learned map prior in the revised manuscript. Regarding Table 1, we have modified the in-text citation style and specified that bold numbers indicate the best performing model across all methods (up to the last significant digit).

Mixed Individual Metrics

Thank you for the keen eye! We'd like to clarify that metrics like mASE, mAOE, mAAE, and mAVE are computed only on true positive detections. In 3D object detection, these metrics may appear worse when a model achieves higher recall, as it successfully detects more challenging objects. This is why the community primarily relies on NDS and mAP as key indicators of overall system performance.

Here we show one particular instance from DETR3D [1]:

Despite achieving higher NDS and mAP, the mATE and mASE are comparable while the mAAE is worse. This pattern is also observed in other strong object detection architectures [2,3].

"How to avoid hash collision for finer resolutions?

While hash collisions do occur at finer resolutions, we hypothesize the MLP decoder that follows the lookup learns to resolve these collisions during training. Such behavior has been noted in Instant-NGP [4].

"How does the method know when to fall back to the baseline and when to use the map prior?"

In our experiments, we use a simple binary indicator (computed from historical data statistics) to determine whether the current location has been visited before.

"are DMP and detectors trained together? ... how to make sure DMP is not bypassed?"

Our approach trains DMP and the detector end-to-end, maintaining all parameters as trainable. The model learns to balance the use of immediate perception and prior information - when current observations provide sufficient information, the model appropriately reduces its reliance on the prior. This adaptive behavior is similar to observations in multi-sensor fusion systems (lidar, cam, radar), where models learn to balance information from different sensor modalities.

References:

[1] Z. Wang, et al., "DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries," CoRL 2022.

[2] Y. Liu, et al., "PETR: Position Embedding Transformation for Multi-View 3D Object Detection," ECCV 2022.

[3] Z. Li, et al., "BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers," ECCV 2022.

[4] T. Müller, et al., "Instant neural graphics primitives with a multiresolution hash encoding," ACM Transactions on Graphics, 2022.

We thank the reviewer for the thoughtful review and constructive suggestions for improving our work. We have revised the manuscript to explicitly specify the map prior representation is learned and does not rely on HD-map data.

"Ablation studies related to map representation

Thank you for the suggestion! We have revised the manuscript to explicitly specify that our map prior representation is learned and does not rely on HD-map data. HD-maps have been shown to improve object detection performance in prior work [1]. We are currently running additional experiments on HD-map enhanced camera-based baselines on nuScenes and aim to include these results before the end of the rebuttal period.

"Comparison to other end-to-end driving methods (UniAD or VAD)"

UniAD and VAD are end-to-end driving frameworks that target planning while incorporating multiple tasks including detection, tracking, map prediction, motion forecasting and planning. After carefully examining their published results and code, we find that unfortunately neither method reports their 3D object detection performance.

While our current work focuses on detection, the reviewer's suggestion inspires a promising future direction of extending our differentiable map prior to tackle end-to-end driving tasks.

Motivation of Table 2 Comparison

We agree with the reviewer that NMP's original setting of online map prediction differs from ours. However, their external memory approach serves as a strong baseline for improving perception using past information. The comparison highlights key differences in approach - while both methods leverage historical information for perception from multi-view camera inputs, we aim to demonstrate how our learned prior design improves performance compared to NMP's formulation.

Additional Related Works

Thank you for bringing these papers to our attention. While these papers share the high-level idea of incorporating map information, they differ in their objectives and technical approaches. MapEX and P-MapNet focus on estimating and generating HD maps by leveraging existing map priors (manually annotated HD maps, SD maps), while our work learns priors from historical traversals to improve 3D object detection performance. We have include discussion of these works in our revised manuscript to better position our contributions within the literature.

References:

[1] B. Yang, M. Liang, and R. Urtasun, "HDNET: Exploiting HD Maps for 3D Object Detection," CoRL 2018.

We thank the reviewer for the thorough review and the feedback on our work.

"What happens if the image information provided to the model gets degraded in some way"

Great idea! We performed experiments to simulate sensor corruption by zeroing out a random subset of the 6 cameras and reporting NDS.

Here, 0 cameras dropped refers to the original results, and 6 cameras dropped is the limit experiment. As we can see, the map prior helps throughout, but more interestingly, the performance is non-zero in the limit.

Additional Hashmap Memory Statistics

We thank the reviewer for the suggestion! To help quantify the memory footprint, we use Boston (6.39 km^2) as a concrete example. For our configuration with a 4 level hash table with feature dimension $d = 32$ (where each level contains an 8-dimensional vector), maximum number of embeddings $T=2^{16}$ per level, and $r_{min} = 0.5m, r_{max} = 25m$, the map prior embedding (stored as 4 byte floats) requires only ~12.3 MB, where most of the memory savings come from the bounded hash table size $T$. In contrast, a "dense" 2D grid representation with $d=32$ at 0.5m resolution would require $6.39 km^2 * (1000 m / km)^2 / (0.5^2) * 32 * 4\ bytes = 3.1 \ GB$.

"Is it (the learned prior) 3D or 2D?"

The hashmap operates in 2D BEV space. Since it builds upon representation from the neural rendering literature, we borrowed the term "voxel", but are happy to use "cells" throughout the paper if this better conveys the 2D nature of our approach.

"How are the 4 levels spaced in voxel resolution?"

The resolution grows exponentially per level, with resolutions computed as $r_i = r_{min} * g^i$, where the growth rate $g = (r_{max}/r_{min})^{1/(L-1)}$. For the configuration used in our experiments:

• Level 0: 0.5m
• Level 1: ~1.8m
• Level 2: ~6.7m
• Level 3: 25m

"Am I right to assume that $T=2^{16}$ means that each side of the 2D grid is $2^8$=256 pixels/voxels wide?"

The parameter T=$2^{16}$ controls the maximum size of the hash table embedding at each level, acting as an explicit bound for the memory footprint. The grid size is determined by the resolution of the level and the desired area of coverage.

"How does the approach handle dynamic objects?"

We kindly refer the reviewer to see our general response above.

We thank the reviewer for their time and feedback.

"The proposed approach, however, does not use the historical information at all. It simply uses the geolocation within the global map to learn point-wise embeddings"

We respectfully believe there is a misunderstanding. We use historical information during training to build up a global map representation (map prior). This map prior is then used at inference to improve model performance.

These point-wise embeddings are precisely how we distill and store historical information from multiple traversals. Through end-to-end training, the spatial embeddings learn relevant features observed over time at that position, effectively encoding historical context.

"short introduction with unclear differentiation from previous approaches"

We aimed for conciseness in our introduction and have expanded the introduction to more explicitly differentiate our work from prior approaches. As the reviewer noted, the way we encode the historical information as point-wise embeddings is the key differentiator from previous methods.

"Why can the proposed map prior improve object detection?"

Please see our general response above.

"How is NMP adapted for object detection?"

We adapted NMP by integrating the DETR detection head used in the BEVFormer and BEVFormer + DMP experiments. The detection head uses a set of learned query anchors to attend to BEV features through a series of transformer decoder layers to predict 3D boxes. We are happy to provide more details and include this in the Appendix.

"What does 'incorporate DMP into three distinct multi-view perception stacks' mean?"

This refers to integrating DMP with three different perception architectures - BEVDet, BEVFormer, and PETR - to evaluate its performance across various model designs and losses.

As the rebuttal period is nearing its end, we would value the reviewer's feedback on our earlier responses to your concerns. Please let us know if any points need further clarification.

We thank reviewers tfjT and 1jMR for bringing up the baseline comparison with traditional map priors. We have conducted additional experiments to directly compare DMP against these explicit map priors.

In these experiments, we rasterize the map annotations for "road_divider", "lane_divider", "pedestrian_crossing", "road_segment", and "lane" classes to be used as the map prior. To ensure a fair comparison, we use a linear projection + the same Convolutional Fusion block from our method to integrate both these traditional map priors and our learned priors with the sensor features.

The results demonstrate that while traditional map priors do provide modest improvements, our fully learned priors achieve stronger performance gains without requiring additional map annotations.