Uni-Map: Unified Camera-LiDAR Perception for Robust HD Map Construction

Thank you for reviewing our paper and for your positive feedback, especially your comment describing our method as "meaningful to the field of autonomous driving and novel." We will integrate all your suggestions in the revised version of the paper.

Q1: Theoretical analysis and insight regarding the method.

A1: Thank you for your valuable feedback regarding the theoretical analysis. We appreciate your insights and would like to clarify our insights further:

Insight of the Projector Module: While in the same space, camera BEV features, LiDAR BEV features, and fused BEV features can still be misaligned to some extent due to the inaccurate depth in the view transformer and the large modality gap (See Fig.7 (a)). To address this, we propose a projector module to align BEV features from different modalities into a shared space thereby enhancing representation learning. As shown in Fig.7, before the Shared Projector module, the camera BEV and LiDAR/Fused BEV features are clearly separated, however after the shared projector module, the BEV features from different modalities are well aligned in a shared/similar space. Therefore, designing a projector module to align BEV features from different modalities into a shared space is essential for enhancing the overall performance of our model.

Insight of the MSM Scheme: First, by stacking BEV features from different modalities that share the same map decoder and ground truth labels, the projector module is supervised (via gradient back-propagation) to implicitly align BEV features from different modalities in the shared feature space. Second, inputting stacked BEV features into the same map decoder increases the diversity of the BEV feature space accessible to the decoder module, thereby improving the model’s generalization ability and robustness across different input configurations. Third, this scheme allows the map decoder module to process BEV features of different modalities. As a result, Uni-Map can flexibly handle various input configurations during inference.

We hope this explanation clarifies the theoretical insights behind our method and addresses your concerns.

Q2: Why the switching modality strategy can be seamlessly adapted to arbitrary modality inputs?

A2: Thanks for your comments. The Uni-Map model trained with camera and lidar data is evaluated with different input configurations (camera/lidar/camera & lidar). Specifically, during inference, our model utilizes a switching modality strategy to seamlessly adapt to arbitrary modality inputs, ensuring compatibility across various input configurations. The switching modality strategy can be formulated as Eq. 6. This switching strategy simulates real-world scenarios where sensors may be missing during the inference phase. As illustrated in Fig. 4, the map decoder can adapt to various input scenarios: camera BEV features are used when LiDAR data is unavailable due to removal or damage, LiDAR features are used when camera data is absent, and fused BEV features are employed when both data sources are available. This flexibility enables the decoder to handle individual modality features or their fusion seamlessly. As a result, Uni-Map supports all three input configurations, enhancing its practicality for autonomous driving. Notably, this modality-switching strategy does not affect inference speed or memory usage. Experimental settings will be detailed in the revised version.

Q3: Why MLPs in Eqs. (1-3) work well?

A3: Thanks for your comments. First, by stacking BEV features from different modalities that share the same map decoder and ground truth labels, the projector module is supervised (via gradient back-propagation) to implicitly align BEV features from different modalities in the shared feature space. After alignment, we concatenate the features along the batch, and inputting the stacked BEV features into the same map decoder enhances the diversity of the BEV feature space, improving the model’s generalization and robustness across various input configurations. As can be seen, Fig.7(a) Before Projector module shows that camera BEV (blue) and LiDAR/Fusion BEV (red/green) features are clearly separated, indicating that although in the same space, camera BEV features, LiDAR BEV features, and fused BEV features can still be misaligned to some extent due to the inaccurate depth in the view transformer and the large modality gap. Fig.7 (b) After the projector module, the BEV features from different modalities are well aligned in a shared/similar space, i.e., red (Fused BEV), blue (Camera BEV), and green (LiDAR) circles are close together after the projector module. Moreover, Tab.3 shows that we can significantly improve the model performance by training the model with the projector module. Therefore, it is demonstrated that the MLP-based projector work well and can align BEV features from different modalities into a shared space.

Thank you for reviewing our paper and for your positive feedback, particularly your comment that our method exemplifies 'High-Quality Quality and Important Significance.' We will incorporate all your suggestions into the revised version of the paper.

Q1: How does the modality switching operate in practice? Is it automated, or does it require prior information about which modality to use, particularly in corruption cases?

A1: Thanks for your comments. First, we do not require prior knowledge of the corrupted modality. All results reported in Figures 5/9 and Tables 12-17 are indeed obtained using the Uni-Map Camera-LiDAR Fusion model. Moreover, Uni-Map Fusion model shows stronger robustness against the 13 types of camera-LiDAR corruptions we designed, compared to the MapTR Fusion model and HiMap Fusion (detailed results are in Figures 5 and 9). Please note that the modality switching operates automatically and is only triggered when data from one modality is unavailable. Implementing this strategy can enhance performance in such cases by allowing the model to rely on available data from the other modality.

Q2: Was corrupted data used solely for zero-shot analysis, or was it included in training? If included, was the MapTR baseline also trained with this data?

A2: Thanks for your comments. The corrupted data was used solely for zero-shot analysis and was not included in the training process.

Q3: Could the authors clarify the total training data used (Table 2) and whether the reported performance improvements may be due to the increased sample count from MSM, rather than intrinsic improvements in the methodology?

A3：Thank you for your insightful question. The training data aligns with the nuScenes training dataset. Our all-in-one model utilizes both camera and LiDAR modalities, identical to MapTR-F. The key difference lies in the decoder’s input features: our model reuses camera and LiDAR features along with the fused representation, incurring minimal additional computation. This overhead is significantly less than the threefold increase required for tripling the encoder’s input (21h57m vs. 47h12m), as shown in the table below. Nevertheless, we trained the MapTR-F model using three times the data volume by increasing the number of epoches, and the experimental results are as follows. Despite this increase in data, our Uni-Map Fusion(mAP:68.1) still outperforms the MapTR-F (mAP:65.4) model. The performance improvement is primarily attributed to MSM concatenates the features along the batch dimension, allowing the stacked BEV features to be input into the same map decoder. This approach enhances the diversity of the BEV feature space, ultimately improving the model’s generalization and robustness across various input configurations. Therefore, merely increasing the amount of data will not lead to the reported performance gains. The novel MSM module and projector module introduced in this paper are crucial for enhancing performance and robustness.

Thank you for reviewing our paper and for the positive feedback, particularly your comment that our method is ‘motivated and effective’. We will incorporate all your suggestions into the revised version of the paper.

Q1: From the Table 1 in Experiment section, previous state-of-the-art method for HD map construction is HIMap, which is different from the statement in Line 32 and Line 96.

A1: Thank you for your insightful feedback. In the current version of our paper, lines 32 and 96 present experimental results related to the popular HD map construction method-MapTR. To clarify this in the revised version, we will make the following modifications:

Line 32 will be revised to: Notably, our unified model surpasses independently trained camera-only, LiDAR-only, and camera-LiDAR state-of-the-art HIMap models with gains of 1.5, 6.5, and 2.4 mAP on the nuScenes dataset, respectively.

Line 96 will be revised to: Our single Uni-Map model outperforms the state-of-the-art HIMap models that are independently trained on camera-only, LiDAR-only, and camera-LiDAR fusion modalities, achieving gains of 1.5, 6.5, and 2.4 mAP, respectively. We appreciate your suggestion and believe these changes will enhance the clarity of our findings.

Q2: Is it appropriate to put Figure 9 from the Appendix to the main body of the paper and put Table 4 to the Appendix?

A2: Thank you for your suggestion regarding the placement of Figure 9 and Table 4. We agree that moving Figure 9 to the main body of the paper could enhance its visibility and relevance to the discussion. Conversely, we will relocate Table 4 to the Appendix, as it contains supplementary information that may not be essential to the main narrative. We appreciate your input and will make these adjustments in the revised version.

Thank you for reviewing our paper and for your positive feedback, particularly your comment that our method exemplifies 'Effective Experiments and Clear Structure'. We will incorporate all your suggestions into the revised version of the paper.

Q1： The motivation of our paper.

A1:Thanks for your comments. We first figure out two critical issues in the current HD map construction task: high costs due to separate training and deployment for each input configuration, and low robustness when sensors are missing or corrupted. To address these issues, we propose the first high-performance all-in-one solution, i.e., one model for various input configurations (camera-only, LiDAR-only, and camera-LiDAR Fusion). The core components of our method (Uni-Map) include the Mixture Stack Modality (MSM) training scheme and the projector module. Specifically, MSM inputs stacked BEV features into the same map decoder, increasing the diversity of the BEV feature space accessible to the decoder module, thereby improving the generalization ability and robustness of the model in different input configurations (See Tab. 3 and Fig. 5/9). Furthermore, we propose a simple yet effective projector module to align BEV features from different modalities into a shared space (See Fig. 7). As shown in Fig. 7, before the Shared Projector module, the camera BEV and LiDAR/Fused BEV features are clearly separated. After the shared projector module, the BEV features from different modalities are well aligned in a shared/similar space. In addition, we also conducted a thorough ablation study to demonstrate the effectiveness of the Uni-Map model's core modules MSM scheme (See Tab. 3) and Shared Projector module (See Tab. 4). Overall, the Uni-Map model achieves significant performance improvements over SOTA approaches (See Tab. 1) and stronger robustness (See Fig. 5/9), with less training time and fewer parameters for various input configurations (See Tab. 2), while maintaining the same inference speed and memory footprint (See Appendix Tab. 6 and Tab. 7).

Q2: Are the stack operations on BEV features merely concatenation, and how do they differ from sensor-fusion methods like BEVFusion[1] or PointAugmenting[2]?

A2: Thank you for your feedback. References [1] and [2] primarily focus on fusing features from different modalities, with the goal of creating a single fused feature map that effectively utilizes the complementary information between these modalities. In contrast, our method focuses on aligning features from different modalities (camera, LiDAR, and their fusion), aiming to achieve a unified model that can effectively handle various input modality configurations. The technical details are as follows: In the BEVFusion model, camera and LiDAR BEV features are concatenated along the feature dimension and processed through a convolutional network to generate fused features for prediction. BEVFusion primarily focuses on creating these fused features. In contrast, our approach passes camera BEV features, LiDAR BEV features, as well as fused BEV features through a projector module to align them in the shared feature space. After alignment, we concatenate the features along the batch dimension, resulting in a 3B batch that is sent to the prediction head. As outlined in the Implementation section (lines 318-320), during training, ground truth labels are duplicated to match the stacked feature map from Equation 4. In summary, the key difference lies in the BEV encoder design of BEVFusion, which uses convolution to fuse modalities across the channel dimension. In contrast, our projector module is specifically designed to align modalities (camera, LiDAR, and fused BEV features) along the batch dimension, representing a fundamentally different approach. We appreciate your attention to this distinction.

Q3: How to verify the MLP-based projector can achieve the alignment?

A3: Thanks for your comments. First, by stacking BEV features from different modalities that share the same map decoder and ground truth labels, the projector module is supervised (via gradient back-propagation) to implicitly align BEV features from different modalities in the shared feature space. After alignment, we concatenate the features along the batch, allowing the stacked BEV features to be input into the same map decoder, which enhances the diversity of the BEV feature space and improves the model’s generalization and robustness across different input configurations. As can be seen, Fig.7(a) Before Projector module shows that camera BEV (blue) and LiDAR/Fusion BEV (red/green) features are clearly separated, indicating that although in the same space, camera BEV features, LiDAR BEV features, and fused BEV features can still be misaligned to some extent due to the inaccurate depth in the view transformer and the large modality gap. Fig.7 (b) After the projector module, the BEV features from different modalities are well aligned in a shared/similar space, i.e., red (Fused BEV), blue (Camera BEV), and green (LiDAR) circles are close together after the projector module. Moreover, Tab.3 shows that we can significantly improve the model performance by training the model with the projector module using the RSM or MSM training scheme. Therefore, it is demonstrated that the MLP-based projector work well and can align BEV features from different modalities into a shared space, thereby enhancing representation learning.

Q4: I think the misalignment is mainly caused by the inaccurate Camera-to-BEV View Transform process? Why not apply the projector only on the camera BEV feature?

A4: Thanks for your comments. The misalignment arises from both spatial mismatches due to inaccuracies in camera-to-BEV transformations and sensor calibration, as well as a substantial modality gap between camera images and LiDAR point clouds. Consequently, projecting camera BEV features into a fixed LiDAR space, by solely applying the projector exclusively to the camera BEV features, is challenging to achieve effective alignment. This paper introduces a projector module to align BEV features from different modalities into a shared space, which is crucial for enhancing the overall performance of our model.

Q5: About the Inference Phase of the switching modality strategy, what training strategy (e.g., loss functions) is used to ensure the effectiveness of the switching modality strategy during inference?

A5：Thanks for your comments. During training, we employ the MSM strategy, which effectively ensures the performance of the switching modality during inference without the need for additional strategies. This is primarily because MSM concatenates the features along the batch dimension, allowing the stacked BEV features to be input into the same map decoder. This approach enhances the diversity of the BEV feature space, ultimately improving the model’s generalization and robustness across various input configurations. As detailed in the Implementation section (lines 318-320), during training, ground truth labels are duplicated and stacked to create a 3B batch dimension, aligning with the stacked feature map from Equation 4. Thank you for your insightful question.

Thank you for your insightful follow-up comments.

Q1: Why did you replace the conv2D operations in BEV used in BEVfusioN/pointaugmenting with the MLP network? What are the advantages of MLP over conv2D for alignment?

A1: The replacement of Conv2D operations with an MLP is primarily for effective feature transformation. MLPs, similar to the feedforward neural networks in transformers, excel at learning complex, non-linear mappings while maintaining the original input structure, thus preserving spatial context. They also offer a global perspective, are easier to implement, and adapt well to irregular data, making them more suitable for alignment tasks in multi-modal scenarios.

Q2: Have you used masking strategies in training, such as randomly masking one modality in training? If one modality is unavailable, do you need to reload the parameters in the map decoder?

A2: To systematically evaluate the effectiveness of the MSM training scheme, we train the model using different schemes and report the mAP results in Tab. 3. In addition to MSM, we introduced the Random Select Modality (RSM) training scheme, which randomly selects inputs from one of the BEV feature maps—either Camera BEV features, LiDAR BEV features, or Fused BEV features—effectively acting as a masking strategy. The results of the RSM training scheme are inferior to the MSM training scheme under both settings (with and without the Projector). This demonstrates the MSM training scheme's advantage in enhancing the map decoder's effective use of camera, LiDAR, and fused features. This increases the diversity of the BEV feature space, resulting in a high-performance integrated model. Moreover, if one modality is unavailable, we will perform inference using the BEV feature from the other modality, employing the switching modality strategy described in formula (6). No parameter reloading is needed since we have only one model for various input configurations.

We will revise the final version according to these constructive discussions. Thanks again and feel free to inform us for any further questions and discussions.