P-MapNet: Far-seeing Map Constructer Enhanced by both SDMap and HDMap Priors

Dear reviewer, thank you for the professional feedbacks. We have updated the manuscript and marked the changes in red. Here are my answers to your related questions.

Questions.1 & Questions.4 & Weaknesses.3:

We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in Appendix A.1 and Fig.5. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.

Questions.2 & Weaknesses.1:

We actually didn't have the HDMap, we just pre-train the mask auto-encoding in the train split dataset and then evaluate in the val split dataset. Furthermore, to verify the generalizability of the HDMap Prior Module, we conducted cross-data experiments as detailed in Appendix B.2. In these experiments, we pre-trained the model on the train split of the Argoverse2 dataset and then evaluated it on the validation data of nuScenes.This approach also yielded an improvement in the mIoU. Additionnally, a difference between our MAE and the conventional MAE may be the cause of this confusion. Specifically, a conventional MAE takes RGB image patches and directly regresses RGB image values of the whole image. Here, the input and output of our HDmap MAE are both rasterized segmentation masks. In our method, the output segmentation mask is supervised by a pixel-wise cross-entropy loss instead of the typical regression loss in a conventional MAE. As such our MAE can be naturally used for refinement during inference since the head is a segmentation head. To note, this segmentation mask MAE is exactly doing masked auto-encoding.

Questions.3 & Weaknesses.2:

We have updated the relevant experiments in Tab.2, and conducted relevant experiments in Fig. 5 to obtain the visualization results and quantification results of the camera-only modality utilizing post-processing vectorization.

Questions.5

Regarding the issue of model runtime, we conducted further experiments to break down the inference time and discovered that the refinement module consumes a significant amount of time. In fact, the HDMap Prior Module is an optional component and can be chosen based on specific requirements. The detailed experimental results are presented in Table 13.

Questions.6

However, localization errors only influence the SDMap obtained from the global SDMAP, the BEV feature is online generated from the onboard sensors.

Minor issues:

We have changed the spelling of the word. Online long-distance HDMap generation can obtain good priors for downstream planning tasks, thereby improving downstream planning and control tasks to be safer and smoother[1].

Reference

[1]Hao Dong, Xianjing Zhang, Xuan Jiang, Jun Zhang, Jintao Xu, Rui Ai, Weihao Gu, Huimin Lu, Juho Kannala, and Xieyuanli Chen. Superfusion: Multilevel lidar-camera fusion for long-range hd map generation and prediction. arXiv preprint arXiv:2211.15656, 2022

Dear reviewer, thank you for the professional feedbacks. We have updated the manuscript and marked the changes in red. Here are my answers to your related questions.

Weaknesses.1 & Weaknesses.4 & Questions.1:

We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in Appendix A.1 and Fig.5. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.

Regarding the effective distance of SDMap, due to GPU memory constraints, we only supported testing up to 240 meters. Theoretically, SDMap has priors at all distances, and in ranges beyond sensor perception, SDMap alone provides valuable information. The model could potentially guess the form of the HDMap based on SDMap, although the generated HDMap may not be very accurate due to misalignment issues.

Weaknesses.2:

Our approach fundamentally differs from the method used in Neural Map Prior[1]. In [1], a Neural Map must be pre-constructed to enhance the online generation of HDMaps, which is then updated as the vehicle repeatedly traverses the same area, aiming to obtain a more accurate HDMap. Additionally, the furthest distance covered in their paper is 160m by 100m. Conversely, our method, based on the SDMap Prior Module, predicts HDMaps at ultra-long distances online, without the need for a prior traversal of the area. Furthermore, the maximum distance in our results extends to 240m by 60m.

Weaknesses.3

Regarding the issue of model runtime, we conducted further experiments to break down the inference time and discovered that the refinement module consumes a significant amount of time. In fact, the HDMap Prior Module is an optional component and can be chosen based on specific requirements. The detailed experimental results are presented in Table 13.

Weaknesses.5

We have completed relevant experiments, see Appendix A.4 for details. SDMap downsampling factors are the same as factor d.

Weaknesses.6

We have updated relevant experiments in Tab.2

Weaknesses.7

A difference between our MAE and the conventional MAE may be the cause of this confusion. Specifically, a conventional MAE takes RGB image patches and directly regresses RGB image values of the whole image. Here, the input and output of our HDmap MAE are both rasterized segmentation masks. In our method, the output segmentation mask is supervised by a pixel-wise cross-entropy loss instead of the typical regression loss in a conventional MAE. As such our MAE can be naturally used for refinement during inference since the head is a segmentation head. To note, this segmentation mask MAE is exactly doing masked auto-encoding.

We have further refined the description and ablation experiments of the HDMap Prior Module, as detailed in Section 4.3 and Appendix B.

Reference

[1] Xiong et.al. Neural Map Prior for Autonomous Driving. CVPR 2023.

Dear reviewer, thank you for the professional feedbacks. We have updated the manuscript and marked the changes in red. Here are my answers to your related questions.

Weaknesses.1 & Weaknesses.2

We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in Appendix A.1 and Fig.5. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.

Weaknesses.3

We have added relevant ablation experiments. Specifically, in Appendix B.5, we conducted 25%, 50%, and 75% experiments. Too high a mask ratio will lead to the lack of valid information and the actual refinement process of the input difference is large, too low a mask ratio can not force the network to capture the HDMap priors。

Weaknesses.4

This issue is also present in the work of MapPrior[1]. Essentially, our goal is to map the output of the SDMap Prior Module to the space of HDMaps. However, this process might not map onto the ground truth with high precision, hence the modest improvement in the mean Intersection over Union (mIoU) metric, which measures accuracy.In order to evaluate the realism after refining, we use a type of Perceptual Metrics, specifically LPIPS [2], to do so, and the results show that our HDMap Prior module has a significant improvement on predicting after refining than without refining. As detailed in Appendix B.1.

Weaknesses.5

It may be a problem with our table layout. You may have misread the results of only cam and fusion model. We have revised the table Tab.2.

Questions.1:

Initially, we pre-trained our model using the train split of the nuScenes dataset, followed by evaluations on the validation split. This approach might have led to some overfitting. To verify the generalizability of the HDMap Prior Module, we conducted cross-data experiments as detailed in Appendix B.2. In these experiments, we pre-trained the model on the train split of the Argoverse2 dataset and then evaluated it on the validation data of nuScenes.This approach also yielded an improvement in the mIoU.

Reference

[1]Xiyue Zhu, Vlas Zyrianov, Zhijian Liu, and Shenlong Wang. Mapprior: Bird’s-eye view map layout estimation with generative models. arXiv preprint arXiv:2308.12963, 2023. [2]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595, 2018.

Dear reviewer, thank you for the professional feedbacks. We have updated the manuscript and marked the changes in red. Here are my answers to your related questions.

Weaknesses.1 & Questions.1:

We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in Appendix A.1 and Fig.5. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.

Weaknesses.2 & Questions.2:

A difference between our MAE and the conventional MAE may be the cause of this confusion. Specifically, a conventional MAE takes RGB image patches and directly regresses RGB image values of the whole image. Here, the input and output of our HDmap MAE are both rasterized segmentation masks. In our method, the output segmentation mask is supervised by a pixel-wise cross-entropy loss instead of the typical regression loss in a conventional MAE. As such our MAE can be naturally used for refinement during inference since the head is a segmentation head. To note, this segmentation mask MAE is exactly doing masked auto-encoding.

We have further refined the description and ablation experiments of the HDMap Prior Module, as detailed in Section 4.3 and Appendix B.

Weaknesses.3 & Questions.3:

Regarding the issue of model runtime, we conducted further experiments to break down the inference time and discovered that the refinement module consumes a significant amount of time. In fact, the HDMap Prior Module is an optional component and can be chosen based on specific requirements. The detailed experimental results are presented in Table 13.

Weaknesses.4 & Questions.4:

Indeed, we have also considered this aspect and supplemented our method with a cross-dataset experiment, detailed in Appendix B.2 and Table 11. We initially pre-trained the HDMap prior module on the train split of the Argoverse 2 dataset and then tested it on nuScenes val dataset. This approach also yielded an improvement in the mIoU.