VQ-Map: Bird's-Eye-View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization

Thank you for your review and feedback!

About clearly explaining the parameters of the ablation method and the table: We apologize for not providing more details for the ablation methods in Tab. 4. The parameters that vary across the different ablation methods mainly include the layer and dimension number settings that depend on whether our deformable attention-based architecture is used or not, and the feature spatial size and dimension that depend on whether the dense or sparse BEV features are used. The supervision signals for training PV-BEV alignment also vary across the different ablation methods. We have included more explainations for each of them when asking the subsequent questions. We will also integrate the clarification for them in the revision.

About clearly describing the difference between (a), (b) and (c): The specific structures of (a), (b) and (c) are briefly described in Lines 276-281. We are sorry for not describing them clearly. Columns (a) and (b) both use the dense BEV features generated from the BEVFusion method while not using the attention-like architecture like in our token decoder module. As (a) directly predicts the BEV maps, it can be seen as a variant of BEVFusion. (b) uses dense features to predict the sparse BEV tokens and finally generate the BEV maps. (c) uses the sparse features generated from our proposed deformable attention-based architecture to directly predict the BEV maps like the traditional end-to-end methods. We will include more details for them in the revision.

About clearly describing the specific meaning of 'Supervision' in (d), (e) and (f): This part of the experiment explores which kind of intermediate results of our discrete representation learning based on VQ-VAE (see Sec. 3.1) is best for the second stage training of PV-BEV alignment. The supervision signals shown in columns (d), (e) and (f) are the explored three kinds of intermediate results. The differences are: (d) uses latent variables that have not been discretized by the codebook, (e) uses the latent variables that have been discretized by the codebook, and (f) uses the codebook indices. It shows that token classification (f) is the most effective method for PV-BEV alignment. We will add this description into the caption of Tab. 4.

About the motivation of using sparse features in our token decoder: In the second stage training for PV-BEV alignment in our framework, the BEV tokens generated from the BEV groundtruth maps are used as the supervision signals as illustrated in Fig. 2. Whether using the dense or sparse BEV features for token decoder can be experimentally determined. We first found that using dense features with 128 $\times$ 128 $\times$ 80 dimensions to predict the sparse BEV tokens in (b) even performs worse than the traditional end-to-end method in (a). Considering that using sparse features to align with our tokenization idea may be more robust against noise and geometric changes (as acknowledged by Reviewer xhZa), we thus tested using sparse features with 25 $\times$ 25 $\times$ 512 dimensions to predict the sparse BEV tokens based on our proposed deformable attention-based architecture, which significantly improves the overall performance (comparing (f) to (b)). We also surprisingly found that our achieved sparse features work better even in the traditional end-to-end framework (comparing (c) to (a)) for the BEV map estimation task.

Due to the failure of using dense features in our experiments, it may be hard to get better performance when using sparse and dense features together (the performance degrades to 61.3 mIoU). However, we experimentally find that training the sparse features to predict tokens can improve the training of dense features which are used to directly predict the maps. Its performance increases from 56.4 to 57.7 mIoU.

About the description in 3.2: This part is designed for predicting BEV tokens, and the supervision signals during the second stage training for PV-BEV alignment in our framework are the BEV tokens generated from the BEV groundtruth maps as illustrated in Fig. 2. We cast it as a classification task and use the focal loss (Lines 188-189) to train the backbone FPN and token decoder. This part does not involve any additional supervisions. We will clarify this point in the revision.

About the different layer and dimension number settings in the ablation experiment of Tab. 4: In the ablation experiment of Tab. 4, columns (a) and (b) both use the dense BEV features generated from the BEVFusion method (not using the attention-like architecture like ours), so comparing our best result 62.2 mIoU using 8-layer and 512-dimensional settings in column (f) to them may be OK. However, columns (c) (d) (e) and (g) all use 6-layer and 512-dimensional settings, it may be not appropriate to compare (f) with them. We apologize for this confusion. We will replace the results for column (f) with 61.8 mIoU results using 6-layer and 512-dimensional settings (see Tab. 3) in the revision.

About more computational complexity results: Following your suggestion, we have added a computational complexity comparison using the number of parameters, MACs, and the training time. Please refer to Tab. R1 in the attached PDF. It clearly shows that our approach not only demonstrates strong performance (also acknowledged by Reviewer 1hbx), but also saves much computational cost in comparison to the recent SOTA methods MapPrior and DDP, in both the training and testing phases. In addition, the two-stage training of our approach introduces some additional training overhead in comparison to BEVFusion. We will include them in the revision.

About more visualization comparisons against MapPrior: Thank you for your suggestion. We show more visualization comparisons against MapPrior and DDP in Fig. R1 of the attached PDF, which can better show the advantage of our proposed solution. We will include them in the revision.

Thank you for your review and feedback! We are happy that you quite like our tokenization idea for the BEV map layout estimation task. We provide responses to the specific points below:

About the state-of-the-art comparison with MapPrior[17] and DDP[19]: Both of the two references of MapPrior[17] and DDP[19] are published at ICCV'2023, and evaluate their approaches in both the camera-only setting and multi-modality setting (with LiDAR information combined). In both settings, the mIoU of DDP always surpasses MapPrior, i.e., 59.4 vs 56.7 in the camera-only setting and 70.6 vs 63.1 in the multi-modality setting. In Tab. 1 of our state-of-the-art comparison, we report the results for MapPrior[17] and DDP[19] only using their published results under the camera-only setting, because our state-of-the-art comparison is completely based on the camera-only setting. We are very sorry for not clearly introducing the camera-only setting used in our main paper. We will remendy this misleading issue in the revision. In Tab. 1, it clearly shows that our VQ-Map can achieve the best performance in comparison to other entries under the camera-only setting. We will also delve into how to incorporate data from other modalities (like the LiDAR) into our framework in the future work.

About the dataset split of nuScenes in the comparison: In the much earlier BEV semantic segmentation works, e.g., LSS[3] published at ECCV'2020 and PON[39] at CVPR'2020, the pixel-wise segmentation is conducted for the layout of both the static objects in the background and dynamic objects in the foreground. However, the nuScenes test set does not provide the groundtruth for the dynamic objects in the foreground. So these works directly use the nuScenes validation set to validate the performance of their proposed approaches. To align with these pioneering works and make the fair comparison, the subsequent works shown in Tabs. 1 and 2 all keep using this dataset split setting by default, where the models are evaluated on the nuScenes validation set. More specifically, for our evaluation in the surround-view experiments in Tab. 1, only the training set is used to train our model and the number of training epochs and total iterations is aligned with the previous work BEVFusion. The model weights from the last epoch are used for final evaluation on the nuScenes validation set. For our evaluation in the monocular experiments in Tab. 2, we follow the practice of the pioneering work PON, which re-divides the (training, validation) split into (training, calibration, validation) split, where the calibration set is used to adjust the hyper-parameters, and the validation set is used for evaluation. We will clearly explain this dataset split setting in the revised version.

Furthermore, to ease the concern with regard to the nuScenes test set, where the groundtruth for the layout of the static objects in the background is available, we directly tested the off-the-shelf trained models from BEVFusion, DDP and our approach on the nuScenes test set to further show the superiority of our approach (although none of the previous works conduct the comparison on the nuScenes test set). As shown in the table below, our approach consistently outperforms DDP and BEVFusion by large margins like in Tab. 1.

Thank you for your review and feedback! We are happy that you find our idea of using a pre-trained codebook interesting in the BEV map generation domain. We provide responses to the specific points below:

About the IoU metric for measuring the performance of BEV map: The common practice of using the IoU metric to reflect the overlap between the models' predictions and the groundtruths can indicate the overall accuracy of the predictions to a certain extent. However, we agree with you that the drivable area boundary is also one of the most important aspects to measure the accuracy of the predictions, and the IoU metric may not faithfully reflect the accuracy of the drivable area boundary. Inspired by your valuable comments, we thus propose to use the Sobel operator to detect the boundaries of the models' predictions and evaluate the boundary quality using the Chamfer distance metric in pixels (the smaller, the better). We compare our method with BEVFusion and the previous SOTA aproach DDP for the surround-view drivable area layout estimation in the table below. Compared to DDP, although our IoU score for the drivable area only achieves 0.2% performance gain, the Chamfer distance for our approach decreases by an average of 0.24 pixels, indicating that our approach can estimate the drivable area layout with more accurate boundary. We will include this new analysis in the revision.

About computing the average performance based on normalization of the raw pixels: For each background class in the map layout estimation, an IoU score is evaluated based on the raw pixels and reported in percentage. The common practice of computing mIoU is to take the average of the IoU scores across all the evaluated background classes as you have pointed out. Following your suggestion, we compute the average performance (Normalized IoU) over all the evaluated background classes based on normalization of the raw pixels as well and show the comparison with BEVFusion, MapPrior and DDP in the table below. We will include this new analysis in the revision.

About the complexity of the cost: Following your suggestion, we have added a computational complexity comparison using the number of parameters, MACs, and the training time. Please refer to Tab. R1 in the attached PDF for “global” response. It clearly shows that our approach not only demonstrates strong performance (acknowledged by Reviewer 1hbx), but also saves much computational cost in comparison to the recent SOTA methods MapPrior and DDP, in both the training and testing phases. In addition, the two-stage training of our approach introduces some additional training overhead in comparison to BEVFusion. We will include this computational complexity analysis in the revision.

About handling small regions: As acknowledged by Reviewer xhZa, tokenization in our approach is more robust against noise and geometric changes, which however may lose some information or detailed spatial information. So our approach cannot handle small regions well. This may be the inherent limiation of our approach due to the No Free Lunch rule. When the dangerous holes and some random obstacles appear in the realistic AD environment, an anomaly detection module may be needed to handle this situation. We will detail this discussion in the limitation section.

About achieving multiple tasks at the same time: Although our approach is a specialized method only for BEV map generation at present, we believe the token-based multi-task modeling for autonomous driving is very promising. Additionally, tokenized intermediate results are well-suited for combining with large language models. We will detail this discussion in the conclusion and leave the token-based multi-task modeling in the future work.

Thank you for your review and feedback! We are happy that you recognized that our paper clearly identifies the challenges in BEV map layout estimation and provides a well-defined solution with a novel approach VQ-Map proposed. We provide responses to the specific points below:

About the idea of using generative model: DDP and DiffBEV propose an end-to-end framework to generate a more comprehensive BEV representation and denoise noisy samples by applying the diffusion model to BEV perception. Although we share the similar idea of using generative model, our approach achieves better performance with much lower MACs (see Tab. R1 in the attached PDF for the comparison with DDP) as the diffusion process may introduce a significant computational overhead. We will clarify this point in the revision.

About the novelty with regard to our token-based decoder and using other attention: In the main paper, we have pointed out (Lines 41-43) that our token decoder module can be an arbitrary transformer-like architecture to be compatible with our novel pipeline VQ-Map. Following the common practice, we design this token decoder module based on deformable attention thanks to its ability to model the prior positional relationship between PV (Perspective View) and BEV (Bird's Eye View) for feature alighment. Deformable attention can naturally use reference points to leverage these priors. Following your suggestion, we also consider other two kinds of attention in our token-based decoder for ablation comparison on the nuScenes validation set, i.e., replacing the deformable attention with (1) the standard attention and (2) the cross view attention proposed in CVT[37], respectively. We experimentally find that the deformable attention performs better than the above considered alternatives as shown in the table below. We will include this new ablation comparison in the revision.

About the comparison with more recent diffusion methods: In Sec. 4.2 of our main paper, we have conducted the comparison with the most recently published diffusion-based BEV map layout estimation works, i.e., DDP[19] and DiffBEV[20] published in ICCV'2023 and AAAI'2024 respectively. In the surround-view comparison in Tab. 1, we report the results of DDP[19] using its camera-only modality results published in the original paper. In the monocular comparison in Tab. 2, we report the published results of DiffBEV[20] based on its original paper. In Tab. R1 of the attached PDF, we have also presented a computational complexity comparison with DDP to better show the superiority of our approach. Note that we are not able to conduct the computational complexity comparison with DiffBEV due to the fact that its publicly available code is not complete. We will detail the analysis in comparison to DDP and DiffBEV in the revision.

About the effect of codebook size: We have conducted the experiments with different codebook size $K\times D$, and the results (shown in the table below) show that our proposed method is insensitive to the size of the codebook in a wide range. We will include this analysis in the revision.

About the different IoU threshold settings for different methods compared in Tab. 1：Most of previous methods publish the results following the evaluation protocol in BEVFusion[1], where the threshold that maximizes IoU is used, while MapPrior[17] uses the same IoU threshold of 0.5 across all the background classes for the BEV map layout estimation. We believe using a constant IoU threshold for all the background classes in our approach can deliver a more fair comparison with the existing approaches without suppressing the performance of the methods that use the threshold maximizing IoU. Note that our approach using the threshold maximizing IoU achieves 62.3 mIoU in comparison to 62.2 in Tab. 1. We will clarify this point in the revision.