Unveiling the Hidden: Online Vectorized HD Map Construction with Clip-Level Token Interaction and Propagation

We deeply appreciate your thorough and insightful feedback. We believe we can enhance the paper's quality and present more concrete results based on your comments. Below are point-by-point responses for your comments, and these will be included in the revised paper.

---

W1-1. Explanation of TTM and MapTRv2

Thanks for your valuable comment. Initially, we tried to avoid overclaiming in terms of model architecture, so we opted to skip the explanation of TTM and MapTRv2. However, we fully understand that this is not good for general readers. Following your advice, we will improve the presentation as follows.

i) Inter-clip Unveiler

In Line #172, we will introduce an equation as

$U^{read}=Read(U_{t-2S:t-S}^{memory}, Q^{map})=S_{N_c}([U_{t-2S:t-S}^{memory}|| Q^{map}])$

Here, the notation $[U_{t-2S:t-S}^{memory}|| Q^{map}]$ denotes the concatenation of two elements.

In Line #178, we will introduce an equation as

$U_{t-S:t}^{memory}=Write(U_{L}^{clip}, U_{L}^{map}, U_{t-2S:t-S}^{memory})=S_M([U_{L}^{clip}|| U_{L}^{map} ||U_{t-2S:t-S}^{memory}])$

If the tokens within the memory are not re-selected in the subsequent steps, it will be removed from the memory, and the selection mechanism will be determined through the learning.

ii) Loss

To provide detailed information about the losses, we will add following equations:

$\mathcal L_{one2one} = \lambda_c^F\mathcal L_{cls}^F + \lambda_p^F\mathcal L_{p2p}^F + \lambda_d^F\mathcal L_{dir}^F$

$\mathcal L_{dense} = \alpha_d\mathcal L_{depth} + \alpha_b\mathcal L_{BEVSeg} + \alpha_p\mathcal L_{PVSeg}$

$\mathcal L_{Frame-level MapNet} = \beta_o\mathcal L_{one2one} + \beta_d\mathcal L_{dense}$

$\mathcal L_{MapUnveiler} = \lambda_c^M\mathcal L_{cls}^M + \lambda_p^M\mathcal L_{p2p}^M + \lambda_d^M\mathcal L_{dir}^M$

where $\mathcal L_{one2one}$ is used for frame-level MapNet, and we set with $\lambda_c^F=2, \lambda_p^F=5, \lambda_d^F=0.005$. $\mathcal L_{dense}$ is an auxiliary loss using semantic and geometric information, and we set $\alpha_d=3, \alpha_b=1, \alpha_p=2$. $\mathcal L_{MapUnveiler}$ is used for MapUnveiler, and we set $\lambda_c^M=2, \lambda_p^M=5, \lambda_d^M=0.005$.

Due to the character limit in the rebuttal, we conceptually show the revised explanation. We will further detail TTM, MapTRv2, and each loss in the Appendix.

---

W1-2. The definitions of the temporal window $T$ and the stride $S$

To provide a clear explanation, we will introduce the following equation.

$C_k = \lbrace f_t \rbrace_{t=kS+1}^{kS+T}$

The notation $C_k$ represents the k-th clip, starting from k=0. Each frame is denoted by $f_t$, represents the t-th frame among the consecutive frames. Therefore, when T=3 and S=2, we obtain clip sets such as $C_0=\lbrace f_1, f_2, f_3 \rbrace$, $C_1=\lbrace f_3, f_4, f_5 \rbrace$, ..., and $C_k=\lbrace f_{2k+1}, f_{2k+2}, f_{2k+3}\rbrace$.

Thus, the temporal stride S refers to the clip stride, not the frame stride.

Additionally, it would be interesting to additionally set frame stride, as you mentioned. If we set the frame stride=2 and the clip stride (S)=1 (to see all frames), MapUnveiler achieves an mAP of 68.8% in nuScenes. This is lower than the model with frame stride=1 and T=1, (69.3%). We conjecture that temporally nearest frames provide the most rich information for unveiling maps because they have large spatially overlap regions, and temporally distant frames can effectively be exploited through the inter-clip Unveiler.

---

W1-3. Inference mechanism

MapUnveiler excutes per-clip. Thus, it can cause response delay if the input frame rate is slower than 12.7 (which is the inference speed of MapUnveiler) because the model will wait until it collects T frames. To avoid this problem, we can set the clip stride (S) to 1 (but it will lead to a slight performance degradation, as presented in Table 7). Alternatively, we can fill in the intermediate frames' results with frame-level MapNet: during the response delay, we can directly construct an online map using map queries generated from the frame-level MapNet. In this case, however, the performance will drop from 69.8% to 68.0%.

W1-4. Open access code

We are planning to release source codes upon acceptance.

---

W2. Detail on how the pre-training and fine-tuning

We end-to-end fine-tune MapUnveiler for 24/6 epochs from pre-trained MapTRv2, which was trained for 24/6 epochs. To address this concern, we present the freeze fine-tune setting in Table 4, but we understand it cannot fully address this concern. So, we additionally present experimental results where we pre-train MapTRv2 for 12/3 epochs and then fine-tune MapUnveiler for 12/3 epochs, totaling 24/6 epochs. The results are given in G1 of the global response.

---

W3. Long training schedules

Thanks for your thorough feedback. We trained 110/30 epochs, and MapUnveiler achieves mAP of 70.6% and 72.9% on nuScenes and Argoverse2, respectively. As you commented, the performance improvement of our model is marginal. We think that we can converge more quickly because we start training from a pre-trained frame-level MapNet. Even though our model lags behind HIMap (73.7% and 72.7%) in the long training schedule setting, we would like to kindly emphasize the following three points: (1) HIMap is heavy (9.7 FPS) and runs slower than ours (12.7 FPS); (2) HIMap requires more training time to fully converge, which may cause overfitting; (3) HIMap was published arXiv two months ago when we submitted our paper, so this work stands as a contemporaneous work.

---

W4. Caption in Table 1

Thanks for the correction. We will remove † in Table 1.

---

W5. HIMap[49], MGMap[24], and MapTracker[A]

Thanks for the thorough check. We will revise them as CVPR2024 and will discuss with MapTracker. We would like to kindly inform the reason for notating as preprints: the official paper list of CVPR2024 was opened on the same day as the deadline for NeurIPS2024 submissions.

Thanks for the detailed response. The added details clarify the architectural design, especially for the clip-based inference mechanism and how $T$ and $S$ work. I believe these will also help other readers better understand your approach.

I appreciate the new results in the general response, which resolve my concern about the fairness of experiments. However, please make sure the results in G4 also follow a fair setting, or you should at least indicate the difference in training schedules in the table. Overclaiming the performance boost with an unfair setting under the hood will hurt the community.

With the above concerns addressed, I will increase my score from 3 to 5 (borderline accept). Reasons for not giving a higher score are mainly two-fold: 1) the clip-wise inference is a bit unnatural, making the output quality temporally inconsistent; 2) the initial writing clarity is very problematic, requiring a massive revision to improve the paper's quality.

If the paper is accepted, I hope the authors can carefully improve the clarity of the method description (W1), fix the discussion/reference of related works (W5), provide fair experimental results and clearly indicate the difference in training settings (W2,3).

We are particularly encouraged that the reviewer finds our method novel and well-motivated. And we highly appreciate your constructive comments and suggestions. Below are our responses to each of your queries, and we will include them in the revised paper.

---

W1. Weave more intuition into the text in 3. Method

We will include our motivation in the method explanation to make it easier to follow and understand our method. For this, we will revise the paper as below (we highlighted the new and revised descriptions in italics):

Clip Token Generator (Line #149-150)

... , respectively. To globally gather intra-clip map features, we opt for a naive cross-attention [38]. Through this step, we obtain compact clip-level map representations, enabling efficient intra-clip communication with small-scale features.

BEV Updater (Line #155)

... original bev features. The main idea of BEV Updater is to avoid heavy computation in spatio-temporal cross attention. To achieve this, we do not directly communicate intra-clip BEV features, but instead decouple the spatial BEV features and temporal clip tokens. We then update the spatial BEV features with compact temporal clip tokens, effectively communicating spatio-temporal information with reasonable computational costs. The updated BEV ...

Map Generator. (Line #161)

... the updated BEV features $U_l^{BEV}$. Since the updated BEV features are spatio-temporally communicated, we directly extract map tokens $U_l^{map}$. Each map token represents a vectorized map element through a 2-layer Multi-Layer Perceptron (MLP). The map tokens ...

---

W2. Rephrase some sentences

We are sorry, we understand that it may be an overstatement. We will rephrase Line 261 as follows:

, while MapUnveiler shows comparable performance MapUnveiler also shows a proformance degradation of 69.8%→63.8% (-6.0%), but it demonstrates a smaller performance gap compared to previous studies.

Following your advice, we also tone down following sentences (Line 242):

Although MapUnveiler incorporates temporal modules, we achieve a fast resonable inference speed (12.7 FPS) compared to frame-level MapNet (MapTRv2 [22], 15.6 FPS), surpassing both performance and speed compared to the heavy state-of-the-art approach (HiMap [49], 9.7 FPS).

---

W. Typos

Thank you for your diligent efforts to enhance the quality of our paper. We will rectify the mentioned typos and strive to correct any other potential typos. We will thoroughly proofread the paper from the beginning.

---

Q1. Model quantization

That's an intriguing idea, and we hope to see the comparison. Unfortunately, we tried to implement the quantized models, but it was quite challenging to implement and evaluate the accuracy, memory usage, and speed of the quantized model. The primary reason behind this is that we developed our MapUnveiler using several custom addition and multiplication operators, which the FakeQuantize function does not support. Implementing it with TensorRT necessitates reconstructing the entire pipeline of our code, hence requiring additional time.

Based on current knowledge, we can consider extreme quantization (4-bit) and a general solution used in TensorRT (16-bit and 8-bit). The performance drop due to 4-bit model quantization may be marginal in CNN architecture if we use advanced quantization algorithms such as [A]. However, transformer-based architectures currently lose more than 20% of the accuracy [B] with 4-bit quantization. As an alternative, if we adaptively combine FP16 and INT8 [C], we can accelerate the model speed x1.87 times with a marginal performance drop (about -1.19%). With this, we can boost MapUnveiler with a large window size (T=5) through quantization. However, the performance gain from a wide window size is marginal (mAP of 70.1% for T=5, and mAP of 69.8% for T=3) as shown in Table 7 of the main paper. We conjecture that our model effectively leverages long time dependencies through the inter-clip Unveiler, thus it may not be necessary to use a large window size. But it would be interesting to see the gains of the main model from quantization.

We plan to implement the quantized model and include the results in the final version.

[A] QDrop, ICLR 2022

[B] RepQ-ViT, ICCV 2023

[C] https://github.com/DerryHub/BEVFormer_tensorrt

---

Q2. The reason for strugglgling with some occlusions

We found that MapUnveiler may be unable to unveil some regions if the regions are occluded for every frame. We visualize an example in Figure 9 of the rebuttal PDF file of the global response. MapUnveiler initially roughly predicts the boundary where we highlighted in green. However, the invisible regions are continuous, and MapUnveiler eventually predicts the region as having no boundary. As such, if the model cannot see a clear region across all frames, MapUnveiler may fail to recognize the occluded map information.

---

Q3. Randomly dropped intermediate frames

A very constructive suggestion. We evaluated three models with randomly dropped intermediate frames. Frames were dropped by converting multi-camera images into black images. The experiment was conducted with drop rates of 20%, 10%, and 5%, and the results are given in Table 13 of the rebuttal PDF file of the globalresponse. MapUnveiler is affected by dropped frames, but the performance degradation is reasonable compared to MapTRv2.

---

L. Other potential limitations

One can be a potential limitation that was discussed in Q2; MapUnveiler is likely to fail in unveiling fully occluded roads across all frames.

Additionally, MapUnveiler requires marginal additional computational costs (15.6 FPS → 12.7 FPS) but requires approximately two times more GPU memory (830.4 MB → 1614.9 MB) and three times more parameters (76.4 MB → 213.9 MB) compared to the frame-level MapTRv2 model. We provide a comparison table in G3 of the global response.

We sincerely appreciate your efforts in reviewing our responses. We are delighted to receive your positive feedback and supports for our work.

Thanks,
The Authors

Thank you for providing the insightful and constructive feedback. We appreciate your acknowledgment that the paper is easy to follow and that the proposed approach is effective. Below are our responses to each comment, and we will include all the results and comments in the revised version.

---

W1. Results without pre-training

Thank you for valuable feedback. We fully understand that our training pipeline may present an unfair advantage since previous works trained their models on vectorized HD map datasets for only 24 epochs, whereas we pre-train MapTRv2 for 24 epochs and fine-tune MapUnveiler for 24 epochs, amounting to a total of 48 epochs.

To address this concern, we conducted an experiment where we pre-train MapTRv2 for 12epochs/3epochs and then fine-tune MapUnveil for 12epochs/3epochs on either the nuScenes or Argoverse2 training set. Even though our proposed modules will be trained for only 12epochs/3epochs, this will undoubtedly provide fair comparisons, as we are training a total of 24epochs/6epochs. The results are given in G1 of the global response.

As shown in the table, our method still demonstrates state-of-the-art performance of mAP on both nuScenes and Argoverse2 validation sets under a more fair setting.

Additionally, we tried to skip the pre-training stage and train MapUnveilr from scratch, but it failed to converge, achieving an mAP of 18.2% in the nuScenes (60x30m) validation set. This indicates that our method necessitates meaningful frame-level map features to learn map unveiling. We would like to kindly note that this pre-training strategy is not own approach but is commonly adopted for training networks with temporal information. For instance, StreamMapNet [45] and SQD-MapNet [40] trained their initial 4 epochs with single-frame inputs. MapTracker [A] employed a three-stage training process (BEV Encoder → Vector Decoder → All Parameters) to facilitate initial convergence.

[A] Chen, Jiacheng, et al. "Maptracker: Tracking with strided memory fusion for consistent vector hd mapping." arXiv preprint arXiv:2403.15951 (2024).

---

W2. How to select the tokens that are visible in certain frames

The model automatically selects the visible and valuable BEV regions through the learning of cross attention. As depicted in Figure 1-(c) in the main paper, each clip token within the temporal window is fully utilized for BEV update. We could limit the clip tokens to select manually choosen visible BEV regions, but we thought that this might not always provide an accurate solution for constructing a clear HD map (e.g., the model might try to select an easy-to-see region even if there are no lanes). Hence, we opted to learn the selection mechanism in an end-to-end manner, which can minimize the losses of the constructed map. This approach is straightforward yet effective, as illustrated in Figure 1: compared to (a) MapTRv2 and (b) StreamMapNet, our BEV feature most clearly represents map elements.

---

Q1. Experiment result for geometric-based dataset split

Thank you for your insightful suggestion. We conducted additional experiments on the geometric-based dataset splits you suggested [1]. The results have been moved to the global response due to the character limit. Please find the result in G2 in the global response.

---

Q2. Additional computing costs

We provided additional computational costs in Table 3 by injecting temporal clip tokens in two aspects: Intra-clip Unveiler and Inter-clip Unveiler. To provide rich information, we further measured GPU memory consumption during inference time and model parameters, and append Table 3. We present new Table 3 in the global response. Please find the result in G3 in the global response.

We thank the reviewer for providing thorough feedback and interesting suggestions. We are grateful for your acknowledgment that the introduction of a clip-level pipeline for vectorized HD map construction is effective and the proposed clip tokens propagate map information efficiently. Below are our responses to each comment, and we will include all the results and comments in the revised version.

---

W1. Validation on geo-disjoint dataset

Thank you for your insightful feedback. To address this concern, we conducted additional experiments on a recent geo-disjoint dataset split. Due to the character limit, we attached the full table in the global response. Please find the results in G2 in the global response.

---

W2. Analysis of added model compacity

That would be a good analysis to provide rich information about our model for readers. Additional to the analysis of accuracy and computational costs of the proposed intra-clip Unveiler and inter-clip Unveiler (Table 3 in the main paper), we add GPU memory consumption during inference time and the model parameters. We present the table in the global response. Please find the results in G3 in the global response.

---

W3. Validate the proposed modules on other baseline methods

That's a valuable suggestion. We believe that any single-frame inference online mapping methods based on DETR (which outputs both rasterized BEV and vectorized query features) can be adapted for our MapUnveiler. Therefore, it would be exciting to additionally claim our contribution in modularization. Consequently, we attempted these experiments with DETR-based models for which the code is available online (MapVR [46], MGMap [24], MapTRv1 [21]). Unfortunately, we are unable to present the results at this time due to a lack of resources and time to set up and conduct experiments within the short rebuttal period. Nevertheless, we would like to address the concern regarding this query, so we experimented with various backbone networks (ResNet-18 and V2-99 {3}). We present the result table in G4 in the global response.

As shown in the table, our method is not limited to MapTRv2 with ResNet50, but can be extended to ResNet18 and V2-99 {3} backbones. This suggests that our method can also work for other various frame-level features as well.

We plan to implement our method on other single-frame inference online mapping methods and include the results in the final version.

{3} An energy and GPU-computation efficient backbone network for real-time object detection, CVPRW. 2019.

---

W4. Consistency of estimated HD Map across frames

Thanks for your thorough feedback. Measuring the consistency of the model would be very helpful and further elaborate the contribution of our model. To quantitatively measure the consistency of the model, we would need to indicate the track ID of each estimated map element. However, this is challenging in our model because we simplify and propose a straightforward pipeline that does not require any complicated spatial warping process across time, which is required for streaming inference (StreamMapNet [45]), nor track ID annotation. Thus, we attempt to show the consistent results qualitatively in Figures 4, 5, 6, 7, and 8.

We recently found that MapTracker {4} presented a consistency-aware mAP (C-mAP) metric and measured it for baseline methods (e.g., MapTRv2) by predicting track IDs through their tracking algorithm. It seems that we can also measure the C-mAP by applying the tracking algorithm proposed in {4}. To measure it, we have to re-train the model with the refined GT data proposed in {4}, so we are unfortunately unable to show the C-mAP result currently due to a lack of resources and time.

We plan to implement C-mAP and add the results in the final version.

{4} MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mapping, arXiv:2403.15951

---

Q1. Map queries

That's correct. Map queries are trained with the same objective function as the map tokens, so it can be transferred directly to vectorized HD maps. Thus, MapUnveiler can be degraded to MapTRv2 if we transfer vectorized HD maps using map queries instead of map tokens. An interesting experiment can be conducted based on this idea; we can use MapUnveiler for both frame-level and clip-level scenarios (as sometimes we cannot utilize temporally consecutive frames due to unexpected communication/sensor errors in real-world scenarios). We experimented by replacing map tokens with map queries for every two frames, resulting in a slight performance degradation from 69.8% to 68.0% in nuScenes (60x30m).

To clarify this, we will add following explanation in Line #130 (we highlighted the new descriptions in italics):

... the map decoder, respectively. BEV features represent rasterized map features, whereas map queries embed vectorized map information; thus, we can directly construct vectorized HD maps using the map queries.

---

Q2. Map decoder

That's correct. The map decoder is adopted from MapTRv2. To clarify this, we will revise the caption in Figure 2:

... vectorized HD maps. All components in the frame-level MapNet (i.e., Backbone, PV to BEV, Map Decoder) are adopted from MapTRv2 [22]. The frame-level MapNet extracts ...

---

Q3. Map tokens

That's correct. The map tokens generated from the intra-clip unveiler are the refined version of map queries. To clarify this, we will revise the paper in Line #158:

... in the previous step. The objective of this step is to generate a refined version of frame-level map queries. As illustrated in ...