InsightMapper: A closer look at inner-instance information for vectorized High-Definition Mapping

We thank the reviewer for your detailed and insightful comments and suggestions. Here is the response and clarification to your questions and concerns.

1.Position of inner-instance self-attention layers

The vanilla attention used in MapTR can learn both inter-instance and inter-instance message passing. And the vanilla attention layer before the cross-attention layer (for deformable attention calculation) should treat these two kinds of message passings equally like DETR, otherwise, some instances may produce duplicate predictions since they cannot “clearly see” other instances, so that instances cannot collaborate with each other very well for global optimal prediction, which leads to severely degraded performance.

The problem of the above pattern is that all queries are always equally treated, so that after the cross-attention, a point will optimize its position based on all other points with equal attention. Intuitively, points that belong to the same instance of it should project more influence. The proposed inner-instance self-attention is located after the cross-attention, and it allows a point to pay more attention to its inner-instance neighbors, to make the optimization direction of all points within an instance consistent. Otherwise, the results might be severely affected by the zig-zag issue.

In short, the vanilla self-attention layer before the cross-attention layer assures that instances can “see” each other clearly, just like what the raw DETR does. The inner-instance self-attention layer after the cross-attention layer enables points to pay more attention to their inner-instance neighbors for local refinement. Both self-attention layers cannot be removed but they have different functions.

In the appendix, we provide detailed ablation studies in Figure. 8 and Table. 12 about attention layer position and decoder network structure. They should be helpful to further justify the design of InsightMapper.

2.Additional baseline works

Thanks for providing more related works, we will cite them in our paper. We did not consider comparing these works because: (1) InstaGram has much inferior performance than MapTR (previous SOTA), and it does not open-source the code (2) Bi-Mapper has a similar performance with HDMapNet, which is much worse than MapTR. Considering the limited page size, we mainly compare InsightMapper with past SOTA methods or most representative methods.

3.Training epoch

In Table 1, we report the results of MapTR and InsightMapper with both 24 epochs and 110 epochs training. In both epoch settings, InsightMapper greatly improves the mAP and TOPO compared with MapTR. Thus, we believe the superiority of InsightMapper is well illustrated.

4.Computation complexity

Thanks for the question. In Table 4, we add additional results for computational complexity, e.g., parameter size.

5.Ablation studies

Please refer to Table 5 to Table 7. In InsightMapper, we mainly propose three modules, and each module has multiple ablation studies to verify its effectiveness. Query generation scheme (Table. 5), query fusion (Table. 6) and inner-instance self-attention (Table. 7). We believe these ablation studies could justify the design of the proposed modules.

4.Vanilla attention and No mask

Sorry for the inappropriate definition and insufficient explanation. Vanilla attention indicates attention layers used in MapTR, which treats inner-instance and inter-instance information equally. “No mask” indicates that the inner-instance self-attention layer does not block inner-instance self-attention with probability $\epsilon$. In section 4.4 and Figure.6, we proposed to randomly block some attentions between inner-instance points with probability $\epsilon$ for better robustness, inspired by Masked Autoencoder. The expression “No mask” could be misleading, and we changed it to “No random blocking($\epsilon=0$)” in the revised manuscript. You may find the results with different $\epsilon$ in Table. 11 of the appendix.

5.Implementation settings

In our experiments, we strictly followed MapTR for parameter settings. Except the parameters reported in the experiment results (Table 1, epochs, backbone, modality), we keep all other settings exactly the same for fair comparison, e.g., batch size, input data resolution, network structure (e.g., layers of decoder), input query numbers, embedding dimension, etc. Thus, the fairness of the experiments is assured.

6. FPS and network structure

The FPS reported in the raw MapTR (Res50) is about 15. The FPS reported in InsightMapper is 7.5. This is because we detect centerlines as an additional class. The number of centerline classes is much larger than other classes, which makes the frame rate drop. But we also provide the FPS of MapTR for 4 classes detection (8.5FPS), which is very close to the FPS of InsightMapper. This indicates that InsightMapper is very efficient, considering it greatly improves mAP and TOPO but minorly affects the FPS.

The drop of FPS comes from additional inner-instance layers, which is minor, and we did not modify other parts of MapTR, like encoder, decoder layers, input query numbers, input image resolutions, etc.

7.Query generation schemes

Strictly speaking, it seems that there is no work for HD map detection utilizing the naïve scheme. But DETR/deformable DETR for object detection widely adopts such query generation scheme, and that’s why we add a comparison with the naïve scheme. PivotNet has variant input query length, which makes their query generation problems different from ours. Since PivotNet is a quite newly released paper and it does not open-source its code, we are afraid we cannot compare with it at this stage. But corresponding comparison experiments could be conducted in the future as long as the authors release their code. In that way, probably the proposed modules in InsightMapper could be plugged into PivotNet for further improvement.

8.Contributions

InsightMapper proposed simple but effective modules for better HD map detection. Although these ideas might not be complicated, corresponding insights about utilizing inner-instance information and improved results still make contributions to the community.

We thank the reviewer for your detailed and insightful comments and suggestions. Here is the response and clarification to your questions and concerns.

1.Data preprocessing

Thanks for the question. At this stage, to make centerline predictable by DETR-based models like other road elements, it has to be represented by polylines instead of complicated graph with intersections. This data preprocessing method has been widely used in past works that focused on centerline detection. Directly detecting centerlines by DETR-based models without decompensating them into polylines could be treated as promising future work.

2.Evaluation with 3 classes For better comparison with MapTR, we report the evaluation results with only 3 classes (i.e., boundary, ped walking and road divide). You may find the results in Table 4 in the revised manuscript. Compared with 4-class road element detection, 3-class detection is much simpler, so the mAP improvement of 3-class detection is as high as 4-class detection, but it still reaches around +3mAP. Similar to the evaluation of Argoverse-2 in Table 3, the more complex the detection task/scenario is, the better performance gain can InsightMapper achieve.

In the comparison experiments, we keep all experiment settings exactly the same for MapTR and InsightMapper, except proposed plugged-in modules.

3.Inter-instance and inner-instance

We apologize for inappropriate definitions and insufficient explanations. You might misunderstand the design of the masked inner-instance self-attention module. Similar analysis can also be found in answer 4. “No mask” does not mean removing the mask blocking inter-instance information, it indicates we do not block inner-instance attention with probability $\epsilon$ for better robustness, inspired by Masked AutoEncoder. We will change “no mask” into "no random blocking($\epsilon=0$)" in the revised manuscript. It should noted that inter-instance information exchange is always blocked in the inner-instance self-attention module.

In your review, there are two claims thought to be contradictory: claim A."Usually, inter-instance correlation distracts the inner-instance information exchange, degrading the final performance", and claim B. "self-attention before cross-attention should treat all queries equally to prevent duplicated predictions". In fact, these two claims are not contradictory since they have different purposes.

In claim A, we talked about how inner-instance and inter-instance information exchange affect the line refinement and point position optimization. This information exchange happens after the cross-attention. Intuitively, points that belong to the same instance of it should project more influence. The proposed inner-instance self-attention allows a point to pay more attention to its inner-instance neighbors, to make the optimization direction of all points within an instance consistent. Otherwise, the results might be severely affected by the zig-zag issue.

In claim B, we talked about how self-attention works for deformable cross-attention. This happens before the cross-attention. The self-attention layer should treat inter- and inner-message passings equally, otherwise, some instances may produce duplicate predictions since they cannot “clearly see” other instances, so instances cannot collaborate with each other very well for global optimal prediction, which leads to severely degraded performance.

Thus, claim A and claim B do not contradict each other, and they are claims for different purposes. Violating either claim may lead to degraded results.

We thank the reviewer for your detailed and insightful comments and suggestions. Here is the response and clarification to your questions and concerns.

1.Evaluation results compared with the original numbers from MapTR

In the original paper of MapTR, the authors only consider 3 classes of road elements (i.e., boundary, road split, and pedestrian walking). In InsightMapper, we add another new class “lane centerline” into the prediction, which enables our model to handle more road elements. Therefore, it is not appropriate to directly compare the results reported in InsightMapper (4 class prediction) and those in MapTR (3 class prediction).

2.Notation of inner-instance query fusion

Sorry for the lack of definition of notation. $w_{i,j,k}$ represent weights for query fusion, which could be learnable weights of fully connected layers or self-attention layers.

3.Vanilla attention and “No mask”

Sorry for the inappropriate definition and insufficient explanation. Vanilla attention indicates attention layers used in MapTR, which treats inner-instance and inter-instance information equally. “No mask” indicates the inner-instance self-attention layer does not block inner-instance self-attention with probability $\epsilon$. In section 4.4 and Figure.6, we proposed to randomly block some attentions between inner-instance points with probability $\epsilon$ for better robustness, inspried by Masked Autoencoder. The expression “No mask” could be misleading, and we change it to “No random blocking($\epsilon=0$)” in the revised manuscript. You may find the results with different $\epsilon$ in Table. 11 of the appendix.

4.Evalution with 3 classes For better comparison with MapTR, we report the evaluation results with only 3 classes (i.e., boundary, ped walking and road divide). You may find the results in Table 4 in the revised manuscript. Compared with 4-class road element detection, 3-class detection is much simpler, so the mAP improvement of 3-class detection is as high as 4-class detection, but it still reaches around +3mAP. Similar to the evaluation of Argoverse-2 in Table 3, the more complex the detection task/scenario is, the better performance gain can InsightMapper achieve.

In the comparison experiments, we keep all experiment settings exactly the same for MapTR and InsightMapper, except proposed plugged-in modules.

5.Vanilla attention and No mask

Sorry for the inappropriate definition and insufficient explanation. Vanilla attention indicates attention layers used in MapTR, which treats inner-instance and inter-instance information equally. “No mask” indicates that the inner-instance self-attention layer does not block inner-instance self-attention with probability $\epsilon$. In section 4.4 and Figure.6, we proposed to randomly block some attentions between inner-instance points with probability $\epsilon$ for better robustness, inspired by Masked Autoencoder. The expression “No mask” could be misleading, and we changed it to “No random blocking($\epsilon=0$)” in the revised manuscript. You may find the results with different $\epsilon$ in Table. 11 of the appendix.

We thank the reviewer for your detailed and insightful comments and suggestions. Here is the response and clarification to your questions and concerns.

1.Attention patterns of MapTR and InsightMapper

The vanilla attention used in MapTR can learn both inter-instance and inter-instance message passing. The vanilla attention layer before the cross-attention layer (for deformable attention calculation) should treat these two kinds of message passings equally like DETR, otherwise, some instances may produce duplicate predictions since they cannot “clearly see” other instances so that instances cannot collaborate with each other very well for global optimal prediction, which leads to severely degraded performance.

The problem with the above pattern is that all queries are always equally treated so that after the cross-attention, a point will optimize its position based on all other points with equal attention. Intuitively, points that belong to the same instance of it should project more influence. The proposed inner-instance self-attention is located after the cross-attention, and it allows a point to pay more attention to its inner-instance neighbors, to make the optimization direction of all points within an instance consistent. Otherwise, the results might be severely affected by the zig-zag issue.

In short, the vanilla self-attention layer before the cross-attention layer assures that instances can “see” each other clearly, just like what the raw DETR does. The inner-instance self-attention layer after the cross-attention layer enables points to pay more attention to their inner-instance neighbors for local refinement. Both self-attention layers cannot be removed but they have different functions.

In the appendix, we provide detailed ablation studies in Figure. 8 and Table. 12 about attention layer position and decoder network structure. They should be helpful to further justify the design of InsightMapper.

2.Difference with MapTR-V2

In answer 1, we analyzed the function and properties of two kinds self-attention layers before (I.e., vanilla self-attention layer) and after (I.e., proposed inner-instance self-attention layer) the cross-attention layer. Both kinds of self-attention layers are essential for expected results. Therefore, even though MapTR-V2 decomposes the vanilla self-attention layer into inter self-attention and intra-self-attention, it only works on the self-attention layer before the cross-attention layer, and it fails to aggregate inner-instance information after the cross-attention layer for line refinement. Queries in MapTR-V2 still cast equal attention to all points, instead of paying more attention to inner-instance neighbors.

3.Hybrid query generation and hierarchical query generation

Thanks for the question. Indeed $q_2^P$ is spatially closer to $q_1^P$ than $q_{10}^P$. But the hierarchical query scheme introduces unexpected correlations between the i-th points of different instances. For example, $q_{1,1}$ indicates the 1-st point in the 1-st instance, and $q_{10,1}$ indicates the 1-st point in the 10-th instance. These two queries should not have correlations at the very beginning, otherwise the optimization of $q_{1,1}$ might affect that of $q_{10,1}$. The unexpected correlation cast unwanted constraints on the model optimization, which leads to degraded performance. In this way, the i-th points of all instances have such unwanted correlations (i.e., $\{q_{1,i},q_{2,i},...,q_{N,i}\}$) and this unwanted constraints should be blocked. In InsightMapper, the proposed hybrid query generation blocks such correlations (points in different instances are independent) and better performance is achieved.