A Siamese Transformer with Hierarchical Refinement for Lane Detection

Thank you for your thoughtful feedback! We address your questions and concerns below. If any other concerns remain, we’ll be happy to discuss them further. If the concerns are addressed well, we would appreciate it if you could consider raising your score.

[The motivation of this paper is not very clear.]

Attention-based methods (e.g., CLRNet) have shown promising capability in lane detection. However, since these methods mainly rely on finer-scale information to identify the position of each key point, their detection results may have large deviations when there are local occlusions or blurring. Furthermore, due to the shortcut to the multi-head self-attention mechanism which neglects the characteristics of different frequencies, there is a gap in accuracy between the contemporaneous Transformer-based methods and CNN-based methods. To tackle these issues, we propose a high-to-low hierarchical refinement Transformer structure to refine key points of lane lines, which integrates global semantics information and finer-scale features.

[It is unclear why the authors refer to several shared-parameter transformer layers as a Siamese transformer, as this seems to be merely a case of parameter reuse.]

To fully integrate global semantics information and finer-scale features, we propose a high-to-low hierarchical refinement Transformer structure for lane detection with shared parameters, which helps identify key points especially when roads are crowded or affected by blurring. The high to low features extracted by the backbone are fed into the Transformer structure with shared parameters and we calculate the loss for all the layers, which is the same definition as the Siamese structure [1][2]. As shown in Fig.5 of the manuscript, our proposed method can refine the key points of lane lines from higher levels to lower levels with fewer parameters by using shared parameters.

[The proposed high-to-low transformer in this paper is very similar to Mask2Former, thus offering limited novelty.]

Our method is different from Mask2Former. Indeed, both methods use hierarchical features as input to the Transformer. However, Mask2Former inputs multi-scale features into a Transformer decoder, which includes 3 transformer blocks, resulting in a large number of parameters and low efficiency. Instead, we feed information from each layer to the Transformer block with shared parameters and design the LATR structure to unify the feature information of different scales. This design not only enables the network to learn the feature information of different scales from high to low but also maintains a low number of parameters and high network efficiency.

[The paper uses a stronger image backbone (Swin vs. ResNet) compared to previous methods, making the comparison unfair. To provide a fair comparison, the authors should replace the backbone of previous methods (e.g., CLRNet) with Swin and then compare the results.]

For the choice of backbone network, we found that the Swin Transformer with hierarchical features can better capture image features for subsequent processing. Thus, we chose Swin Transformer as our backbone network, which is an open-source backbone widely used by other works. Accordingly, we designed the LATR and Siamese Transformer structures to reduce the overall number of network parameters. The results show that our approach achieves better results with higher FPS and lower GFlops. We also replace the backbone of the previous methods (e.g., CLRNet) with Swin Transformer and train these models in the same setting. The results are shown below.

The results show that our method performs better with the same backbone compared with previous methods. We’ll add these results to the final vision of our paper if accepted.

[The writing quality of the paper is poor.]

Thank you for your advice. We’ll improve the writing of the paper.

Reference

[1] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr. Fully-convolutional Siamese networks for object tracking. In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14, pages 850–865. Springer, 2016.

[2] He A, Luo C, Tian X, et al. A twofold siamese network for real-time object tracking[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4834-4843.

Thank you for your constructive comments! The detailed responses to each concern are given below.

[Currently, only one LATR module is used for each pyramid level. It is advisable to add more such modules to assess the performance, i.e. whether more LATR modules are redundant or improve the performance.]

We add a LATR module for each pyramid level, and the results are shown below.

From the results, one can see that adding more LATR modules helps improve the accuracy of lane detection recognition. Considering the efficiency, we think using two LATR modules for each pyramid level is more appropriate.

[Multiple LATR modules can be used at the lowest or second lowest feature level. It would be great to see the results during the rebuttal.]

We add 2 LATR modules for the lowest and second lowest feature levels. The results are shown below.

It can be seen from the results that more LATRs added at the lower feature levels can improve performance, while not leading to much increase in parameters. Hence, we think it is effective to add two LATR modules on both the lowest or second lowest feature levels.

Thank you for your thoughtful feedback! We address your questions and concerns below. If any other concerns remain, we’ll be happy to discuss them further. If the concerns are addressed well, we would appreciate it if you could consider raising your score.

[From the perspective of the topic selection, 2D lane detection is less valuable than 3D lane detection. More importantly, this topic is not fully suitable for the NeurIPS conference.]

2D lane line detection in different scenarios, especially under extreme conditions, is an important yet challenging task. It helps in detection and in the field of autonomous driving. Work on this topic has been published in past NeurIPS conferences, e.g., CARLANE[1], Openlane[2], 3D-LaneNet+[3], and BEVFusion[4].

[Its overall framework is quite similar to CLRNet, and the proposed structured Loss has also been preliminarily explored in CLRNet and UFAST networks.]

Both our network and CLRNet use multi-scale features extracted by the backbone. Many methods such as CondlaneNet use multi-scale features as input so that the network can learn features at different scales. But the next step is different. Our network employs a Siamese Transformer structure that feeds the different scale features directly into our designed Transformer structure called LATR with shared parameters. To follow up the structure of the Transformer so that the network is end-to-end, we employ Swin Transformer as the backbone of the entire network. In contrast, CLRNet employs FPN to further extract multi-scale features, and then uses a detection header to output lane line information, which is different from our work. For the loss, we propose a novel Curve-IoU loss to supervise the fit of lane lines at different locations, which helps the regression of the curves. Our proposed loss is dedicated to the fitting of distal curves, which is different from previous work.

[The authors only reported results under the Swin Transformer, which makes me skeptical about the fairness of the SOTA comparison.]

For the choice of backbone network, we found that the Swin Transformer with hierarchical features can better capture image features for subsequent processing. Thus, we chose Swim Transformer as our backbone, which is an open-source backbone widely used by other works. Accordingly, we designed the LATR and Siamese Transformer structures to reduce the overall number of network parameters. The results show that our approach achieves better results with higher FPS and lower GFlops. For a fair comparison, we also replace the backbone of the previous methods (e.g., CLRNet) with Swin Transformer and train these models in the same setting. The results are shown below.

The results show that our method performs better with the same backbone compared with previous methods. We’ll add these results to the final vision of our paper if accepted.

[The FPS speed does not have an advantage, and the model size (params) is not reported.]

Our approach achieves the best results on many open-source datasets with higher FPS and lower GFlops. Typically, the size of GFlops reflects the size of the model parameter count. We’ll add the number of parameters of our model to the final paper version if it is accepted. The model size of our proposed network is shown below.

Reference

[1] CARLANE: Stuhr, B., Haselberger, J., & Gebele, J. (2023). CARLANE: A lane detection benchmark for unsupervised domain adaptation from simulation to multiple real-world domains. In NeurIPS.

[2] Openlane: Wang, H., Li, T., Li, Y., ... Li, H. (2023). OpenLane-V2: A topology reasoning benchmark for unified 3D HD mapping. In NeurIPS.

[3] 3D-LaneNet+: Efrat, N., Bluvstein, M., Oron, S., Levi, D., Garnett, N., & Shlomo, B. E. (2020). 3D-LaneNet+: Anchor Free Lane Detection using a Semi-Local Representation. In NeurIPS.

[4] BEVFusion: Liang, T., Xie, H., Yu, K., Xia, Z., Lin, Z., Wang, Y., Tang, T., Wang, B., & Tang, Z. (2022). BEVFusion: A simple and robust LiDAR-camera fusion framework. In NeurIPS.

Thank you for your constructive comments! The detailed responses to each concern are given below.

[Have you considered using self-supervised learning techniques to reduce the dependency on large labeled datasets? ]

We have considered using self-supervision to reduce the dependence on large labeled datasets and to further improve the recognition accuracy of our approach in different scenarios. We noticed that some self-supervised work performs well on different tasks (e.g. CLLD [1]). We’ll include this part in future work.

[How does the model perform in extreme weather conditions (e.g., heavy rain, snow) that can significantly affect lane visibility?]

The model is able to recognize lane lines under extreme conditions. In the manuscript, "Extreme Weather" in Table 1 and "Hlight" in Table 2 show our lane detection performance under extreme weather conditions.

[Are there any plans to release the code and trained models to facilitate reproducibility and further research?]

We’ll release the code and pre-trained models once the paper is accepted.

Reference

[1] CLLD： Smith, J., & Doe, A. (2023). An innovative approach to deep learning. arXiv.