DV-3DLane: End-to-end Multi-modal 3D Lane Detection with Dual-view Representation

Thank you for your encouraging comments and for recognizing the strengths of our work. We appreciate your acknowledgment of the "Really novel idea that does this task of lane detection using query clustering from multiple perspective", "SOTA results" and your observation regarding "runtime optimization". We assure you that we will release our code.

Q1: Regarding comparison with GroupLane.

We compare our method against the state-of-the-art model LATR. It is unfair to ask us to compare with GroupLane, which is under review at ICLR2024, the same as us. More critically, based on the results from GroupLane, when utilizing the same ResNet50 backbone, GroupLane exhibits inferior performance than LATR. Therefore, we would like to claim that our comparison is fair and competitive.

Q2: Regarding using deformable attention.

We acknowledge the reviewer's concern regarding the innovation aspect of using deformable attention for aggregating multi-modal features. It is important to clarify that:

1) Our contribution does not reside in using deformable attention per se, a well-established method widely employed in multi-modal/multi-view feature fusion.

2) Instead, we propose the 3D DV Deformable Attention, an extension and adaptation of the original deformable attention.

In Section 3.3, we meticulously outline the architecture and functionality of our 3D DV Deformable Attention, accompanied by Algorithm 2 and Figure 5 for clarity. Besides, we conduct a thorough comparison of our method with other methods that are also built upon the original deformable attention, showcasing the effectiveness of our approach. This comprehensive evaluation is summarized in Table 5, providing a nuanced understanding of our method's performance relative to existing approaches.

W2: Regarding questions to Table 4.
In fact, we have explained this phenomenon in Section 4.5, titled 'Effect of Unified Query,' in the main text. The results indicate that solely using queries from one view does not effectively aggregate features from another view. However, initializing queries with nn.embedding allows queries to learn how to match features from different views. Furthermore, by equipping our proposed clustering strategy, the model achieves the best results, which strongly supports the efficacy of our query clustering in integrating multi-modal queries from different view spaces. We believe this strategy could facilitate other multi-view or/and multi-modal studies as well.

W3: Regarding the contribution of feature fusion in the backbone and decoder.

We appreciate the reviewer's comment regarding the feature fusion aspect of our work. Our primary goal is to advance the state of 3D lane detection by introducing an effective end-to-end multi-modal framework. To achieve this, we have developed the dual-view model to harness feature representations from different views.

While we acknowledge that our work incorporates feature fusion in both the backbone and decoder, it's essential to emphasize that our contribution extends beyond this specific aspect. Although we have demonstrated the effectiveness of our fusion methods compared to alternative approaches, our research is not solely centered on the fusion techniques themselves. Instead, it addresses the broader challenge of multi-modal exploration for 3D lane detection and presents innovative solutions to tackle these challenges.

One of our key contributions is the introduction of lane-centric clustering, a novel technique designed to cohesively synergize queries from different views. This aspect is integral to our approach and constitutes a significant innovation in our field.

Thus, while the feature fusion in the backbone and decoder is a component of our model, the breadth and depth of our contributions, particularly in multi-modal exploration and lane-centric clustering, are what we believe to be the major points of novelty and impact in our work.

We thank the reviewer for throwing time into reviewing our paper. To address the reviewer's concerns and questions, we would like to first highlight our contributions:

1) To bridge the gap of multi-modal exploration in 3D lane detection, we propose the DV-3DLane, an end-to-end multi-modal 3D lane detection framework. To the best of our knowledge, this is the first end-to-end multi-modal 3D lane detection framework. Besides exploring multiple modalities, our method also harnesses the power of both Perspective View (PV) and Bird's Eye View (BEV) representations. All four other reviewers acknowledge this contribution.

2) To achieve the goal of end-to-end multi-model 3D lane detection: First, we propose to fuse multi-modal features bidirectionally among PV and BEV spaces and further generate a lane-aware query set within each space. Then, to unify these two query sets, from two spaces, we propose a lane-centric clustering strategy. Afterward, to effectively aggregate PV features and BEV features into the unified queries, we propose the 3D DV Deformable Attention to consistently sample features from two different spaces via 3D deformed points, which are generated by predicting both 3D reference points and their 3D offsets.

3) We conduct comprehensive experiments on the OpenLane benchmark and our method significantly improves the performance, compared with the previous multi-modal approach M$^2$-3DLaneNet and the SOTA LATR. Notably, we significantly cut the errors in both X and Z directions within near/far ranges by about 50%. Moreover, we propose to use a more strict evaluation metric, 0.5m (previous 1.5m), to study the problem in a nuanced manner, towards enhancing real-world driving safety.

Below, please find our response to each of the comments and questions:

Presentation: 2 fair This contradicts your 2nd strength point:

The paper is well written and easy to follow.

And there is no specific weak point or question about the presentation. We, therefore, kindly appeal the reviewer’s re-evaluation of the “presentation” rating and final rating.

W1-1: Regariding performance results in Table 3.

First, we highlight that Table.3 does not compare the camera-only method and the multi-modal one, as stated by the reviewer. Instead, Table 3 focuses on comparing various designs in the context of multi-modal fusion, with each design incorporating both LiDAR and camera data. The results in Table.3 highlights the effectiveness of our bidirectional fusion within dual-view spaces compared to non-fusion/unidirectional-fusion ones (during the backbone stage).

As highlighted by the reviewer's observation from Table 1, the majority of methods in the literature lean towards a single-modality solution, with M$^2$-3DLaneNet being the exception, albeit now surpassed by subsequent works like LATR. Recognizing the significance of point cloud data, in tasks like 3D detection, we aim to explore the underexplored territory of combining LiDAR and image data for 3D lane detection. Our work addresses this gap in research (as mentioned by reviewer 7Rt7), emphasizing the relevance and importance of leveraging both modalities in this field.

W1-2: Introduce LiDAR to some methods for a comparative performance evaluation.

First, we highlight that modifying other methods into multi-modal ones is non-trivial. Besides, we compared M$^2$-3DLaneNet, which compared several multi-modal fusion approaches, e.g., PointPillar, BEVFusion results with inferior performance than M$^2$-3DLaneNet. The reviewer has acknowledged the multi-modal lack of research.

To specifically address your concern, we integrated the SOTA method LATR with LiDAR inputs, projecting all points into the image and using them as additional features in the network. The results clearly demonstrate the positive impact of LiDAR, although it falls below the performance achieved by our proposed method. This comparison underscores the unique contribution of our work.

We thank the reviewer for their positive feedback and address their concerns as below:

W1: Tolerance of calibration parameters to noise.

Thank you for your insightful comment. In response, we conducted experiments to assess the impact of noise on the calibration parameters.

Noise Settings:

Following the methodology of Yu et al. (2023) [1], we introduced two noise settings: 'Noise' and 'Stronger Noise (SNoise)', which include rotation and translation randomly. In the 'Noise' setting, the rotation angle ranges from 1 to 5 degrees, and the translation ranges from 0.5cm to 1.0cm. In the 'Stronger Noise' setting, these values are doubled for a more pronounced impact.

Results without training noise:

We first test our model, which has not been exposed to any added noise during its training phase, under various noisy conditions. The results, presented in the first row of the table below, show a notable performance decline as noise intensity increases. Specifically, under a 0.7 probability setting, the performance deteriorates from 63.5 to 32.4/31.4 in the 'Noise'/'Stronger Noise' settings. For subsequent analyses, we designate this experiment, utilizing our original model, as the baseline for reference.

Enhancing robustness via training:

To enhance robustness, we incorporated calibration noise during the training phase. This strategy substantially reduces the performance degradation caused by noisy calibration, as shown in the second and third rows of the table below.

Comparative analysis:

When compared to the baseline (first row), we can observe that training with calibration noise significantly strengthens our model's robustness. It effectively maintains comparable results in noisy calibration conditions. The model trained with 'Stronger Noise' exhibits greater robustness than that trained with less intense noise, underscoring the benefits of this training strategy.

Note: each result in the cell consists of F1-score / Accuracy.

We have included these robustness experiments in our updated Appendix A.3 for a comprehensive understanding, with a reference in purple font at the beginning of the main text on page 8. We appreciate your insightful feedback, which has been instrumental in strengthening our work.

W2: Generalizability

We appreciate your concern regarding the generalizability of our proposed method. OpenLane, being the first real-world and large-scale 3D lane dataset built upon Waymo, stands as a unique and valuable resource. The coupling of LiDAR data with OpenLane further enhances its significance.

Currently, our evaluation is limited to this distinctive dataset due to its pioneering nature and the accessibility of coupled LiDAR data. While we acknowledge the importance of assessing generalizability across diverse datasets, the current focus on OpenLane provides a comprehensive exploration of our multi-modal method within the constraints of available resources. We plan to extend our evaluation to additional datasets as they become available and feasible for our study.

We appreciate your understanding and consideration.

Reference

[1] Yu, Kaicheng, et al. "Benchmarking the robustness of lidar-camera fusion for 3d object detection." CVPR 2023.

W2-1:Regarding performance on OpenLane-1000 and implications of Camera-only/LiDAR-only methods.

We appreciate the reviewer's insightful query about the performance implications of our method with Camera-only and LiDAR-only modalities on the OpenLane-1000 dataset. To provide a comprehensive answer, we conducted two sets of experiments: one with single modality and single view, and another with single modality but dual views.

In the first experiment, we evaluated single-modality and single-view models, as detailed in Table.2 of the main paper, now applied to the OpenLane-1000 dataset. The outcomes, shown in rows #1 and #2 of the table above, present the performance using individual modalities.

In the second experiment, we delved into single-modality but dual-view approaches:

• For the image-based model, we adopted a strategy akin to BEVFormer, utilizing deformable attention to transform image features into BEV features. Then, we apply our dual-view decoder upon this, and the outcomes are illustrated in row #3 of the table above.
• For the LiDAR-based approach, we projected LiDAR point cloud features onto the 2D image plane to generate perspective-view features. The results of this approach are presented in row #4 of the table above.

Our results, as demonstrated in the table above, underscore that the dual-view representation significantly enhances the performance of baseline models in single-modal scenarios (comparing #1 with #3 and #2 with #4). This improvement confirms the effectiveness of our dual-view approach in learning 3D lane detection. Most notably, the combination of image and LiDAR modalities, coupled with our dual-view representation, achieves the best results, as shown in row #5. This synergy of modalities underlines the superiority of our proposed method.

W2-2: Regarding camera-only performance on the Apollo dataset.

To address your concern, we conduct our image-only model on Apollo following the Table 2 setting of our main paper.

W3: Including more methods in Figure.1

We appreciate the reviewer's valuable suggestion regarding enhancing Figure 1 by including additional methods.

Our current results for GenLaneNet and 3DLaneNet are referenced from prior works, like PersFormer [1], Anchor3DLane [2]. Unfortunately, the pretrained models for these methods are not yet publicly available, which hinders our ability to directly evaluate their performance under the 0.5m setting. We will attempt to reproduce these results and assess them in the 0.5m setting. We kindly request your understanding in this matter, given the time constraints of this discussion period.

Reference

[1] Chen, Li, et al. "Persformer: 3d lane detection via perspective transformer and the openlane benchmark." ECCV 2022.

[2] Huang, Shaofei, et al. "Anchor3dlane: Learning to regress 3d anchors for monocular 3d lane detection." CVPR 2023.

We thank the reviewer for recognizing that our method fills the gap of missing LiDAR modality in prior work.

W1-1:Regarding performance improvement and saturation under 1.5m setting

We appreciate the reviewer's observation and would like to clarify the evaluation results. First, we'd like to highlight the substantial error reduction by our method in both X and Z near/far ranges, showcasing impressive 46.12%/48.26% and 57.33%/49.04% improvements, respectively (see Table.1 of the main paper).

While the F1 score may not fully capture the extent of this improvement, especially given the leniency of the 1.5m evaluation threshold, we believe there's room for further optimization, such as exploring more effective lane line representations.

Meanwhile, to push beyond the results under the current lenient threshold, we propose to adopt a more strict threshold, 0.5m distance tolerance, for evaluation. This would offer a more nuanced examination of the model's accuracy, with the aim of encouraging further optimization and enhancing performance in alignment with considerations for real-world driving safety.

Further, to address your concern regarding comparison, we integrated the SOTA method LATR with LiDAR inputs, projecting all points into the image and using them as additional features in the network. As shown in the following table, the results clearly demonstrate the positive impact of LiDAR, although it falls below the performance achieved by our proposed method. This comparison underscores the unique contribution of our work.

It's worth noting that in the prior study, M$^2$-3DLaneNet, several multi-modal fusion approaches, such as PointPillar and BEVFusion, were compared, which exhibit subpar performance in the 3D lane task compared to M$^2$-3DLaneNet, falling behind our proposed method.

W1-2: Regarding the backbone choice

Thank you for your thoughtful feedback. We appreciate your observation regarding the backbone choice. Our intention was to design an efficient model, leading us to adopt a smaller backbone. However, in response to your concern, we conducted experiments with ResNet-50, demonstrating that a larger backbone can further enhance model performance. The results are presented in the table referenced in W1-1. We hope this clarifies our design choice and addresses your comment effectively.

W1-3: Regarding using OpenLane300 for the ablation

We follow previous works [1] [2] [3] to conduct ablation studies on the subset.

Reference

[1] Chen, Li, et al. "Persformer: 3d lane detection via perspective transformer and the openlane benchmark." ECCV 2022.

[2] Huang, Shaofei, et al. "Anchor3dlane: Learning to regress 3d anchors for monocular 3d lane detection." CVPR 2023.

[3] Luo, Yueru, et al. "LATR: 3D Lane Detection from Monocular Images with Transformer." ICCV 2023.

Thanks author for the detail reply and additional experiment provided.

W3: GenLane have their open source code on github and the result should be pretty easy to reproduce.

W4: I still find the term 'prior knowledge' confusing, work [1] and [2] shows example of adopting 'prior' in lane formation. What you describe is more like 'lane aware query' or 'lane aware anchor', than 'prior knowledge'. If you still want to use the word 'prior', you should provide explanation comparing prior method also adopting 'prior'. [1]Reconstruct from top view: A 3d lane detection approach based on geometry structure prior [2]Learning to predict 3d lane shape and camera pose from a single image via geometry constraints

Author should consider putting results on other dataset, like Apollo into the main paper with detailed comparison to prior method, as this is one of the earliest and most popular 3d lane detection dataset. Instead, you can put detail ablation to appendix. Beside, author should modify the paper as promised above to reduce misunderstanding in mismatch in experiment settings. And integrate all changed above to your paper.

Other than that, I am satisfied with the current work. I have changed my rating to 6.

We appreciate the feedback from the reviewer. Below, we address the reviewer’s concerns:

W3: We thank the thoroughness of the reviewer. GenLaneNet is indeed accessible on GitHub, based on Apollo. We have attempted to adapt it to the OpenLane dataset but didn’t achieve the results reported in prior work. Therefore, for the sake of precision in our comparison, we haven't included GenLaneNet in the figure under the 0.5m setting. Nonetheless, we are still attempting to reproduce the results and aim to evaluate them under the 0.5m setting. We appreciate your understanding and would love to share any additional insights gained from these efforts.

W4-1: Regarding to 'prior knowledge'

We acknowledge the potential misunderstanding, and to ensure clarity, we have modified our manuscript to consistently use the term "lane-aware knowledge", without using 'prior', to avoid any confusion with the previously used "geometry prior"[1] or "geometry constraints"[2].

These revisions have been incorporated into our updated manuscript, and we appreciate your diligence in bringing this concern to our attention. Please refer to the revised version for further clarification.

Results on Apollo:

We appreciate the valuable suggestion from the reviewer. It's important to note that the Apollo dataset is exclusively an Image-only dataset without LiDAR information, while our work specifically focuses on multi-modal 3D lane detection.

Further, given the constraints of space in the main paper and the need for a comprehensive table to present results on Apollo, we have opted to include the detailed experimental results in Appendix A.6. This ensures maintaining an integrated presentation in the main body of the paper. We believe this approach allows for a more in-depth examination without compromising the main narrative. we appreciate your understanding.

Singel-modality & Dual-view Results on OpenLane-1000:

We appreciate the valuable suggestion from the reviewer.

• We have integrated the single-modality results (Image-Only / LiDAR-Only) based on OpenLane-1000, LATR + LiDAR results, and the results of our ResNet50-based model (now termed DV-3DLane-Large) into the main results in the updated main text (See Table.1). Our previous ResNet34-based model is now termed as "DV-3DLane-Base" and ResNet18-based model is now termed as "DV-3DLane-Tiny" in the Table.1 of the manuscripts.

Correspondingly, the performance of DV-3DLane-(Tiny / Base / Large) are all included in the Figure.1 & Appendix various scenario study table.

Further, we have moved the bidirectional feature fusion ablation study into our Appendix A.5.

All the above changes have been included in our revised version, highlighted in red. Please kindly refer to our updated manuscripts.

[1] Reconstruct from top view: A 3d lane detection approach based on geometry structure prior.

[2] Learning to predict 3d lane shape and camera pose from a single image via geometry constraints.

W1: Typo and Presentation

Thank you for bringing the minor typo and visualization to our attention.

• We have fixed the typo, changing "Instruction" to "Introduction."
• We have addressed the concern regarding the size of row (d) in Figure 7 by making improvements to the plot. Moreover, We have included corresponding plot improvements in the Appendix. Additionally, a zoom-in reminder has been incorporated into the captions, highlighted in brown font for the convenience of your review (Please kindly refer to the updated manuscript).

We believe these adjustments effectively address the raised concerns. Your thorough review is greatly appreciated.

Q1: We thank you for your positive evaluation of our work and appreciate your suggestion to test our models on the OpenLane-Huawei dataset, which features more horizontal lines. We acknowledge the potential benefits of such an experiment; however, there are specific reasons we have not yet applied our method to this dataset during this discussion period.

First, the OpenLane-Huawei dataset, primarily designed for topology reasoning among centerlines and traffic elements, is based on multi-view images. Our current methodology is based on the single-view setting, and adapting it to multi-view contexts is non-trivial. We recognize the importance of this adaptation and plan to explore it in future studies.

Second, the OpenLane-Huawei dataset comprises two subsets: SetA and SetB. SetB, sourced from Nuscenes, Z values of centerline annotations are set to 0s built upon HD map. In this work, we focus on 3D space lane detection, making SetB unsuitable for our current approach.

Regarding SetA, sourced from Argoverse-v2, it does have valid z-coordinate information. Nonetheless, the OpenLane-Huawei dataset, centering on the vision-centric approach, does not officially provide LiDAR data. While it's possible to obtain this data from the Argoverse repository, the large scale of the dataset posed a challenge for us to download and integrate within our work during the limited discussion period.

In light of these considerations, we were unable to apply our method to the OpenLane-Huawei dataset during this discussion period. We sincerely appreciate your understanding in this regard. Moving forward, we plan to extend our method to incorporate both its camera and LiDAR data and will ensure to share our findings, as well as release the source code.

Thank you again for your insightful suggestions and support for our work.