ESDMotion: End-to-end Motion Prediction Only with SD Maps

Q1. [Model presentation] The presentation of the method (section 3) could be significantly improved.

Thank you for your suggestion. We re-write the method section to make it clear and hierarchical.

• In Section 2.1, we present the overall structure of ESDMotion (2.1.1) and the modules required to extend the original motion prediction model into SD map oriented end-to-end framework (2.1.2). We indicate which modules are from existing work and which are proposed by us.
• In Section 2.2, we discuss the challenges faced by anchor-free models when using SD maps as input, and introduce the corresponding solution, the Enhanced Road Observation module.
• In Section 2.3, we address the challenges and solutions for anchor-based models, specifically the Pseudo-Lane Expansion and the newly proposed Adaptive Pseudo Lane Expansion.

Please check our updated paper and we sincerely thanks for your detailed suggestion.

Q2. [Pseudo-lane expansion] It looks like the proposed pseudo-lane expansion simply amounts to increasing the size of the sampling kernel in denseTNT.

Thanks for your in-depth thinking. We clarify the difference below.

Simply increasing the sampling kernel size to achieve a coverage similar to pseudo-lane expansion would require a kernel size of nearly 10. This would lead to an excessively large number (2000+ for a lane) of densely sampled points, significantly reducing model efficiency.

In contrast, Pseudo Lane Expansion uses smaller sample kernel and generates pseudo lanes that hypothesizes the approximate locations of potential drivable paths. Through lane scoring, the model identifies the pseudo lanes most “real” and most likely to represent road structures. Dense sampling is then applied only around these selected pseudo-lanes. This approach reduces the number of densely sampled points and is specifically designed to adapt to the coarser and less aligned SD maps. It improves efficiency and offers greater interpretability. There is a figure showing the difference. The Table below shows that our Pseudo Lane Expansion achieves better performance than simply increasing the size of the sample kernel.

We add the above discussions and experiments to Appendix C.1 of our updated paper and thanks for raising your advice to better illustrate the advantages of ESDMotion.

Q3 [Prior works]Add prior works that try to infer HD online maps from SD-maps and sensor-data.

Indeed, these are pioneering and inspiring works for ESDMotion. We add more discussions about them to our updated paper and thanks for your advice to better position this work.

Thanks for your reply! We response below:

• Regarding Q4, your proposed two-step setting could be one way to make the comparison more fair. However, [1] and [2] are not open sourced yet while [3] is a survey. [1] does not give any link in their paper (https://arxiv.org/pdf/2311.10517) and we could not find a github repo after searching while [2] is an empty repo yet (https://github.com/missTL/PriorDrive).
• Indeed, your understanding is correct. We find a substitute way to solve your concern regarding the fairness: There are two maps: a predicted HD-maps from MapTRv2-CL and the GT SD maps that are concatenated together and used as the input map.
• In this way, these methods have the SD map as inputs as well. In our humble opinion, it is fair since all methods have SD maps when conducting motion prediction while online mapping based methods have the extra advantages of HD map supervision. We are wondering about your opinion regarding this setting.

Regarding the typos, we will revise them.

We want to express our sincere thanks for your in-depth reading and thinking of the paper. Your commet helps us dig deeper into the new setting and demonstrate more comprehensive comparisons with existing works, especially on the points of fair comparison and relations with DenseTNT under the unexplored sparse SD map setting. Additionally, your careful writing advice also helps us a lot, which leads to a more clear Method section.

As the first work to explore in this direction, we humply request you to consider improving the score if our response solves your concerns. We believe our findings and solutions could be insightful for the future works in the field.

Q4. Although SD maps are globally covered, there are small areas like parking lots (which indeed exist in nuScenes) that are not covered. How does DenseTNT handle such situations?

This is indeed a common limitation of anchor based methods like DenseTNT which we believe should not be the reason to punish the ESDMotion.

Additionally, inspired by your concerns, we try to make adaptations to DenseTNT to handle such special scenarios. In areas not covered by maps, we define a set of pseudo lanes at specific angles to simulate the potential driving directions of the vehicle, as shown in this figure. With the positions of other vehicles and visual information provided by the BEV feature, the model could select appropriate goal points and paths in these situations.

The DenseTNT adapted to empty-map situation is named DenseTNT+, and the ESDMotion with this adaption is ESDMotion+. This adaption improves performance.

We add the above discussions and experiments to Appendix C.1 of our updated paper and thanks for expressing your concerns to improve the method.

Q5. Why is ESDMotion unable to outperform BEVPred on DenseTNT?

The reason is that: Anchor-based methods like DenseTNT use lanes to generate goal candidates. The sparesity and misalignment of lanes in SD map would lead to missing of nearby candidates and thus bad performance. As a result, compared to anchor-free methods like HiVT, the gains of DenseTNT is undesirable.

Inspired by you and other reviewers' thoughts, we improve DenseTNT under SD map scenario in two ways:

1. There are small areas like parking lots that are not covered by maps. For these areas, we define a set of pseudo lanes at specific angles capturing the potential driving directions to generate goal candidates, as shown in this figure. This updated version is named DenseTNT+ and ESDMotion+. As shown in Table below, such modifications explicitly improve the performance of DenseTNT.

2. In the origin version of Pseudo Lane Expansion, we expand a fixed number of parallel lines at uniform distances, which introduces too many candidates and thus hurts performance. To reduce those unnecessary candidates, we propose to dynamically adjust the parameters of Pseudo Lane Expansion based on the position and density of SD map. Specifically, for misaligned SD map as shown in figure (upper part), we generate more pseudo lanes on the side closer to the vehicle and reduce the number on the opposite side. To deal with different density of SD lanes as shown in figure (lower part)/Figure 5 in our paper, we increase the number of pseudo lanes to ensure better coverage for spare ones while decrease the number for dense ones. This further updated version is named ESDMotion++. As shown below, the performance is further enhanced, which

We add these discussions, methods, and experiments into our updated paper. We sincerely thank you for the constructive comments.

Thanks for your acknowledgement and kind advice. Regarding your concerns, we give responses below:

Q1. The statement of “only use SD maps” is misleading readers.

Thanks for your suggestion. We change our title to ESDMotion: SD Map Oriented Motion Prediction Enhanced by End-to-end Learning. We revise all relevant statements in our updated paper and describe ESDMotion as a motion prediction method that “utilizes only SD map for map information”. This phrasing is more precise and clearly distinguishes our approach from methods that leverage both SD map priors and HD map supervision. Any further advice is welcomed.

Q2. The emphasis on the end-to-end architecture of ESDMotion as a contribution is questionable.

We agree that with the rapid development in recent years, end-to-end learning in autonomous driving becomes more and more common. Our method is certainly not the first end-to-end approach.

However, our contribution lies in that:

• We aim to address a previously unexplored problem: using SD maps as the sole map information for motion prediction.
• We find out issues (low resolution and poor alignment) caused by using SD maps. To overcome these issues, we design an end-to-end framework and tailored modules to better utilize SD maps.
• As a result, our method demonstrates that even with the low-cost SD maps, it is possible to achieve motion prediction performance comparable to that using HD maps
• In fact, our approach is the first to specifically design a method for SD maps in the context of motion prediction and the first end-to-end approach addressing this problem. This focus on SD maps as a practical, cost-effective alternative represents the core of our contribution.

We compare the contributions of the three works below:

Q3. From Table 2, it can be seen that for HiVT, the minADE and minFDE using HDMap and SDMap are quite similar. While ESDMotion claims to improve minADE and minFDE through an end-to-end architecture or BEV features, as I mentioned in point 2, I believe this end-to-end architecture should not be counted as a contribution of this paper. This improvement can also be observed in the comparison between BEVPred and offline HDMap in Table 1.

As an anchor-free approach, the precision of the map (SD and HD) has a relatively smaller impact on HiVT comapred to anchor-based DenseTNT. In fact, for the original HiVT model, there is still a measurable gap in performance between using GT HD map and GT SD map (0.013 in minADE). Our method reduces this gap (to 0.0035 in minADE). As mentioned in response to point 2, the primary goal of our work is to leverage SD maps rather than merely building an end-to-end model. The end-to-end part is to fix the issue caused by SD maps. Our method demonstrates that by using more accessible SD maps, we can achieve performance that surpasses current methods relying on HD maps. We believe the SD map oriented designs constitutes a meaningful contribution of this paper.

Thanks for your acknowledgement and kind advice. Regarding your concerns, we give responses below:

Q1. The baseline methods (DenseTNT and HiVT) are too old.

We agree that there are more advanced methods emerged recently. We choose DenseTNT and HiVT as baselines primarily for fair comparison with existing online map based motion prediction methods (MapUncertaintyPrediction and MapBEVPrediction). According to your kind advice, we implement a more recent motion prediction model MTR, into the ESDMotion framework. Due to time constraints, we complete four pioneering experiments to provide a preliminary reference for its performance.

Results with MTR:

Q2. The proposed ESDMotion aims for end-to-end motion prediction however it still relies on SD maps as extra inputs, which is different from previous end-to-end works.

Indeed, the two approaches you mentioned are pioneering and valuable works in the field of end-to-end autonomous driving. The two approaches both propose to end-to-end conduct detection and motion. PIP additionally conducts online mapping.

However, we would like to clarify that:

• The cooperative relations of detection and motion prediction is not the focus of this work. In this work, our primary contributions are replacing HD Map with low-cost SD Map for motion prediction.
• The protocols of the two pioneering works introduce the influence of detection into the motion prediction results. To study the influence of map in a decoupled way, we adopt the protocols of MapUncertainty (CVPR24 Best Paper Final List)/BEVPred (ECCV24) so that the conclusion would not be influenced by the detection module. End-to-end feature usage are our solutions for the issues caused by using SD maps.
• Compared to the ViP3D requring HD Map as inputs and PIP requiring HD Map for supervision, ESDMotion only requires the easily accessible SD Map (for example, Google Map), which could significantly reduce the cost.

We add the above discussions to Appendix D of our updated paper and thanks for mentioning the two important related works to better position this work.

Q3. The use of SDMap feature fusion as a way to enhance BEV features is already common and lacks novelty.

Indeed, in the field of online mapping, there are already several works utilizing SD map as priors for HD map generation in 2024.

However, we argue that our work has unique contributions:

• We are the first to study the influence of directly using SD Map for motion prediction. The study aims to even not conducting HD map online mapping, which further reduces the cost.
• We find out issues (low resolution and poor alignment) caused by using SD maps for motion prediction and propose tailored solutions for it.
• By applying tailored designs, using SD maps could achieve comparable performance with using HD maps under fair settings, as shown in the updated Table 1.

Q4. The nuScenes dataset mostly contains simple road structure scenarios (PARA-Drive, BEVPlanner), in some complex scenarios, will the ESDMotion scheme proposed by the authors perform more excellently, for example, compared with PARA-Drive?

This is an interesting point. Unfortunately, PARA-Drive is not open-sourced yet (https://xinshuoweng.github.io/paradrive/), and BEVPlanner is designed for the planning task even without a motion prediction module, which differs from our motion prediction task. Therefore, we are unable to directly compare our method with these approaches.

We agree that exploring the application of SD maps in more complex road scenarios, as well as their potential use in planning tasks, is an intriguing and worthwhile direction for future research. We add the above discussions to Appendix E of our updated paper as future work.

Thanks for your acknowledgement and kind advice. Regarding your concerns, we give responses below:

Q1&Q2. The Pseudo Lane Expansion strategy proposed is a relatively coarse goal point sampling approach, and may result in misalignment inaccuracies; The anchor-based trajectory prediction approaches does not demonstrate a performance advantage over methods that utilize online-predicted maps.

Thanks for your in-depth thinking. We agree that there could be potential misalignment of generated pseudo lanes with actual road structures. As a result, compared to predicted HD maps, the sparesity and misalignment of lanes in SD map would lead to missing of nearby candidates for anchor based methods like DenseTNT and thus worse performance.

Inspired by you and other reviewers' thoughts, we improve DenseTNT under SD map scenario in two ways:

1. There are small areas like parking lots that are not covered by maps. For these areas, we define a set of pseudo lanes at specific angles capturing the potential driving directions to generate goal candidates, as shown in this figure. This updated version is named DenseTNT+ and ESDMotion+. As shown in Table below, such modifications explicitly improve the performance of DenseTNT.

2. In the origin version of Pseudo Lane Expansion, we expand a fixed number of parallel lines at uniform distances, which introduces too many candidates and thus hurts performance. To reduce those unnecessary candidates, we propose to dynamically adjust the parameters of Pseudo Lane Expansion based on the position and density of SD map.

• As shown in figure (upper part)/Figure 5 in our paper, SD lanes may align well with the road center but may also misalign and extend outside the drivable area. In misaligned cases, the target vehicle is often farther from the SD lane. For misaligned cases, we propose to generate more pseudo lanes on the side closer to the vehicle and reduce the number on the opposite side, thereby minimizing the degree of misalignment.
• As shown in figure (lower part)/Figure 5 in our paper, the density of SD map is different in different areas. Thus, for those scenarios with sparse SD maps, we increase the number of pseudo lanes to ensure better coverage of the road surface. Conversely, in areas with dense SD maps (e.g., intersections), we decrease the number and spacing of pseudo lanes to avoid overlap and interference. This further updated version is named ESDMotion++.

We add the above discussions and experiments to Section 2.3, Table 1, and Table 3 of our updated paper and thanks for raising your concerns regarding anchor-based methods.

Thanks for your response and careful thinking! We are glad that we could discuss more regarding our contributions:

BEVPred can directly remove the supervision from HD map construction and perform the same end-to-end trajectory prediction based on BEV features as ESDMOTION.

• First, we agree that end-to-end is not our contribution, which has been done in BEVPred.

• However, our end-to-end design is tailored for SD Map. For example, as shown in Fig. 4, Equ. 5, and L263-L283, we design a deformable attention mechanism - Enhanced Road Observation, where it first deformably attends to all potential parallel areas and then fuses these information with predicted confidence h_i for those different areas. This deformable attention is specifically designed for the sparse SD map lanes so that it could adaptively find out meaningful areas. The proposed methodology is beyond the scope of BEVPred.

The BEV-SD interaction work in the HD map construction domain can incorporate SD priors into trajectory prediction. These works also identify and aim to solve the issues of low resolution in SD maps and the misalignment between SD and HD maps.

• We agree that there are multiple works combines SD Map to generate HD Map where their focus is on utilizing the SD Map as strong priors for HD map generation.

• Regrading These works also identify and aim to solve the issues of low resolution in SD maps and the misalignment between SD and HD maps, we argue that these issues are actually motion prediction specific, caused by the removal of reliance on HD map in motion prediction. Thus, our methodologies are different from online mapping methods as we focus on the downstream task - motion prediction. As mentioned in your summary, our high level goal is to expand SD instances to avoid the missing of map elements when predicting agents' future trajectories. We aim to let the motion prediction model takes these noisy expansion as inputs and adaptively selects useful information aided by end-to-end learning.

The contribution of this paper is essentially limited to using specific rule-based expansions of SD instances to improve the effectiveness of SD maps in trajectory prediction.

The reviewer indeed has very in-depth thinking of our methodology and we appreciate this point. We agree that the high level inspirations of all of our technical designs are to expand the SD map lane as they are sparse and sometimes inaccurate.

However, we argue that our designs should not be understood as rule-based post-processing methodologies. Enhanced Road Observation utilizes attention mechanism to find out meaningful ares. Pseudo Lane Expansion and Adaptive Pseudo Lane Expansion (Please check supplymental Section C) change the training process for anchor-based methods like DenseTNT under usage of SD Map. All of designs are innovations of learning algorithms. We aim to design proper ways to fuse the noisy expanded SD instances and we design end-to-end learning modules to do it.

I still believe the contributions of this paper are currently insufficient to justify the acceptance of this paper at ICLR, a conference on learning algorithms.

Beyond the technical contributions mentioned above, we would like to also discuss about the types of contributions one work could have. We argue that taking the initial steps to explore new possibilites are as equally important as methodology innovations. For example, the pioneering work Unc[I] explores the setting of adopting online mapping models for motion prediction. It gives in-depth analysis regarding the issues of online mapping models and proposes an simple yet effective solution - uncertainty. From the perspective of learning algorithms, the concept of uncertainty is not new. However, it shed insights regarding the topic and thus even entered CVPR24 best paper final list.

ESDMotion is the first to explore the possibilities of replacing HD Map with SD Map for motion prediction. Following your and other reviewers' kind advice, we conduct detailed experiments to compare the all potential settings of SD Map VS HD Map. We believe these findings are also insightful contributions, worth sharing in the community.

With all due respect, we deeply appreciate your in-depth thinking and your willingness of taking your valuable time to discuss about our contributions.

Thanks,

Authors of Submission 2920

[I] Producing and Leveraging Online Map Uncertainty in Trajectory Prediction. CVPR 2024