Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map

Weakness

1. Motivation. Current navigation maps such as Google maps or Apple maps already have lane-level traffic rules adhering to ego-vehicles position. To be specific, in the figure 1, information such as driving direction per lane (go straight or turn left), speed limitation, and bus lane effective data are already available in the navigation maps like Google maps. Could the author discuss in detail about the necessity of the task 1 and 2 stated in this paper? This is my biggest concern about this paper.
2. Key difference to OpenLane-V2. In Table 1, the main difference to OpenLane-V2 seems to be the formatted driving rules. While in OpenLane-V2 there are also some annotations about driving directions. To be specific, it would be great if the author could provide a more detailed comparison between MapDR and OpenLane-V2, highlighting specific differences in the formatted driving rules, level of detail, or types of rules captured, and explain how these differences translate to practical advantages for autonomous driving applications.

1. The LLM-based autonomous driving works[1,2,3,4] are poorly mentioned in this submission. As the target of this dataset is estimate the whether autonomous driving models can correct actions given the visual input of lane-based rules, it is important to discuss this field in the related work and evaluate their performance with your benchmark.
2. There is poor comparison for the proposed baseline methods in this paper. The models are only compared to the base BERT and ALBEF models. No autonomous driving related works are compared.
3. There is no benchmark result for autonomous driving related works on their dataset.
4. As the target is to reason lane-based rules from visual inputs to perform correct action, the necessity to split this task to two sub-tasks is not discussed. It seems the two sub-tasks can be solved end-to-end. More justification for these two-sub tasks are expected.
5. The diversity of the proposed dataset seems to be limited. RS10K contains 10k images from 31 cities in China. However, MapDR in this work mainly covers Beijing and Shanghai in China.

[1] Zhenhua Xu, et.al. DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model. Arxiv 2023

[2] GPT-Driver: Learning to Drive with GPT. NeurIPS 2023 Workshop

[3] Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving. ICRA 2024

[2] DriveLM: Driving with Graph Visual Question Answering. ECCV 2024

1. Unclear Motivation for the Proposed Task: If the primary goal of this work is to enhance the efficiency of HD map construction, as suggested by Figure 6, then using HD maps as input makes sense. However, if this is indeed the goal, the need for the proposed dataset and method remains unclear. What are the limitations of using human annotators to label and associate traffic rules with lanes? The authors should clarify the motivation to strengthen the case for this work.

2. Ambiguity in Problem Formulation: The introduction frames the work from the perspective of visual traffic scene understanding, which involves building a visual model that predicts road structures, identifies lanes and traffic signs, and links traffic sign semantics with the corresponding lanes. If this is the motivation, then I agree that MapDR and the proposed task represent a valuable and challenging contribution to the community. In this scenario, the proposed framework at inference should not include vectorized maps as input. Therefore, the motivation and formulation of the proposed task require clarification, as it may necessitate a different set of experiments altogether.

3. Justification for Constructing a New Dataset: The authors should clarify the rationale for collecting a new dataset specifically in Shanghai and Beijing. For example, they might demonstrate that existing datasets lack sufficient traffic sign data, which limits the study of tasks such as rule extraction from signs and rule-lane correspondence reasoning. Providing detailed statistics on the limitations of existing datasets could help underscore the necessity of collecting new data.

4. Ensuring Annotation Quality: The authors should detail the methods used to ensure annotation quality. For instance, how many reviewers verified the accuracy of labeled rules? Without these specifics, it is difficult to assess the dataset’s quality and, consequently, the contribution of this work, given that dataset creation is a primary contribution.

5. Need for a Simpler Baseline than VLM: Establishing a rule-based baseline could justify the necessity of using Vision-Language Models (VLMs) for this task. For instance, one might first identify lanes and map attributes from HD maps, then use OCR outputs to establish rule correspondences. Evaluating this rule-based approach could provide insights into the complexity of the proposed task and determine whether simpler methods might suffice, reducing the need for more complex models like VLMs.

• Having a more diverse set of locations, and corresponding rule sets would greatly enhance the relevance of this dataset. Specifically,
• The paper would benefit from pointing to the definitions of the rules, and how they are established, as in the paper and in the supplied dataset they are often referred to with an index, but no specification which will allow to determine whether the set of rules is comprehensive. How are they considered a full set / how specific are they? How complete is the coverage of rules over the environment? It’s hard to tell.
• How are the rule extraction handling, e.g. intersections of 2 lanes or more (e.g. merges or roundabout entries)? Hard to say.
• It seems the dataset only allows image-to-rule/lane detection, but ignores the non-visual association question of match vehicle behaviors to rules, which is an important cue, especially since the ultimate goal of rule detection is to enable planning and reasoning in AD/ADAS systems. While it seems like the authors had a localization system running, but did not use detection and tracking algorithm, even an imperfect solution there would have allowed a much broader set of algorithms to be tested.
• Would be good if the authors better justified and ablated possible architectures, such as the use VLM vs OCR addition, or the pipeline of (VLM+OCR) -> MEE -> output, as opposed to other choices (such as joint inference, or iterative updates) or shed light on some of the other design choices. As a dataset paper that explores a pretty diverse phenomena (traffic rules), the dataset is relatively limited, which raises the bar on the algorithm, even if it’s considered a baseline for the problem.
• How were the 3 locations selected? For a dataset that is examining extraction of human behavior, the 3 location being in China, and Chinese signs, etc, may limit the reproducibility of the results (cf. the author’s claim that only Qwen got reasonable results from all the LLMs tested), would similar data be possibly annotated on existing datasets that have English semantics, or other rules? The 3 locations feel like they are very specific deployment sites, rather than a full attempt for a dataset, but I might be wrong.

Additional minor comments include:

• L281 - “sever” -> “server”
• L300 - E \in {0,1}^{|R|x|L|} should inclusion, not “\in”
• L969 - “seed” -> “seeds”
• Would be good to cover other approaches for rules in driving, e.g. logic-based approaches, possibly as an additional subsection in the related works, with possible example works such as “Liability, Ethics, and Culture-Aware Behavior Specification using Rulebooks”, "Vehicle Trajectory Prediction Using Generative Adversarial Network With Temporal Logic Syntax Tree Features", “Formal methods to comply with rules of the road in autonomous driving: State of the art and grand challenges”, “Formalization of Interstate Traffic Rules in Temporal Logic”, and others.

1. The main contribution of the proposed MapDR dataset over existing datasets, such as OpenlaneV2, is the formatted rules. However, the paper mentions the comparison on this point briefly. Please provide more discussion on why this contribution is significantly beneficial for autonomous driving.

2. The dataset lacks nighttime scenarios, which can significantly affect the visual effect. It would be beneficial to include more tidal flow lane scenarios because tidal flow lane rules are important for traffic compliance.