Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?

1. Overclaiming: The paper argues that existing 2D VLMs struggle in achieving 3D perception, but it has been shown that by pre-training a 2D tokenizer, 2D VLMs can outperform traditional 3D detection methods [1]. The authors are suggested to compare the performance curves of the 2D and 3D tokenizers as the amount of traing data grows. Additionally, other works [2] utilize 3D encoders like depth maps and extrinsics for 3D scene understanding. How does DETR-style tokenizer compare to or improve upon the mothods in [2]?

2. Lack of Innovation: a) In terms of model designs, some studies [3] have already explored spatial understanding tasks using 3D tokenizers from a BEV perspective. Sec. 3.1's content appears superficial and lacks novelty, making it challenging to discern the paper's contributions. How does Atlas differs from the work in [3], particularly regarding the use of 3D tokenizers. b) Data preparation, task design, and evaluation metrics mentioned in [1] cover topics discussed in the paper. Could the authors provide a more detailed comparison with [1] to highlight any unique aspects of their work.

3. Based on Fig. 4 and supp Fig. 5, it is not clear whether the model can accurately follow instructions to generate reasonable driving trajectories. Instead, it appears to list all potential trajectories, raising doubts about the model's capability to achieve end-to-end planning tasks. How does the model select the most appropriate trajectory from the multiple options? The authors should provide specific examples or metrics that demonstrate the model's ability to follow instructions.

[1] Zhou, et al. "Embodied Understanding of Driving Scenarios".

[2] Zhu, et al. "LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness".

[3] Choudhary, et al. "Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving".

1. About the framework design: The necessity of connecting a 3D tokenizer to an LLM is questionable. A baseline experiment could clarify this by showing whether a simple combination of a 3D tokenizer and MLP could achieve similar planning results. The main advantage of incorporating an LLM seems to be interpretability, as LLMs may generate explanations, but whether they can control the trajectory as effectively remains unclear. Additionally, the LLM introduces limitations, such as reduced speed and difficulty in producing accurate trajectory scores. The paper would benefit from discussing the necessity of connecting an LLM to the 3D tokenizer.
2. About the performance: The performance when using the LLM is subpar, especially when ego states are not incorporated. As shown in Table 3, models without ego states underperform significantly compared to BEV-Planner. Conversely, experiments that include ego states are of limited value, as the network may rely on shortcuts derived from ego states rather than genuinely learning to predict trajectories.
3. About the benchmark: The paper lacks experiments on the CARLA simulator, which is critical as the nuScenes dataset alone is inadequate for validation. Although the authors mention that CARLA may lack realism, nuScenes predominantly consists of straightforward driving scenarios, heavily influenced by ego status, making it a limited and potentially biased metric for this work.

• While this work introduces advancements in spatial perception by incorporating 3D tokenizers, it would benefit from a comparison with recent open-source studies that explore large language models’ spatial understanding in driving scenarios, such as [1].
• The impressive performance in open-loop planning is acknowledged, yet the detailed aspects contributing to this strength remain unclear. Further explanation of the performance-driving factors would enhance the clarity and reliability of these results.
  • Such as, if we use a non-pretrained transformer to replace the LLM, will the results being same?
• Since the 3D tokenizers are trained on labeled driving datasets, solely relying on them could limit the model's adaptability to novel or unseen scenarios, potentially constraining its utility in unpredictable real-world environments.
• To more comprehensively assess environmental understanding, incorporating question-answering tasks, as demonstrated in datasets like DriveLM[2], would provide a well-rounded evaluation of the model’s interpretive abilities.
• In the original 3D tokenizers, object detection relied only on some tiny MLP layers, while the current setup incorporates a more complex LLM, Vicuna. The application of Vicuna following StreamPETR and TopoMLP slightly reduces 3D perception performance.
• Currently, separate tokenizers are used for object detection and lane recognition. For an autonomous driving system that requires multi-tasking, adding tokenizers for each task could pose challenges. Given that each tokenizer includes an image backbone and maybe transformer encoder/decoder, scaling this approach may affect the model's real-time capabilities.

[1] Zhou, Yunsong, et al. Embodied understanding of driving scenarios. In ECCV 2024. [2] Chonghao, Sima, et al. DriveLM: Driving with Graph Visual Question Answering. In ECCV 2024.

• Using 3D perception as tokens will directly benefit from the perception labels, significantly increase the model size, and thus make it unfair to compare with those VLMs with images as inputs.

• The paper uses StreamPETR as the 3D perception backbone. However, from Table 1 joint training with LLMs significantly harms the perception performance, which indicates the ineffectiveness of the proposed method.

• Table 2 also shows a performance degradation in lane detection.

• Open-loop planning is not sufficient for evaluating LLM's planning capability. The authors need to evaluate their method in CARLA. Although it is not a real-world simulator, it is still sufficient for just evaluating the closed-loop planning capabilities of the methods

• The paper lacks technical novelty as it just includes explicit 3D perception and lane detection as inputs to LLMs.