GAP: Scalable Driving with Generative Aided Planner

• Effects of data on the results can be massive. I think the main weakness of the paper is the fact that little is talked about the dataset and different . Over the years, people has been getting better on curating and making good datasets for methods, even the CILRS method has been reported having drastically better results by just increasing input resolution [1] [2]. Changes in the dataset curation can have massive impact. The fact that the paper just mentions that it uses "a dataset of 27 then a dataset of 256 over 8 towns" gives insuficient information. I would like to see how selection of different parts of this dataset really impacts the model. I am not convinced that this method is indeed better than MILE or CIL++ for this application. I would like to be convinced that the tokenization is the way forward but I feel this gives me insufficient information. Something like training a simple CIL++ or a MILE and GPT2 in the same dataset. I know this is a lot of work but is necessary since data plays a very crucial role on this application.

• Limited evaluation specially due to tokenization effects on real world datasets. The fact that this paper only does evaluation on CARLA does not convince me that the tokenization approach scale for real world images. Closed loop test is the main . However, CARLA only tests lacks the variability perceptually to validate a tokenization approach of the image space. CARLA scenes are very repetitive and easily tokenizable, even when considering the towns with more assets. There are kind of closed loop datasets like NuPlan or waymo with the new waymo agents that would allow to test the tokenization approach.

This leads to my third point:

• Lack of comparison with motion prediction literature. Even though the use of this architecture is new on the so called End-to-End driving literature it is not new at all on the motion prediction literature with many papers using GPT like architectures like autobots or scene transformers. UNIAD variation papers [3] also incorporate tokenziation even thought they do use extra information than just the pixels. When compared to those methods the novelty of this architecture is a bit more limited.

[1]Hu, Anthony, et al. "Model-based imitation learning for urban driving." Advances in Neural Information Processing Systems 35 (2022): 20703-20716. [2]Xiao, Yi, et al. "Scaling self-supervised end-to-end driving with multi-view attention learning." arXiv preprint arXiv:2302.03198 2 (2023). [3] Jiang, B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., ... & Wang, X. (2023). Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8340-8350).

I think this paper is not presented soundly and rigorously. Some parts are technically incorrect leading to overclaimed contributions.

1. Current usage of GAP is not autoregressive. The model does not predict the next token based on previous predictions. Additionally, if temporal information is utilized is unclear.
2. The model is not fully self-supervised, because it still needs actions for supervision. If this method can be regarded as self-supervision, then a lot of previous works like Transfuser (the original CVPR 2021 version), CILRS, and more, are all self-supervised. This is absolutely unreasonable. Besides, the authors claim that using 'Expert' is also a kind of extra label. However, MILE uses Roach expert, and this paper follows MILE for data collection. That indicates this paper relies on privileged information for expert-based data collection as well. Another thing is, LBC does not use expert feature distillation and Roach either. These incorrect technique details weaken the claimed contributions.
3. Some descriptions when introducing related works or highlighting the differences of this paper are not accurate or sound. For example, Zheng et al. (2023) Line 148 is not a video-based world model. It predicts occupancy. The OccWorld work also proposes a GPT-based model and validates the scaling law in occupancy prediction. Besides, the authors missed an important related work: CarLLaVA [1]. CarLLaVa also proposed a language model-based transformer to scale up the model. The authors should provide more context and comparison regarding these issues clearly or rigorously. Considering this, I think the novelty of this paper needs further discussion.
4. One important weakness of this paper is GAP's most essential factor is not the next token prediction or other claimed contributions. As shown in Table 2, the driving-oriented feature from CNN improves the performance by 25.6 DS, while the next-token prediction enhances only 4.6 DS. The CNN encoding is not novel and is commonly used in each end-to-end driving paper. The most important driving information comes from the ConvNet, instead of the VQGAN network.
5. I do not fully get the motivation of generative modeling. The predictive feature (next-token prediction) is not fully utilized in the framework, based on the above discussions. How about predicting the future and then decoding the action, or using the VQGAN training as a pre-training stage similar to some occupancy prediction world models, similar to methods in robotics [2]?

This paper does not include the leaderboard evaluation results. That would be considered a much more convincing benchmark.

The writing is relatively casual. The format is incorrect for all citations. The authors should use \citep in most cases. There are limited citations in the introduction to support various claims. I cannot grasp the idea in Fig. 1(c) when comparing it with (a-b). Line 268: Multi-layer Perceptron.

[1] Renz, Katrin, et al. "CarLLaVA: Vision language models for camera-only closed-loop driving." arXiv preprint arXiv:2406.10165 (2024).
[2] Yang, Mengjiao, et al. "Learning interactive real-world simulators." arXiv preprint arXiv:2310.06114 (2023).

1. The paper would benefit from including a comprehensive report on latency to better evaluate real-time feasibility.
2. In Fig.3, it is more informative to compare the scaling effects against other existing methods, rather than just contrasting GPT-small and GPT-medium.
3. Including standard metrics such as FID to evaluate generation quality would be valuable for understanding the overall quality of the generated images.
4. Directly outputting control signals has certain limitations, as control signals can vary significantly between different vehicles, making generalization difficult. It is encouraged to report the model's results when outputting waypoints.
5. Based on Figure 5, an investigation into the impact of temporal sequence length on the performance of the autoregressive model could provide valuable insights.
6. The organization of Table 2 is somewhat strange. It would be clearer to present results for discrete features, discrete features + ground truth traffic lights, and discrete features with driving-oriented features under consistent data conditions.

1. Lack of ablation studies on the importance of VQVAE features. The visual inputs of the proposed methods come from both ConvNet and VQVAE features, but the author only reported the ablation studies of ConvNet features in Sec. 4.4. Therefore the importance of VQVAE features is less obvious. A naive way to demonstrate its importance is to build a baseline model with ConvNet features only and without autoregressive modeling by directly regressing the action. Comparisons to such a baseline model should be included.

2. Lack of analysis of the inference latency. As a larger model size comes at the cost of higher inference latency, the inference speed of both small and medium models should be included.

3. Lack of comparison of scaling properties with other methods. As shown in Table 1, the results of MILE and the proposed method are comparable with 32 hours of training data. However, it's unclear how the MILE performs with a higher volume of data (such as 256h). The authors are encouraged to draw the performance curve of both MILE and GAP for comparison, with different scales of data.