FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving

We would like to express our sincere appreciation for your perceptive comments and the rigorous review of our manuscript. Your guidance has been instrumental in refining the scientific rigor and clarity of our paper. We have revised the manuscript accordingly and offer our itemized responses to your queries below:

---

Q1: The model structure itself lacks sufficient novelty.

A1: We sincerely apologize for any confusion caused. We would like to clarify that the core innovation of our work is not to propose a new underlying model architecture. As stated by reviewers 4rpa, TPKG and NHPc, our primary contribution is twofold: firstly, we introduce a novel visual reasoning paradigm specifically for autonomous driving; secondly, we propose a unified and efficient pre-training method to activate the visual generation capabilities of existing LLMs. Furthermore, our proposed unified architecture for generation and understanding can be applied to any existing multi-modal large model, demonstrating its broad applicability and scalability.

---

Q2: The claim of using only 0.3% of training data relies solely on experiments conducted on nuScenes. And validating the results on other datasets.

A2:

We thank the reviewer for their valuable feedback and sincerely apologize for the lack of clarity that led to this confusion.

• We would like to clarify a key misunderstanding: our reference to "using only about 0.3% of the data" does not pertain to 0.3% of the nuScenes training set. Instead, this figure is relative to the billion-scale data required by large models that aim to unify visual understanding and generation (e.g., Chameleon, VILA-U). Compared to these models, our approach significantly reduces training costs.
• Furthermore, we have now evaluated the planning performance of FSDrive on the NAVSIM dataset. FSDrive outperforms previous methods, achieving a PDMS of 85.1, demonstrating excellent generalization capabilities.

---

Q3: The auto-regressive formulation in Equation 6 may not satisfy the real-time constraints required in real-world driving systems.

A3:

FSDrive is efficient for real-time tasks. By integrating vLLM and quantization, it achieves a 0.31s decoding latency, surpassing models like DriveVLM and Senna. This performance establishes a solid foundation for real-time applications, poised to improve further with ongoing developments in large model acceleration.

---

Q4: Comparing FID scores across different output resolutions can be misleading. The authors should provide more visual comparisons.

A4: We sincerely thank you for your insightful suggestions. We would like to address your concerns from the following perspectives:

• Our choice of a lower resolution was a deliberate and carefully considered trade-off between inference efficiency and informational sufficiency. Adopting a higher resolution would significantly compromise the model's inference speed.
• More importantly, we empirically observed that increasing the generation resolution did not lead to a corresponding improvement in the final planning accuracy. We attribute this to the fact that the number of visual tokens would vastly outnumber the planning tokens, causing the negative impacts to outweigh the potential benefits. Consequently, metrics like FID and fine-grained resolution details are not our primary focus. As detailed in Section 3.3 of our paper, our core objective is to ensure the physical plausibility of the generated scenes and to guide the model's attention, ultimately to support the downstream planning task.
• It is also worth noting that when compared with DriveDreamer at the same resolution, our method's FID score is substantially better (10.1 vs. 52.6), marking an 81% improvement. This demonstrates that our autoregressive approach remains highly competitive against these diffusion-based methods. We will provide more extensive visual comparisons for a more thorough qualitative analysis in the revised manuscript.
• Furthermore, as advised, we have included richer visual comparisons in the revised manuscript to better support the qualitative evaluation of our method.

---

Q5: Format of Tab6: A more compact and structured format would help improve the overall clarity of the paper.

A5: Thank you for your valuable suggestions. We have reformatted Table 6 for better conciseness.

We are deeply grateful for the time and effort you have dedicated to reviewing our manuscript. Your constructive suggestions are immensely valuable and have provided us with an excellent opportunity to substantially enhance our work. We have carefully considered each of your points and our detailed responses are provided below:

---

Q1: Inference speed for generating future frames is unreported.

A1:

FSDrive is efficient for real-time tasks. By integrating vLLM and quantization, it achieves a 0.31s decoding latency, surpassing models like DriveVLM and Senna. This performance establishes a solid foundation for real-time applications, poised to improve further with ongoing developments in large model acceleration.

---

Q2: Only front-view future frames are generated, while autonomous driving requires 360° environmental awareness.

A2: We sincerely thank the reviewer for their insightful comments and constructive feedback. We offer the following clarifications:

• In line with established previous methods [1,2], our approach generates only the front view, as this view contains the essential information. This design choice allows us to achieve a practical trade-off between inference efficiency and informational sufficiency.
• We would like to clarify that while the proposed method generates future frames exclusively for the front view, its input consists of images from six cameras, encompassing a full 360° perception of the environment, allowing model to leverage information from all other views. Furthermore, we have proactively and explicitly acknowledged this point in the "Limitations and broader impacts" section on page 9 of our paper and have designated its extension as a direction for future work.

---

Q3: Noisy annotations (e.g., occlusions) for lane dividers and 3D boxes.

A3: We directly use the official annotations from the nuScenes dataset. The dataset creators have already addressed ambiguous cases during the collection process and provide reliable ground truth labels even under conditions of occlusion. And when the object is occluded in the front view, since our input is in the surround-view, the model can still obtain the information of the occluded object from other perspectives.

References

[1] Hassan, Mariam et al. “GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control.” CVPR 2025.

[2] Yang, Jiazhi et al. “Generalized Predictive Model for Autonomous Driving.” CVPR 2024.

Thank you for your thoughtful review and constructive feedback. Your comments were invaluable in helping us strengthen the manuscript. Please find our point-by-point responses below:

---

Q1: The improvement in trajectory planning based on LLaVA-7B over other SOTA methods is only marginal. And the improvement achieved on the DriveLM benchmark is still relatively limited.

A1:

• As shown in Table 1 of the paper under the ST-P3 metric, our method without ego status achieves a 29% reduction in L2 error (0.60 vs. 0.84) and an 80% reduction in collision rate (0.19 vs. 0.94) compared to OmniDrive (LLaVA-7B), which are very significant improvements. While it's true that all methods perform well when ego status is used, the ST-P3 metric is not stringent enough to reveal meaningful differences.
• Therefore, for a clearer comparison, we evaluated OmniDrive using UniAD's metrics in the table above. Without ego status, our method achieves a 23% lower L2 error (1.09 vs. 1.42) and a 64% lower collision rate (0.51 vs. 1.40). With ego status, we still see a 14% L2 reduction (0.56 vs. 0.65) and a 67% collision rate reduction (0.21 vs. 0.64). This evidence confirms that our improvements are not "marginal," but rather robust and substantial across different evaluation standards.
• It is important to note that on the DriveLM benchmark, we compare our 2B model against the 7B models from competing methods. In VQA tasks like this, which typically benefit significantly from model size, our smaller model achieves superior performance.

---

Q2: Training with a lower resolution may yield a lower FID score, but this does not necessarily indicate better performance.

A2: We sincerely thank you for your insightful suggestions. We would like to address your concerns from the following perspectives:

• Our choice of a lower resolution was a deliberate and carefully considered trade-off between inference efficiency and informational sufficiency. Adopting a higher resolution would significantly compromise the model's inference speed.
• More importantly, we observed that increasing the generation resolution did not lead to a corresponding improvement in the final planning accuracy. Consequently, metrics like FID and fine-grained resolution details are not our primary focus. As detailed in Section 3.3 of our paper, our core objective is to ensure the physical plausibility of the generated scenes and to guide the model's attention, ultimately to support the downstream planning task.
• It is also worth noting that when compared with DriveDreamer at the same resolution, our method's FID score is substantially better (10.1 vs. 52.6), marking an 81% improvement. This demonstrates that our autoregressive approach remains highly competitive against these diffusion-based methods. We will provide more extensive visual comparisons for a more thorough qualitative analysis in the revised manuscript.

Thank you for your valuable time and insightful comments. Your feedback has been crucial for improving our manuscript. We have addressed your comments point-by-point as follows:

---

Q1: The paper did not provide the metrics of inference latency and real-time performance.

A1:

FSDrive is efficient for real-time tasks. By integrating vLLM and quantization, it achieves a 0.31s decoding latency, surpassing models like DriveVLM and Senna. This performance establishes a solid foundation for real-time applications, poised to improve further with ongoing developments in large model acceleration.

---

Q2: The paper does not provide failure cases of the generated CoT, and there is no dedicated module to handle cases where the CoT contains errors.

A2:

1. On CoT Failure Cases

We thank the reviewer for this insightful suggestion. We will incorporate a failure case analysis in the revision. Our key observations are twofold:

• Our model can achieve correct planning even with minor imperfections in the visual CoT. It extracts the essential gist of the future scene, a finding that corroborates GenHancer [1], which states, "imperfect visual generation can still be beneficial for understanding."
• Due to the data distribution challenges in complex scenarios, while a degraded visual CoT can lead to trajectory deviations, the model's sometimes success under these conditions indicates it has learned a crucial fusion strategy. It effectively balances the generated future (CoT) with current observations, rather than relying solely on the former.

2. On Handling Erroneous CoT

While FSDrive does not include a dedicated module for CoT error correction, its design provides robustness. This resilience stems from two key aspects of our approach:

• Dual-Source Fusion: The final planning policy is not solely dependent on the CoT. It learns to fuse and balance two information streams—current observations and the generated future—thereby mitigating the impact of an failed CoT.
• Progressive Pre-training: Our design proactively reduces CoT errors from the outset. As detailed in Sec. 3.2, the pre-training phase employs a progressive, easy-to-hard settings. This strategy systematically improves the model's predictive accuracy, thus minimizing the generation of erroneous CoTs.
• Future Work: We agree that an explicit safety module is a valuable future direction. For instance, the system could discard a low-quality CoT (e.g., below an FID threshold) and trigger a re-planning mechanism, adding another layer of safety.

---

Q3: The generated CoT is an output of the model's inference, not the fundamental reason for the decision itself.

A3: As argued in [2], a CoT is an essential mechanism for tackling complex tasks, rather than a mere output. To address your concerns, we elaborate on this from the following three aspects:

• Sequential Generation Process: As shown in Equation 6, our model operates under an auto-regressive paradigm. The generation of our CoT and the subsequent trajectory planning is a sequential process. Critically, the prediction of trajectory planning tokens is conditioned on the previously generated CoT tokens, mirroring the standard CoT generation in LLMs[3].
• Quantitative Impact: Our ablation study (Table 5) confirms the CoT's critical role. Its removal significantly degrades performance, increasing the collision rate from 0.40% to 0.58%, which directly proves its impact on decision-making.
• Qualitative Analysis: As shown in Figure 3, when challenged with an incorrect command, our model leverages the CoT to mitigate the faulty instruction and execute a safe maneuver. This demonstrates the CoT's function as a safety-critical reasoning layer, not a passive output.

---

Q4: Provide the results of an ablation study using only the three generation-related tasks without the VQA task.

A4:

We will address your concerns from the following two aspects:

• VQA pre-training was specifically incorporated to boost performance on the DriveLM benchmark, successfully raising the score from 0.55 to 0.57. Notably, our model without VQA pre-training (0.55) still surpasses the OmniDrive (0.53).
• Our ablation study reveals that VQA's impact on planning-centric tasks is modest. Its removal leads to only a minor performance decrease of 0.02-0.03, suggesting it provides a slight enhancement but is not a dominant factor. We maintain that exploring the synergy between understanding and generation remains a vital research direction for unified generation and understanding LLMs.

---

Q5: The evaluation results on other datasets, such as Argoverse 2.

A5:

The Argoverse 2 dataset is not designed for planning tasks; consequently, prior works have generally not evaluated planning performance on it. To bridge this gap, we evaluated FSDrive on the NAVSIM dataset. Our method surpasses existing approaches, achieving a final PDMS score of 85.1, demonstrating excellent generalization capabilities.

---

Q6: Report results specifically for intersections.

A6:

We sincerely thank the reviewer for this insightful suggestion. To clarify, the 'Intersection' metric represents a fine-grained breakdown of the overall 'Collision'. This is the primary reason we did not report it individually in the initial submission. As requested, we now present the results for the Intersection metric. As shown above, our method outperforms OmniDrive by 16% on this metric as well, further demonstrating the robustness of our approach. We will add this result to the revised manuscript.

References

[1] Ma, Shijie et al. “GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers.” ICCV 2025.

[2] Feng, Guhao et al. “Towards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective.” NeurIPS 2023.

[3] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." NeurIPS 2022.