CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We have carefully addressed your concerns in detail. We hope you might find the response satisfactory.

Weakness 1:

Thank you for pointing out this issue. We have revised the manuscript and corrected the citation for FusionAD accordingly.

Weakness 2:

We apologize for the unclear expression. The temporal instruction is primarily composed of three parts: 1.A fixed prompt: “In the previous frame,” 2.The main threat detected in the last frame, such as “a white car ahead is moving away from the ego vehicle,” or “a car ahead is slowing down.” If no threat is detected, the default is “there is no safety threat.” 3.The action taken by the ego vehicle, such as “the vehicle continues in its current driving state,” “the vehicle is turning left,” “the vehicle is turning right,” or “the vehicle is slowing down.” This temporal instruction clearly conveys the ego vehicle’s previous action, which is crucial for determining the next appropriate action, as prior actions often influence current decision-making.

Weakness 3:

We have revised Section 3.4 of the manuscript to improve its readability. The specific concerns have been addressed in detail in the question part (please refer to Q1, Q2, Q3, and Q4). We hope that our responses to Q1, Q2, Q3, and Q4 could address your concerns.

Weakness 4:

We appreciate the reviewer’s insightful suggestion. In response, we have added a detailed explanation of the datasets used for pre-training both the student and teacher VLMs. Section 3.5.1 of the manuscript has been revised accordingly. The updated content is as follows:

During pre-training, the teacher and student VLMs are trained separately using simulated and real-world datasets, respectively. The simulated dataset, referred to as CARLA-VLM, is collected using the CARLA Leaderboard v2 simulator and comprises 61.6% normal scenes and 38.4% challenging scenarios (e.g., traffic jams, near-miss vehicle interactions, and pedestrian intrusions). The real-world dataset used is prepared from the publicly available nuScenes dataset. To accelerate training, we initialize both models with checkpoints from LLaVA-v1.5 [40]. Each model is fine-tuned using image frames, temporal instructions, and end-to-end prompts, enabling the Chain-of-Causality (CoC) VLMs to align instruction, visual, and temporal information effectively. Efficient training is further facilitated by sampling frames at fixed intervals and applying random temporal shifts as augmentation.

We use a real-world dataset (nuScenes-VLM) and a simulated dataset (CARLA-VLM) to pre-train the student and teacher VLM models, respectively. nuScenes-VLM dataset: The student VLM is pre-trained on the nuScenes-VLM dataset, which is derived from the publicly available nuScenes dataset and enriched with textual prompts (as illustrated in Appendix B.2). We follow the official train/validation split provided by nuScenes.

CARLA-VLM dataset: We use this dataset as our simulation dataset to pre-train teacher VLM model. This dataset is configured with settings similar to nuScenes and annotated with textual prompts consistent with those in the nuScenes-VLM dataset. To enhance dataset transparency, we provide the statistical distribution of scenarios within the CARLA-VLM dataset. A new supplementary table presents detailed counts for each scenario category. We believe these additions improve the clarity of our dataset description and enable a more rigorous evaluation of the model’s performance, particularly in challenging driving scenarios.

Actually, In all our experiments, we only used two datasets: the nuScenes-VLM dataset (real-world dataset) and the CARLA-VLM dataset (simulation dataset).

Weakness 5:

Thank you for your suggestion. We have revised the manuscript to enhance transparency. In our anonymous submission, our original intent was to explain the contribution of the back-propagation strategy in Section 3.5.2, with its corresponding data flow illustrated in Figure 1. We have now revised and clarified this content in Section 3.5.2 as follows:

3.5.3 Adversarial Training with Multi-Step Back-Propagation

As illustrated in Fig.1, the adversarial training process involves multiple steps that incorporate our unique back-propagation strategy. Initially, the pre-trained teacher and student VLM models are loaded. The training procedure then proceeds in two steps:

Step 1:

Forward-Propagation

In this forward propagation, we freeze the teacher VLM model and student VLM model, only train our discriminator. Simulation data and real-world data are fed separately into the teacher and student VLM models. The features extracted from both CoC VLMs are then passed into the discriminator. Back-Propagation

After the discriminator outputs the final logits, we compute the discriminator loss and perform back-propagation. Since only the discriminator is trainable in this step, the gradients are back-propagated solely through the discriminator, and only its parameters are updated. The parameters of the teacher and student VLM models remain unchanged.

Step 2:

Forward-Propagation

In this process, only the student VLM model and the discriminator are involved in the training pipeline. Both simulation and real-world data are fed into the student VLM model, and the resulting features are passed through the discriminator to produce logits. Back-Propagation

This back-propagate step introduces our key contribution in terms of back-propagation. The discriminator loss is back-propagated through the discriminator, but without updating its parameters. Then, the gradients from this discriminator, together with the VLM-specific losses, are used to update only the parameters of the student VLM model.

Q1:

In line 187, our intention was to convey that temporal information from the previous frame is aggregated into the current frame to enhance temporal features. The term “temporal consistency” was used to indicate the continuity of information across the temporal dimension. To achieve this, we have employed a rule-based filter, this filter removes the reasoning part of End-to-end Answer, extract the final ego vehicle's action instruction such as "There is no safety threat, the vehicle maintains its current speed.", as described in lines 230. This filtering step reduces the input token length while preserving the most critical information from the previous frame.

Q2:

Dear Reviewer, we have carefully addressed your concerns as follows.

$X^{S}$ and $X^{R}$ are macro-level representations denoting the data distributions of the simulated and real-world autonomous driving domains, respectively. A domain consists of an infinite set of data. Human only sample a finite subset from these distributions to form the training and testing datasets. In our paper, the datasets represent small sample sets drawn from both the virtual and real-world domains. Additionally, we use $y$ to represent the labels/ground truth..

Q3:

Here is our explanation, our labels/GT corresponding to samples data $x^{s}$ and $x^{r}$ are mainly generated by GPT-4 and subsequently verified manually. These GTs, such as "There are two pedestrians ... In front of the ego vehicle..." are distribution samples from GPT4 outputs. Since natural language allows for diverse expressions of the same underlying information, it can have plenty of expression to describe the GT, for example "A pedestrian is on the left ...., another pedestrian is 10m away from ego...". To formalize this process, we use the statistical learning theory (also called VC-theory) to express general mathematical framework for estimating dependencies from samples. Different from most empirical methodologies (like deep learning) that often rely on intuitive, asymptotic, and/or biological arguments. We use an expression that returns a label $y$ for every input $x$, according to a conditional distribution $p(y|x)$, this is determined by GPT4 and manual verification. The expectation is: $\bar y = \int dp(y|x))$ In our statistical field, we normally select enough samples to mitigate the bias. But in deep learning, like the existing LLM/VLM methods, they use few samples as the final expression. Thus, we explicitly highlight the distribution expression of language content generation.

The distribution $p^s$ and $p^r$ are determined by GPT4 and manual verification.

Q4:

In our paper, the f-divergence is employed to measure the discrepancy between two domains. To enable optimization, the f-divergence function must be convex. Therefore, we adopt a convex function $\delta$ as our f-divergence.

Thank you for the great efforts and valuable comments. We have carefully addressed the main concerns in detail. We hope you might find the response satisfactory.

Weakness 1

Response: Thank you for pointing this out. Specifically, we have revised Figure 1 by merging the flow lines to make the illustration more concise and clear. Additionally, we have moved the complex forward and backward propagation expression to the appendix (now shown as Figure 4 in our manuscript) and provided the corresponding explanations below: Before adversarial training, we load the pre-trained models for both the teacher and student VLM models. The adversarial training process consists of multiple steps, incorporating our unique back-propagation strategies.

Step 1:

Forward-Propagation: We freeze both the teacher and student VLM models and train only the discriminator. Simulation and real-world data are fed into the two VLMs, respectively, and the resulting features are passed to the discriminator to outputs the logits. Backward-Propagation: Using the discriminator's output logits, we compute the loss and perform back-propagation. Since only the discriminator is being trained at this stage, we update only its parameters while keeping both VLM models frozen.

Step 2:

Forward-Propagation: In this process, only the student VLM and the discriminator are involved. Both real-world and simulation data are input into the student VLM, and the resulting features are passed to the discriminator to produce logits.

Backward-Propagation: This step introduces our novel contribution. We first back-propagate the discriminator loss through the discriminator without updating its parameters. Then, the propagated gradients, along with the VLM-specific losses, are used to update the student VLM model. This strategy reduces convergence instability and accelerates the adversarial training process.

Weakness 2

Response: We have removed the excessive blank space in the Introduction section, particularly before and after Figure 1.

Weakness 3

We have condensed the Sections 3.2 and 3.3 into the first paragraph of Sections 3. The modified content is shown below:

As illustrated in Fig. 1, the proposed architecture consists of three components: a Teacher Visual Language Model (Teacher VLM), a Student Visual Language Model (Student VLM), and a Visual Language Model Discriminator. Both the Teacher and Student VLMs share a common base architecture, referred to as the Chain-of-Causality Visual Language Model (CoC VLM), which processes multi-view image pairs, end-to-end prompts, and historical instructions from previous frames to generate end-to-end outputs. The Teacher VLM is trained on simulated data to address diverse and rare scenarios, such as pedestrian trespassing, driving violations, and traffic accidents. In contrast, the Student VLM is trained on real-world data and serves as the final inference model, enabling the transfer of knowledge from simulated to real-world contexts.

Additionally, we have relocated Subsection 3.5, Model Training, to the beginning of Section 4 as its first subsection. Following this reorganization, the overall structure is significantly clearer and more readable.

Weakness 4

We apologize for the oversight. We have revised the formatting inconsistencies in the tables by adding separate columns to Tables 2, 3, and 4, consistent with the format used in Table 1.

Q1:

Thank you for your suggestion. We agree that "CoC answer" is more appropriate than "End2End answer." Accordingly, we have revised the manuscript to replace all instances of “End2End answer” with “CoC answer.”

Q2:

We have conducted an experiment using two cached frames, with the results presented in Table 6. The findings indicate that, compared to using one cached frame, there is no significant improvement in ADE or Collision Rate. Therefore, we have chose to use a single cached frame to reduce the number of input tokens. Table 6. Ablation study on N frame temporal information.

Q3：

Existing Chain-of-Thought (CoT) autonomous driving methods typically rely on multiple rounds of dialogue, with each round performing a single-step reasoning process. For example, OmniDrive employs a sequence of prompts such as:

1. What traffic elements should I be aware of while driving in this area?
2. If I decide to accelerate and make a left turn, what could be the consequences?
3. What should be my next action given the current driving situation, and why?

These Chain-of-Thought approaches often compromise the integrity of reasoning, resulting in fragmented and incoherent decision-making chains. In contrast, our Chain-of-Causality (CoC) method outputs entire CoC answer (End2End answer) that preserves internal causal linkages across the entire process. Specifically, it follows the sequence: T cached information → perception → prediction → planning → T+1 cached information. For example, if the ego vehicle was turning left in the previous frame, the model should detect an oncoming vehicle in the opposite lane, predict its trajectory, and generate an appropriate plan accordingly.

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. Your insightful comments have significantly improved the quality of our research. Below, we address your concerns point by point.

Weaknesses 1:

Response: Thank you for your thoughtful suggestion. We conducted a more in-depth analysis to demonstrate our model’s ability of handle uncommon cases. Specifically, we split the challenging subset of the nuScenes dataset (comprising 122 scenes) as our test set, while the remaining, easier cases were used for training. Here are challenging scene examples: Scene-0026_379: The ego vehicle is intercepted by a construction worker to give way to a construction truck approaching from the left. Scene-0046_568, Scene-0094_948, Scene-0131_1153, Scene-0162_1556: Pedestrian intrusions. Scene-0150_1358: A construction worker blocks the road using traffic cones or water barriers in front of the ego vehicle. Scene-0201_1978: The ego vehicle is obstructed by a car attempting to park. We trained and evaluated our model using this challenging subset. The performance results are presented below.

Furthermore, to analyze the performance on easier cases (e.g., straight-line driving), we designed a dummy regressor baseline that always predicts a straight trajectory. This baseline helps quantify how much of the evaluation performance on the new nuScenes split can be attributed to genuine methodological improvements. The dummy regressor, termed the Straight Regressor Baseline, is based on our method. To generate a straight trajectory, we ignore the y-axis and predict only the x-axis trajectory values (whereas existing methods predict both (x, y) coordinates). We trained this Straight Regressor Baseline, and the results are presented below.

Weaknesses 2:

Response: We sincerely thank you for your suggestion. We employed NeuroNCAP [1] as the closed-loop simulator, as it supports the nuScenes dataset and provides pretrained rendering model checkpoints, making it well-suited for our method. We cloned the NeuroNCAP [1] and NeoRAD-Studio [2] repositories, replaced the example model (UniAD) with our own pretrained model from the main experiment, and integrated our checkpoints. Additionally, we downloaded the NeuroAD weights, modified the evaluation script accordingly, and conducted the closed-loop evaluation. Since NeuroNCAP offers a standardized benchmark and evaluation metrics, and is commonly used by other methods, we followed its recommended configuration. We evaluated our method on the suggested scenarios (e.g., scene-099, scene-0103, etc.). As shown in the table below, our proposed method outperforms UniAD, VAD, and our student baseline trained on the nuScenes dataset without adversarial transfer learning.

Weaknesses 3:

Response: We have provided the statistical distribution of the simulated challenging scenarios to clearly characterize the dataset. A new supplementary table presents detailed counts for each scenario category. We hope these additions enhance the transparency of our dataset and enable a more rigorous assessment of the model's performance in handling rare but critical driving events.

Q1:

We apologize for the unclear explanation. The filter referenced here is a rule-based system that removes the reasoning component of the End-to-End Answer and extracts only the final action instruction for the ego vehicle—for example, "There is no safety threat, the vehicle maintains its current speed." as described on line 230. This filtering process reduces the input token size while preserving the most critical information from the previous frame.

Q2:

The "dedicated end-to-end answer generation module" refers to our CoC VLM model. Unlike existing approaches that typically rely on multiple rounds of dialogue (they firstly generating a perception answer, then feeding it into the model to produce a prediction answer, and finally combining both to generate a planning answer), our model generates the end-to-end answer directly. This design simplifies the pipeline and streamlines output generation.

[1] W. Ljungbergh, A. Tonderski, J. Johnander, H. Caesar, K. Åström, M. Felsberg, and C. Petersson, “Neuroncap: Photorealistic closed-loop safety testing for autonomous driving,” European Conference on Computer Vision (ECCV), 2024. [2] A. Tonderski, C. Lindström, G. Hess, W. Ljungbergh, L. Svensson, and C. Petersson, “NeuRAD: Neural rendering for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 895–14 904.598

Dear Reviewer, We truly appreciate your time and effort in reviewing our manuscripts and offering your valuable feedback. We thank you for allowing us to revise, which greatly moved us. We will carefully revise our manuscript.

Once again, thank you for raising score.

Wish you all the best.

Warm regards

Authors