Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving

Response to Reviewer gAQK.

Q1. Do this explicit intervention may diminish the strength of the work in comparison to purely end-to-end imitation learning approaches?

A1. Thank you for your insightful feedback. While our PGS framework introduces auxiliary losses derived from perception outputs—such as lane geometry or predicted trajectories—it remains a fully trainable end‑to‑end model. All perception, planning, and policy modules are jointly optimized without introducing hard-coded or frozen components.

This approach offers a principled way to improve safety and reduce imitation noise by guiding the network to focus on causal features (e.g., lane following and collision avoidance cues), without sacrificing flexibility or learning capacity.

By contrast, purely end-to-end imitation often encounters causal confusion, where models learn spurious correlations rather than underlying causal mechanisms—especially under distributional shift. As empirically documented, more observational inputs can indeed degrade performance if the model incorrectly infers cause from effect. The auxiliary supervision in PGS can help mitigate such confusion by nudging the model toward robust, causally relevant patterns during training.

Q2. Adding experiments in environments built on real-world datasets such as nuScenes will be ideally.

A2. Thank you for this valuable recommendation. We fully acknowledge the significance of evaluating autonomous driving methods on real-world datasets. Indeed, nuScenes is a highly influential benchmark with rich annotations that has notably advanced research in perception and prediction. However, for assessing the performance of end-to-end driving systems, nuScenes exhibits several key limitations:

Firstly, nuScenes provides primarily open-loop data. Recent studies (e.g., Reference [19]) have explicitly demonstrated that open-loop performance often fails to reflect realistic closed-loop capabilities accurately. Remarkably, simple baseline models—such as a single-layer MLP conditioned only on the ego vehicle’s current state—can achieve superficially strong open-loop performance metrics on nuScenes, underscoring this limitation.

Secondly, nuScenes predominantly comprises simple, straight-driving scenarios, which constrains its capacity to probe causal reasoning and to evaluate generalization in rare yet safety-critical situations.

In contrast, our work focuses explicitly on improving closed-loop performance in interactive and high-risk settings. Bench2Drive encompasses a wide range of driving scenarios and offers relatively comprehensive evaluation metrics, which constitutes the primary reason for selecting it as our main evaluation benchmark.

Lastly, we sincerely hope to further validate and extend our method on benchmarks that combine real-world data with closed-loop evaluation.As part of our future work, we plan to explore real-world evaluation using Bench2Drive-R, which provides real-world driving data with richer map annotations and supports closed-loop benchmarking. We believe this will offer a more principled and reliable framework for evaluating decision-making performance in the real world.

Q3. Could PGS also be leveraged for the branches that output object trajectories and mode-classification scores?

A3. Thank you for your valuable suggestion. The proposed PGS framework can indeed be extended to the mode-classification branch, as it shares similar decision-level semantics with ego planning.

However, applying PGS directly to object trajectory prediction is less straightforward. In ego planning, the future intention (e.g., turning left or right in form of target ponint) is clearly defined and remains applicable during both training and deployment phases. In contrast, for surrounding objects, such intent information is typically unavailable or ambiguous, especially at intersections. To effectively apply PGS in this context, additional intent estimation or behavior modeling would be required. We consider this a promising direction for future work. We believe exploring such extensions could further enhance the model’s holistic scene understanding and planning reliability.

Q4. What does $\beta$ represent in line 240?

A4. $\beta$ represents the threshold distance used to determine potential collisions. In our model, we set $\beta$ = 0.5 meters to define a safe margin between agents.

Q5. What accounts for the imbalance of high driving-score and success-rate and lower scores for efficiency and comfort?

A5. Thank you for your thoughtful question. Please refer to our response to Reviewer qCVeQ, Q2 for a detailed explanation.

Q6. Are “PPGSS” and “PGS” in Tables 1 and 2 the same?

A6. We really appreciate your attention to detail. Yes, "PPGSS" is a typo and will be corrected to "PGS" in the revised version.

Q7. Why there is a performance gap between VAD-Base and PGS_Base?

A7. Thank you for your thoughtful question. As stated in line 154 of the main paper, the proposed $PGS_{base}$ model does not adopt the cascaded prediction–decision architecture used in VAD. Instead, it employs a unified decoder for both ego trajectory planning and object motion prediction. This architectural difference inherently introduces changes in representation and decision coupling.

Furthermore, we would like to offer an important clarification. Our model does not incorporate the auxiliary planning losses used in the VAD framework. The original intent was to adopt the full loss suite from VAD’s perception module only. We sincerely apologize for the confusion and will revise the manuscript to clearly state this distinction.

Response to Reviewer RHTg.

Q1. Does the system performance collapse if the perception system fails to detect some vehicles?

A1. Thank you for your valuable question. This is indeed a critical consideration, especially in the context of real-world deployment. A similar concern was also raised by Reviewer M41J, Q1, to which we have provided a detailed response. We kindly refer you to that reply for a comprehensive discussion of this issue.

Q2. Did the authors try to implement and test PGS in any real-world datasets/simulations such as DriveArena, NavSim?

A2. Thank you for your valuable suggestion. We have carefully considered your comment. DriveArena is currently primarily used for data generation and reinforcement learning in closed-loop training settings. In contrast, our method is specifically developed for training on offline datasets—collected from real vehicles—to enhance closed-loop performance via novel algorithmic and architectural improvements. Indeed, training on off-line datasets remains the predominant paradigm for deploying end-to-end autonomous driving models both in academia and industry, particularly due to the practical constraints associated with online training. Furthermore, DriveArena currently lacks standardized evaluation protocols and publicly available benchmarks, which makes it difficult to adopt at this stage. Nonetheless, we fully agree that closed-loop training represents a promising future research direction, and we intend to explore this in subsequent extensions of our work. Regarding experiments on the NavSim, please refer to our response to Reviewer M41J, Q5.

Response to Reviewer M41J.

Q1. What happens if the perception module fails catastrophically?

A1. We appreciate your thoughtful comment. We fully recognize that our framework depends on perception outputs, which naturally raises questions about robustness under perception errors. We would like to clarify our motivation, underlying assumptions, and the scope of our contributions in relation to this concern.

The design of our paradigm is motivated by a key empirical insight: in existing end-to-end driving systems, perception is rarely the dominant performance bottleneck. In the majority of scenarios, modern perception modules already provide sufficiently accurate outputs for downstream tasks. Despite this, end-to-end systems often suffer from causal confusion in their decision-making process—where models latch onto spurious correlations rather than semantically meaningful or causally relevant signals.

Based on this observation, the primary objective of our work is to improve causal modeling within the planning and control module and to verify its impact on the performance of end-to-end systems. And we believe that the proposed improvements can enhance ego vehicle behavior in most scenarios where perception is effective.

We acknowledge that, like all perception-dependent systems, our method inherits the limitations of upstream perception. In rare or long-tail scenarios where perception fails, performance degradation is expected. However, this sensitivity is not unique to our method; rather, it is an intrinsic property of any perception-dependent autonomous driving approach—whether end-to-end or modular. Errors originating from the perception module naturally propagate downstream, cumulatively affecting planning and control decisions.

Crucially, our framework neither introduces additional vulnerabilities nor exacerbates existing issues beyond those inherently present in conventional perception-driven approaches. In fact, enhancements in perception robustness directly translate into improved performance for our approach. Therefore, we see our contribution as complementary and orthogonal to existing perception improvement efforts.

In summary, while acknowledging the inherent sensitivity to perception quality, we emphasize that our method primarily targets the critical issue of causal confusion—demonstrably improving system robustness under standard perception conditions. Improving perception robustness itself remains a broader and longstanding goal in the autonomous driving research community, and advancements in this area would directly benefit the effectiveness of our proposed framework.

Q2. The usage of the term “self-supervision” may be inconsistent with its conventional definition, as the supervision in this work comes from perception outputs trained with labels.

A2. Thank you for raising this insightful question. We had indeed discussed this point extensively prior to the submission, as your description precisely captures. We ultimately chose to use the term self-supervision because the main contributions of our work are organized around a novel learning paradigm for the planning and control module. This terminology was intended to emphasize the conceptual distinction from existing imitation learning paradigms. As a result, we finalized the term perception guided self-supervision after careful deliberation.

This term is meant to clarify the differences between our approach and standard definitions of self-supervision: specifically, our supervisory signals are derived from an on-the-shelf perception module. We will revise the manuscript to clarify the use of this terminology and more explicitly articulate its intended meaning, in order to avoid potential confusion.

Q3. Will the code be released?

A3. Yes, we fully agree that open-sourcing the implementation would benefit the community by facilitating reproducibility and further research. We will release the code upon acceptance of the paper.

Q4. How does the method perform when perception results are inaccurate or noisy?

Thank you for your question. This concern closely aligns with the point raised in Weakness 1 of your comment. For a detailed response, please refer to our answer to Q1, where we provide a comprehensive discussion addressing this issue.

Q5. Have the authors considered evaluating the method on real-world datasets such as NAVSIM?

A5. Thank you for your valuable suggestion.First, existing studies have demonstrated that although methods evaluated under open-loop settings may yield promising L2 error and collision rates, such models often exhibit significant performance degradation when deployed in real-world scenarios or evaluated under closed-loop settings. Therefore, our work focuses primarily on enhancing end-to-end model performance under closed-loop evaluation, which we believe holds greater practical value for autonomous driving systems.

For this reason, we chose Bench2Drive (B2D) as our primary benchmark. On one hand, this dataset encompasses a wide range of complex traffic scenarios, including 44 interactive categories such as ParkingExit, ParkingCutIn, ParkedObstacleTwoWays, and HazardAtSideLane, along with 23 different weather conditions (e.g., sunny, foggy, rainy), effectively covering the majority of real-world driving contexts. The evaluation set ensures comprehensive coverage across all categories. On the other hand, the evaluation is conducted in a fully closed-loop manner—meaning the behavior of surrounding agents dynamically responds to the ego vehicle’s actions—thereby better aligning with the dynamics of real-world deployment.

In contrast, the NAVSIM dataset is derived from nuPlan, which provides approximately 1,200 hours of human driving data. However, it does not directly include adverse conditions such as heavy rain or nighttime driving.  Furthermore, its evaluation is semi-closed-loop—or arguably still open-loop—as the behavior of other agents does not react to the ego vehicle's decisions. This makes the evaluation metrics insufficient to fully reflect the algorithm’s performance in real-world closed-loop conditions.

Second, the core motivation of our work is to address the causal confusion issues that frequently arise in closed-loop or real-world testing. A classic example of such confusion is observed in how autonomous vehicles behave at traffic light intersections: models suffering from causal confusion fail to learn the correct association between traffic light signals (red/green) and the ego vehicle's actions, instead basing their decisions on the behavior of surrounding vehicles. This represents one of the most critical and realistic forms of causal confusion, which is not captured in NAVSIM's evaluation metrics. NAVSIM focuses solely on metrics such as collision rate, drivable area violations, efficiency, and target goal achievement, without assessing whether the model correctly understands the causal relationship between traffic signals and ego behavior at traffic lights or stop lines. In contrast, the B2D benchmark explicitly reflects this type of causal failure: confusion regarding traffic light cues can lead to vehicles becoming stuck or violating traffic regulations, ultimately resulting in task failure. Furthermore, B2D includes a dedicated set of scenarios involving interactions with traffic signals and stop signs (i.e., the “traffic sign” scenario group), which allows for a more targeted and comprehensive evaluation of a model’s causal reasoning capabilities.

Thus, although B2D may have a larger domain gap from real-world data in terms of visual appearance and sensor modality, we argue that from an algorithmic perspective, B2D and its evaluation protocol more accurately reflect real-world performance.

In summary, due to the comprehensiveness of B2D's scenario coverage, its alignment with real-world deployment dynamics, and its ability to reflect the most challenging causal reasoning problems in end-to-end driving, we selected B2D as a more suitable and challenging benchmark. In contrast, NAVSIM may not be well-suited for addressing the specific issues our work targets.

Lastly, we sincerely hope to further validate and extend our method on benchmarks that combine real-world data with closed-loop evaluation. We have assessed the feasibility of adapting our method to NAVSIM, and found it would involve substantial effort, including dataset format conversion, constructing intermediate supervision signals (e.g., verifying whether NAVSIM includes lane-level annotations such as lane centerlines), validating perception-guided self-supervised signals (e.g., tuning thresholds for relevant lane filtering modules, and visual verification of MTPS, STPS, and NTPS losses). Given the limited time available during the rebuttal phase, it is technically infeasible to robustly adapt our method to NAVSIM within this timeframe.

Luckily, we have noted that the B2D team announced in December 2024 the upcoming release of an extended version—Bench2Drive-R—which will include both real-world and simulation data. This extended benchmark will better fulfill both data-level and evaluation-level alignment with real-world conditions. We plan to validate our method on this dataset as soon as it becomes publicly available.

Q6. Should "Limitation learning" be "Imitation learning" in Figure 1?

A6. Thank you for your careful review. Yes, this is a written error and should be corrected to “Imitation learning.” We will fix it in the revised version.

Response to Reviewer qCVe.

Q1. For unstructured cases, show how PGS with only NTPS works.

A1. This is a reasonable suggestion, and thank you for pointing out our oversight. We have conducted additional experiments specifically addressing your concern by training a variant of our model (denoted as $PGS_{NTPS}$) using only the NTPS loss, thus entirely removing dependencies on structured map elements. As summarized in Table 1, compared to the complete PGS model, the NTPS-only variant demonstrates significantly improved performance in merging scenarios. This result can be attributed to NTPS’s strong guidance in collision-prone, complex intersection scenarios. Conversely, performance slightly decreased in overtaking scenarios, as NTPS may lead to overly conservative behavior when static obstacles are present in the ego lane. Upon further incorporating MTPS and STPS losses, structured road priors provide richer spatial guidance, effectively enlarging the decision space. This allows the model to learn more flexible maneuver patterns, such as overtaking by borrowing adjacent or opposing lanes. Consequently, the overall success rates improve in both merging and overtaking scenarios. Both quantitative and qualitative analyses confirm our original hypothesis: incorporating structured road priors significantly strengthens causal reasoning, enhancing decision robustness under structured driving conditions. In unstructured environments where road priors are unavailable, we expect the full PGS model to behave similarly to the NTPS-only variant.

Table 1: Ablation Study of the Proposed PGS Framework with NTPS-only PGS Model.

Q2. Why the introduced losses may cause the efficiency and comforness degrade significantly in Table 1? Shouldn't the inference cost of the proposed method the same as the VAD-base?

A2. Thank you very much for this thoughtful comment. To clarify, the efficiency metric is computed based on comparisons between the ego vehicle’s speed and that of surrounding traffic, rather than the inference cost. This terminology might have caused confusion. Since our method shares similar architectural complexity with VAD, it does not introduce additional computational overhead during inference. The comfortness metric, on the other hand, is derived from acceleration and jerk measurements.

First of all, prompted by your constructive question, we identified a critical issue in our evaluation process. Specifically, we inadvertently used an outdated version of the official atomic_criteria.py file (located at Bench2Drive/scenario_runner/srunner/scenariomanager/scenarioatomics/atomic_criteria.py). This earlier version incorrectly computed the efficiency metric using only cases with minimum-speed infractions (i.e., speed < 100%), while entirely omitting scenarios labeled as "Perfect" (i.e., completed without speed violations). This issue is confirmed by the JSON files included in our supplementary materials (b.PGS_all_metric_files).

The latest version of the official script, which we have now adopted, correctly follows the evaluation protocol described in the original paper by recording speed measurements at every 5% interval along the route to ensure comprehensive and fair metric computation.

Under the corrected protocol, our updated Efficiency and Comfortness scores are 180.9974 and 12.0039, respectively. Notably, our model achieves a new state-of-the-art result in Efficiency. These values will be updated in the final version of the paper.

However, we would like to emphasize that Efficiency and Comfortness should be interpreted under comparable scenario success rates to ensure fair and meaningful comparisons. For example, if a vehicle becomes stuck at an intersection and fails to complete the scenario, the computed Comfortness may be deceptively high due to the lack of motion. Conversely, a model that completes the scenario at high speed—but with a collision—may report high Efficiency despite safety violations. Therefore, we argue that Driving Score and Success Rate are more indicative of real-world performance, and comparisons of secondary metrics such as Efficiency or Comfortness should be contextualized under similar success conditions.

Secondly, in challenging driving scenarios such as overtaking by temporarily borrowing the opposing lane, our model prioritizes safety, often resulting in conservative maneuvers. Specifically, the model tends to decelerate or come to a full stop before accelerating rapidly to complete the maneuver safely. While this behavior closely resembles cautious human driving, it inevitably leads to increased acceleration and jerk, thereby negatively impacting the comfort score.

Moreover, our method leverages lane-centerline-based pseudo trajectories as supervision signals. These pseudo trajectories effectively guide the model to adhere more accurately to road topology at intersections, thereby reducing erroneous lane choices. However, because these pseudo labels prioritize spatial correctness over temporal continuity, the resulting trajectories may exhibit slight discontinuities compared to human expert demonstrations, which further degrades the comfort metric. Addressing this temporal discontinuity to improve trajectory smoothness is a key part of our future work.

Q3. Issues with notation clarity and paper formatting.

A3. We appreciate your careful review. We will revise the manuscript accordingly to improve the clarity of formatting, as per your suggestion. Specifically, the "⊕" sign indicates concatenation between features, The term “Limitation learning” in Figure 1 is a written error and will be corrected to “Imitation learning.”

Q4. Will the quantisation of STPS cause some discontinuity in the target trajectory? How does that affect the performance?

A4. Your observation is correct. Quantizing spatial trajectories in STPS inevitably introduces minor discontinuities compared to smooth human expert trajectories. These discontinuities manifest as subtle velocity fluctuations, which contribute to reduced comfort. Addressing this concern, we plan to implement spatial smoothing techniques in future iterations to alleviate trajectory discontinuities and thus enhance comfort metrics.