Diffusion Guided Adversarial State Perturbations in Reinforcement Learning

Dear Reviewer 3Yfq:

Thanks for your insightful feedback and we will address your concerns.

Q1a: Semantic change is not considered as stealthy. What is the exact definition of stealthiness?

A1a: In the state perturbation threat model, the agent only observes perturbed states and has no access to the true states, unlike in traditional vision settings. Therefore, semantic changes can be considered stealthy if they remain physically realistic and plausible, as defined in Definition 3.2.

Our definition of stealthiness is grounded in three apsectives: Realism (Def. 3.2), Historical Alignment (Def. 3.4), and Trajectory Faithfulness (Def. 3.5). Due to the computational intractability of projection onto the valid states, we use approximate metrics to evaluate stealthiness: autoencoder reconstruction error (for realism), Wasserstein distance (for historical alignment), and SSIM/LPIPS (for trajectory faithfulness).

Table 2 provides quantitative evidence: SHIFT achieves strong attack performance while maintaining stealth, as indicated by lower reconstruction error and Wasserstein distance comparable to low-budget baselines. Additional detection results in Appendix D and preliminary DIRE outcomes (see Q2) reinforce this finding. Although SSIM and LPIPS are not directly usable for detection—since they require access to the true states—they help assess how trajectory faithful the perturbations are.

In summary, SHIFT surpasses all other baselines in attack performance and has comparable stealthiness with low-budget baselines, justifying our claim in lines 391–392.

Q1b: Unfair comparison for reconstruction error.

A1b: The unfair comparison concern is addressed by our ablation study (Figure 6a): SHIFT-O without autoencoder-based realism enhancement achieves reconstruction error of 1.06±0.5 in Freeway and is still lower than all baseline attacks (In Table 2). This occurs because our conditional diffusion model is trained upon clean trajectories and can generate realistic states. The conditional diffusion model ensures physical plausibility, making SHIFT's realism advantage intrinsic rather than evaluation-biased.

Q2: Diffusion generated sample detector.

A2: We conducted a preliminary evaluation of the DIRE detector. Given an input image $x$, DIRE first performs a reverse diffusion sampling process to add noise until reaching the standard normal prior $\mathcal{N}(0, 1)$, then generates a new image $x’$ via the diffusion sampling process. The DIRE distance is defined as the $l_2$ distance between $x$ and $x’$. Since diffusion-generated images tend to remain closer to the diffusion sampling distribution and the real image distribution is different from the diffusion sampling distribution, DIRE assumes that real images will have higher distances than samples synthesized by a diffusion model. We tested this on the Pong environment. Non-attack diffusion samples yield low DIRE distances (0.05 ± 0.15), while true states show higher distances (0.61 ± 0.77), consistent with DIRE's assumptions. However, SHIFT-O, which introduces policy guidance during sampling, shifts the distribution further, achieving an even higher distance (0.94 ± 0.60). Although the higher mean might seem distinguishable, the high variance prevents reliable single-frame detection.

As an example, here are five consecutive samples of DIRE distances (true state, SHIFT-O) from our experiment: [(0.59,0.63), (0.77,0.51), (0.72,1.23), (0.60,0.44), (0.60,0.48)]. Despite SHIFT-O having a higher average, 39% of true states throughout an episode with 2000 steps have a higher DIRE distance, making instance-level detection unreliable. Although it is still possible to use the mean DIRE distance to detect the existence of attacks, this requires aggregating statistics over a trajectory. By the time the attack is detected, it may have already influenced the victim’s decision-making and degraded its performance.

These findings suggest that DIRE is not effective for detecting SHIFT attacks. More broadly, detection methods trained solely on clean data may struggle to identify adversarial samples since adversarial attacks may distort the distribution in an unexpected way.

Q3a: Why was SA-MDP’s considered perturbation set not adapted to use, e.g., the reconstruction error from the auto-encoder, or some other notion better aligned with SHIFT?

A3a: SA-MDP's $l_p$ norm constraint enforces spatial proximity but cannot capture SHIFT's multi-dimensional stealth requirements: realism, history alignment, and trajectory faithfulness. These properties are inherently interdependent yet distinct—no single metric (like reconstruction error) suffices. While reconstruction error partially addresses realism, it ignores history alignment and trajectory faithfulness. Forcing SHIFT into SA-MDP’s framework would sacrifice its core innovations for theoretical convenience.

Q3b: Finetuned DMBP against SHIFT?

A3b: As long as the attacker has the knowledge of how the victim finetune the diffusion model, the attacker can mimic the same process to generate the diffusion model as its new conditional diffusion base and reapply policy/realism guidance.

Q4: Questions for the trilemma.

A4: The trilemma is an empirical observation, not coming from SHIFT’s framework. We do not rule out the possibility of future methods overcoming this trade-off. Existing attacks like PGD, MinBest, PA-AD, and attacks based on high-sensitivity directions (e.g., blur, rotation) prioritize historical alignment and trajectory faithfulness but lack semantic change. SHIFT-O achieves semantic change and trajectory faithfulness, with slightly lower historical alignment, while SHIFT-I achieves semantic change and historical alignment, with slightly lower trajectory faithfulness.

Q5: Table 2 only covers the Freeway environment and DP-DQN defense.

A5: We provide extra results for Pong and SA-DQN below. Since SA-MDP is a relatively weak defense, other attack baselines also perform well. In terms of the stealthiness metric, it shows the same trend as Table 2. Table 2 focuses on DP-DQN because it represents the strongest existing defense, robust against large-budget attacks like PGD-15/255.

Q6: What architecture and training schema is used for autoencoders? Why only perform a single step and step size 1 in the realism guidance? Resources used by GRAD?

A6: The autoencoder uses 3 convolutional layers followed by a linear layer; the decoder mirrors this structure. It is trained with the Adam optimizer and MSE loss. We use a single realism guidance step with step size 1 as a balanced choice—small enough to preserve sample quality, yet large enough to enhance realism. A single step also ensures efficiency, as multiple steps would significantly increase computing time during SHIFT generation. GRAD uses V100 GPU and 20 hours for training defenses for environments such as Hoppers and Walkers, which does not use images as states. Thus, it is hard to migrate GRAD to the image domain due to computational overhead. We only have a single RTX 3090 and the training time for SHIFT is around 3-4 hours.

Q7: What is the main real-world use case for SHIFT? In what situations can an attacker fully control the victim’s visual input? How is it realistic or relevant in practice?

A7: SHIFT applies to real-world control tasks with visual input, such as robotics and autonomous driving. It assumes control over the agent’s observations, which is a common setup in prior work (e.g., SA-MDP, PA-AD). We also evaluate SHIFT under limited budgets (Appendix D.6). This is not a true best-case for the attacker, as SHIFT only alters observations, not actions. A stronger attacker would have total control of the victim and could directly manipulate the agent’s actions, which we do not assume.

Q8: Questions related to semantic changed states effectiveness.

A8: Attack effectiveness means the perturbed states cause the agent to take lower-reward actions. For example, in Figure 2, SHIFT generates semantically altered states that mislead the car into driving through a crosswalk with a pedestrian, causing a collision and reducing reward. While non-semantic attacks like PGD can degrade DQN performance, they fail against stronger defenses like diffusion-based DP-DQN (Table 2), where SHIFT remains effective.

A9: During the actual testing episode of the state perturbation attack, the victim agent only sees the perturbed states and has no access to the true states. Therefore, the victim cannot compute SSIM nor LPIPS.

A10: There may be multiple valid projection points, so we introduce D(⋅) to provide a mathematically rigorous and well-defined definition.

A11: The reference policy is used to collect clean trajectories for training the diffusion model with some random actions (i.e., exploration) to ensure the data covers a diverse range of trajectories, including both low and high reward cases. Also see A1 to Reviewer u3xt.

A12: Black et al. use reinforcement learning, an optimization method, to train a diffusion model, which is a generative model. This bridges generative modeling and optimization-based training.

We hope our detailed rebuttal address could your concerns.

Dear Reviewer 3Yfq,

Thanks for carefully reading our rebuttals and we will address your remaining concerns one by one.

Q1a: Concerns about being stealthy toward humans.

A1a: When only a single frame is available, both SHIFT-O and SHIFT-I appear realistic and plausible to human observers, due to the visual fidelity provided by diffusion models. However, high-sensitivity attacks (e.g., rotation) reported in [31] are easily noticeable to humans even in single-frame settings. When multiple frames are available, however, SHIFT-O becomes less stealthy to humans with domain knowledge because of its relatively weak historical alignment, whereas SHIFT-I maintains stealthiness by generating consistent illusionary trajectories. If true states can be queried, however, SHIFT-I can also be detected, as its generated trajectories may diverge significantly from the true state.

Q1b: Setting of Figure 6a and how to compare with Table 2.

A1b: Figure 6a was conducted in the Freeway environment using a vanilla DQN policy under SHIFT-O attack, and the 1.06(0.5) reconstruction error was derived from the raw data. Hence, it is not directly comparable to Table 2, which used DP-DQN as defenses. Following the reviewer’s suggestion, we reran the experiments in the Freeway environment under the DP-DQN policy without realism guidance and report the results: SHIFT-O: 1.07 (0.4), SHIFT-I: 1.11 (0.5). These results are now under the same setting as Table 2 and are directly comparable to all other baselines.

Q2: Question regarding DIRE and MAD and CUSUM detectors.

A2: Our SHIFT attacks can adapt to various defense methods by using the defense policies to guide the generation of perturbed states. However, DIRE is different from typical defenses that train a robust policy mapping from states to actions. Instead, DIRE is a detection method for states that are generated by a diffusion model. Thus, SHIFT cannot directly use DIRE for policy guidance. But, it is possible to incorporate DIRE distance into SHIFT, similar to our realism enhancer, by minimizing DIRE distance. However, this requires additional designs and experimentation, making it an interesting future work. For the DIRE experiment, we used our diffusion model conditioned on true historical states and actions to generate clean diffusion samples.

Both MAD and CUSUM operate on Wasserstein-1 distances between consecutive observed states and use statistics from clean trajectories as baseline detection metrics. For MAD, the baseline is computed as median + 5 (threshold) × MAD, which is 0.00116 in the Freeway environment. MAD flags an anomaly when the Wasserstein distance exceeds this baseline for three consecutive steps. CUSUM employs a more complex calculation (details in [41]). With drift = 1.5 and threshold = 3, the resulting baseline in Freeway is 0.00185. CUSUM triggers detection when the cumulative sum of deviations surpasses this baseline. For both methods, we set the hyperparameters (threshold and drift) to the lowest values that ensure clean trajectories remain undetected.

Q3/5: Concerns regarding the evaluations

A3/5: We note that DP-DQN is not tailored to either $l_p$-norm-based or diffusion-based attacks. It uses a diffusion model to purify perturbed states, forms a belief about possible true states, and applies a pretrained max–min strategy to choose robust actions. Its robustness is not specific to any attack type. The DP-DQN paper only tested $l_p$-norm attacks because no well-established attacks beyond $l_p$-norm constraint existed at the time. Thus, using DP-DQN as a defense to evaluate SHIFT and other attacks is fair.

Further, as discussed in A3 of our response to Reviewer vqkJ, designing a strong defense capable of mitigating SHIFT is non-trivial. Even combining reconstruction error with the SA-MDP framework would require substantial investigation, since reconstruction error from an autoencoder cannot be directly measured or clipped in the same way as an lp​-norm constraint. Developing such a defense remains a promising direction for future work.

Q4: Trade-off between different attack properties

A4: Our “trilemma” observation is mainly empirical and is derived from our evaluation of SHIFT and other attacks, but we do not have a formal proof that a trade-off must exist. In our experiments, a trade-off emerges for our semantics-changing SHIFT-O and SHIFT-I attacks. As shown in Table 2, SHIFT-O attains higher trajectory faithfulness, while SHIFT-I achieves stronger historical alignment. However, this pattern does not hold for PGD, which does not produce essential semantic changes under a small budget. Shrinking the PGD budget from 15/255 to 1/255 improves both historical alignment and trajectory faithfulness. These results suggest that when semantic changes are present, a trade-off between historical alignment and trajectory faithfulness would occur.

We thank you again for your valuable feedback, and we will integrate all clarifications into our manuscript.

Dear Reviewer 3Yfq:

Thanks for the follow-up questions and we will respond to your last concern here.

Q1: Relations between non-$l_p$-norm-based attacks (SHIFT and [31]) and $l_p$-norm-based defenses

A1: We agree that it is important to highlight that current $l_p$-norm-based defenses, such as SA-DQN and WocaR-DQN, are specially designed to handle state perturbations constrained within a fixed $l_p$-norm distance. Our SHIFT attacks, as well as the attacks presented in [31], operate outside of this framework and exploit this limitation. These non-$l_p$ attacks can bypass $l_p$-norm-based defenses because the robustness regularizers used during their training do not account for perturbations beyond the $l_p$ constraint.

In our evaluation, we also included two diffusion-based defenses, DMBP and DP-DQN, which are not limited by $l_p$-norm assumptions. These defenses aim to reconstruct clean observations and are therefore more robust to a broader range of perturbations. As shown in Table 2 and Appendix D.8, the attack from [31] is largely ineffective against these diffusion-based defenses, because it does not introduce meaningful semantic changes. In contrast, to the best of our knowledge, SHIFT is the only attack that successfully bypasses these diffusion-based defenses.

Additionally, as noted in A1 of our previous response, the attacks from [31] are more perceptible to humans in single-frame settings, making them less stealthy than SHIFT.

We hope this response fully addresses all your remaining concerns. If so, we would greatly appreciate it if you could consider updating your final rating to reflect our discussion.

Sincerely,

Authors of Submission 9986

Dear Reviewer 2wp2:

Thanks for your insightful feedback, and we will address your concerns in the following.

Q1: Computational Complexity. The use of diffusion models and the need for classifier-free and policy guidance introduce significant computational overhead. Training the conditional diffusion model and the autoencoder-based anomaly detector can be resource-intensive, which may limit the practical applicability of SHIFT in real-time scenarios.

A1: Although SHIFT may incur a certain level of computational overhead, training the diffusion model takes around 4 hours and the autoencoder-based anomaly detector takes about 1 hour on the Freeway environment—both of which can be done in parallel. In comparison, other learning-based attack methods such as PA-AD[1] require around 12 hours on the same hardware to train an attack policy on Freeway. Therefore, SHIFT requires significantly less training time than PA-AD, a state-of-the-art learning-based attack method.

On the defense side, prior defense methods such as SA-DQN and WocaR-DQN require more than 24 hours to train the defense policies, which is longer than the training time of our SHIFT attack. This creates a sufficient time window for SHIFT to inject adversarial samples before such defenses are deployed.

[1] Sun et al., Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. ICLR 2022

Q2: Sampling Efficiency. While the authors mention using the EDM formulation to improve sampling efficiency, the overall computational cost of generating perturbations in real-time remains a concern. This could be a barrier for deploying SHIFT in environments that require rapid decision-making.

A2: While SHIFT may introduce a moderate sampling overhead, we emphasize that all experiments were conducted using a single RTX 3090 GPU. Despite this relatively limited computing resource, SHIFT achieves practical sampling times (approximately 0.2 seconds per sample). With more powerful hardware, the efficiency of SHIFT could be further improved, making real-time deployment increasingly feasible.

Q3: Limited Scalability. The diffusion model needs to be trained separately for each distinct environment, which can be impractical for large-scale deployment. This limits the scalability of SHIFT, especially in scenarios where the RL agent operates in multiple or dynamically changing environments.

A3: Thank you for highlighting this important scalability concern. In our current setup, we do train a separate conditional diffusion model for each environment. However, a promising direction for future work is to train a single conditional diffusion model across multiple environments by leveraging the history conditioning mechanism. The history input provides contextual cues that help the model infer which environment it is operating in, preventing mode collapse or sample confusion.

A similar idea has been explored in TD-MPC2 [2], which learns transition dynamics across multiple environments in a shared latent space. Their results show that a single model can handle multiple tasks simultaneously. We could extend this approach by adopting latent diffusion models and performing attack guidance in the latent space, allowing a single diffusion model to support various environments. This strategy would significantly improve SHIFT’s scalability, making it more suitable for complex or dynamically changing environments.

[2] Hansen et al., TD-MPC2: Self-Supervised Decision Making with Latent World Models, NeurIPS(2023)

We hope our detailed rebuttal could address most of your concerns. If you have further questions please let us know and we will try our best to clarify them.

Sincerely,

Authors of Submission 9986

Dear Reviewer u3xt:

Thanks for your insightful feedback and we will address your concerns in the following.

Q1: In lines 242–243, is an offline dataset collected from a clean environment required, or is it necessary to generate trajectory data by interacting with the clean environment using $\pi_{ref}$ to assist training? What are the requirements for $\pi_{ref}$? Does it need to ensure a certain level of coverage over the state or trajectory distribution?

A1: Yes, SHIFT requires collecting sufficient trajectory data from a clean environment to train the conditional diffusion model. To this end, we use a pre-trained reference policy $\pi_{ref}$ that performs well in the unperturbed environment. To ensure adequate coverage over the trajectory distribution, we incorporate a small amount of random actions (i.e., exploration) during data collection. This helps the conditional diffusion model learn not only high-reward behavior but also the broader dynamics of the environment, including suboptimal trajectories.

Q2: Why does SHIFT-I not incorporate information about $a_{t-1}$? What would the performance of SHIFT-I be if $a_{t-1}$ were included? Similarly, how would the performance of SHIFT-O be affected if it also omitted ${a_{t-1}}$?

A2: This is a great question. In SHIFT-I, we deliberately omit $a_{t-1}$ to relax the conditional generation space, allowing the diffusion model to search for perturbed states that better align with the observed history. Including $a_{t-1}$ in SHIFT-I would overly constrain the generation distribution, introducing subtle inconsistencies—similar to those observed in SHIFT-O—that harm historical alignment. Empirically, including $a_{t-1}$ in SHIFT-I results in an attack performance of 11.3 (4.1) in the Freeway environment against DP-DQN, which is comparable to the original result of 10.6 (2.5). However, stealthiness is reduced: the Wasserstein distance of SHIFT-I increases from $0.84 (0.2) × 10^{-3}$ to $0.90 (0.2) × 10^{-3}$, indicating degraded historical alignment. Combined with SHIFT-I’s inherently lower trajectory faithfulness, this makes the attack significantly less stealthy.

On the other hand, omitting $a_{t-1}$ from SHIFT-O improves attack performance (from 21.8 (3.2) to 14.1 (2.8)) in the Freeway environment against DP-DQN but at the cost of trajectory faithfulness: SSIM drops from 0.9990 (0.0014) to 0.9915 (0.0015), and LPIPS increases from 0.0008 (0.0014) to 0.0121 (0.001).

In summary, the inclusion or omission of $a_{t-1}$ reflects a trade-off between historical alignment and trajectory faithfulness. Each SHIFT variant is designed to balance this trade-off in accordance with its design goal.

Q3: Regarding the experiments in Table 2, how would the performance of SHIFT be affected if the attack frequency were reduced, for example, in variants like SHIFT-I-0.25 and SHIFT-O-0.25?

A3: We provide an ablation study on different attack frequencies—including 1.0, 0.5, 0.25, and 0.15—for both SHIFT-I and SHIFT-O in Appendix D.6. The results show that even at lower frequencies such as 0.25, SHIFT maintains a strong attack performance, demonstrating its effectiveness under limited perturbation budgets.

Q4: The generation time of SHIFT is approximately 0.2 seconds. It remains unclear whether, in practice, the agent can detect the presence of an attack by observing differences in generation times of subsequent states between normal (no-attack) and SHIFT attack scenarios. Furthermore, it is of interest to investigate how increasing the generation iterations of methods such as PGD, PA-AD-TC, MinBest, and PA-AD, or decreasing the number of reverse steps in SHIFT—thereby aligning their adversarial state generation times to the same scale—would affect their comparative performance.

A4: The 0.02 seconds reported for PGD, MinBest, and PA-AD are based on 10 steps of gradient descent. As shown in the SA-MDP paper[1], using 50 steps is also common. We tested these methods with 50 steps in Freeway under DP-DQN defense. In terms of attack performance, PGD(1/255) with 50 steps has 29.8(1.0) compared with 10 steps 30.0(0.9) and PGD(15/255) with 50 steps has 29.2(0.8) compared with 10 steps 29.0(1.0). This is mainly because even with 10 steps, perturbation has reached the budget limit under PGD attack. While the generation time increased to an average of 0.1 seconds, their attack performance did not improve, as they still failed to bypass DP-DQN. In this setting, SHIFT takes about twice the time but offers significantly stronger attack performance. However, it is challenging to reduce SHIFT's generation time further—reducing the number of reverse steps in SHIFT would degrade the quality of adversarial samples. That said, our experiments were conducted under limited compute resources (a single RTX 3090). With more powerful hardware, the sampling time of SHIFT could potentially be further reduced.

We hope our detailed rebuttal could address most of your concerns. If you have further questions please let us know and we will try our best to clarify them.

Sincerely,

Authors of Submission 9986

[1] Zhang et. al., Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. NeurIPS 2020.

Dear Reviewer vqkJ,

Thank you for your insightful feedback. We will address your concerns in the following.

Q1: The setting is unrealistic as an adversarial attack. If the attacker had this much control over the observations, they would likely have access to control the hardware directly and would not need to resort to this level of complexity. In what scenario would an attacker have the ability to execute this attack while not having the means and inclination to do a much more direct attack? (like taking over the steering directly)

A1: The state perturbation attack setting assumes that the attacker can manipulate only the observations received by the victim agent, thereby misleading it into choosing suboptimal actions. This assumption is standard in prior literature, including SA-MDP[1] and PA-AD[2]. We argue that gaining access to an agent’s sensory input (e.g., camera or radar) is considerably more feasible than gaining full control over its actuation system. For example, in autonomous driving, it may be easier for an attacker to hack into perception components, such as cameras or LiDAR, to subtly manipulate visual inputs, rather than breach the entire control system to directly steer the vehicle.

Moreover, our attack is relevant to real-world robotic and autonomous systems that rely on visual observations. SHIFT can be deployed in such scenarios to induce subtle, semantically meaningful perturbations without needing full system control.

[1] Zhang et al., Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. NeurIPS 2020.

[2] Sun et al., Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. ICLR 2022.

Q2: In addition, it is not clear how making it look semantically reasonable makes the attack undetectable. It is quite noticeable that the pedestrian has been removed from the image in Figure 2. If both videos are saved for audit, or if a human were watching, it would be quite visible.

A2: As mentioned in Q1/A1, during testing the victim agent only observes the perturbed states and has no access to the true states. Therefore, it cannot detect perturbed states or compare perturbed and true states in real time. Moreover, even if both true and perturbed trajectories are saved for future auditing, the attack would have already influenced the victim’s decision-making and degraded its performance before being detected. Our focus is on real-time evasion during deployment, where SHIFT remains hard to detect under typical assumptions in the state perturbation threat model.

Q3: Further, if we were aiming to use this for adversarial training it is unclear what we would want the agent to do. Ultimately if the image looks indistinguishable from a clear road ahead I can see no way that any system would be able to avoid hitting pedestrians. This is why perturbations for ML models in adversarial attacks have been norm-ball restricted, because we aim to preserve the semantics of the image as to find situations where the agents decision changes when it should not. In the situations generated in this paper, the behavior of the system ought to change.

A3: This is an insightful point, and we agree that applying SHIFT for adversarial training is a promising future direction. Compared to traditional $l_p$ norm-based perturbations, SHIFT induces semantic changes, allowing us to go beyond simple pixel-level perturbations. Instead of assuming small norm perturbations that preserve semantics, agents under SHIFT can be trained to form a semantic belief about the underlying true state and make robust decisions based on this belief when encountering perturbed observations. We also acknowledge that in some scenarios, the semantic change may be too strong for the agent to act optimally,e.g., when a pedestrian is fully removed. However, this highlights the value of budget-aware sensing: if the agent has a limited probing budget (e.g., can occasionally access true observations), SHIFT can help identify high-risk situations in which the agent should query the environment more carefully. Thus, SHIFT can facilitate learning robust policies by revealing when perception failure is most consequential.

Q4a: The definition of "valid states" would preclude the sorts of semantically sensible generations that we see from diffusion models at scale which are combinatorial generalisations of realistic images like "an astronaut riding a horse on the moon". Given that this technique is based on diffusion models it would likely allow for these sorts of combinatorial generalisations but they are disallowed by definition 3.1.

A4a: The conditional diffusion model used by SHIFT is trained specifically on data collected from the target environment. For example, in the autonomous driving scenario, we train the model from scratch using trajectories recorded in clean, unperturbed driving environments. As a result, the trained diffusion model captures the semantics and physical constraints of the environment and does not generate out-of-distribution samples such as "an astronaut riding a horse on the moon."

Q4b: More concerning is that the definition of semantics-changing states does not quite match the name. There are semantically identical images that are quite far in pixel space, only changing the texture, color, hue, or other irrelevant aspects of the image. I would have expected "semantics-changing" to involved aspects of the image which would change the policy.

A4b: This is a great point. We agree that certain image-level changes, such as texture, brightness, or contrast, can appear large in pixel space but remain semantically identical. However, state-of-the-art diffusion-based defenses like DMBP and DP-DQN are specifically designed to denoise and project the input back onto the valid states. As shown in Appendix D.8, attacks based on high-sensitivity directions (e.g., blur, brightness changes) fail to affect these defenses.

In contrast, SHIFT generates semantically altered states that can bypass diffusion-based denoisers and actually lead the agent to take different actions. Still, as shown in Appendix D.8, attacks based on high-sensitivity directions can change the policy behavior under weaker defenses or no defense at all.

This leads to an important nuance: if we define “semantics-changing” solely by whether the agent’s action changes, the definition becomes policy-dependent. For example, brightness changes might alter the actions of a vanilla DQN agent but have no effect on a DP-DQN agent. To avoid such ambiguity, our definition of semantics-changing states (Def. 3.3) does not rely solely on policy behavior and does not exclude non-essential semantic changes—reflecting the broader goal of evaluating stealthy, history-aligned, and behavior-impacting perturbations.

Q5: The approach should cite works from the generative environment literature, as it is essentially training a generative world/environment model. In essense, this can be seen as an application of a generative world model, or a way of leveraging a pre-trained generative world model for adversarial search.

A5: Thanks for pointing out this line of related work and we will add a section in the related work in the revised version of our paper. We will include papers such as [3][4][5].

[3] Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination. ICLR 2020.

[4] Pérez et al., EnvGen: Generating Diverse and Challenging Environment Variants. ICLR 2023.

[5] Zhou et al., DINO World Model (DINO‑WM): World Models on Pre‑trained Visual Features Enable Zero‑Shot Planning. ICML 2025.

We hope our detailed rebuttal could address most of your concerns. If you have further questions, please let us know and we will try our best to clarify them.

Sincerely,

Authors of Submission 9986

Dear Reviewer vqkJ:

Thanks for reading our rebuttal and we will clarify your further concerns.

Q1: Totally controlling the sensors is not realistic in practice. Is there any chance to defend SHIFT?

A1: While total control of the victim’s sensors is indeed a strong and often unrealistic assumption, we have considered more practical scenarios in our study. Specifically, we investigate the case where SHIFT operates under a limited budget, restricting the attacker to inject perturbations only during a fraction of the time steps. As shown in Appendix D.6, even with such constraints, SHIFT can still significantly degrade agent performance. Additionally, we acknowledge that attackers may only be able to perturb a subset of pixels in each state, which is more realistic than full sensor control. Our preliminary idea involves applying SHIFT within a masked region of the state, constrained by a pixel budget. Identifying an optimal mask under these constraints is non-trivial and represents a promising direction for future research.

Regarding defenses, we do not rule out the possibility of defending against SHIFT. As mentioned in A3, agents could leverage historical information to build semantic beliefs about the true state, enabling robust decision-making even under perturbed observations. Moreover, the inherent signatures of diffusion models offer another avenue for defense. For instance, detectors like DIRE can identify images generated by diffusion models, even if they appear realistic to humans. While current methods are not yet effective against SHIFT (see our response to Reviewer 3Yfq), we believe that further exploration of diffusion model’s inherent signatures could lead to more efficient defenses.

Q2: “Valid States” definition is too narrow.

A2: We appreciate the reviewer’s clarification regarding the concern of “valid states.” We agree that diffusion models can generate combinatorial generalizations not present in the training set. However, whether these generations are considered valid states depends on the underlying world model.

To illustrate, consider the example “an astronaut riding a horse on the moon”:

1. If the world model is sufficiently general to simulate various humans riding different objects on different planets, the training data would include both humans riding horses on Earth and astronauts driving rockets on the moon. In this case, the diffusion model could generate “an astronaut riding a horse on the moon,” and since the world model can simulate this scenario, it would be considered a valid state.

2. If the world model is narrower and only simulates humans riding different objects on Earth, the training data would not include moon scenarios, and “an astronaut riding a horse on the moon” is very unlikely to be generated from a diffusion model traind from scratch using only data from this world model.

3. If the world model used in the first case incorporates real biological simulations—meaning, for example, that a horse cannot survive on the Moon. While the diffusion model might still generate an sample of “an astronaut riding a horse on the Moon”, such a scenario would be prohibited in the world model due to its adherence to biological constraints. As a result, “an astronaut riding a horse on the Moon” would be considered invalid state in this case.

We also agree that there is no universally “correct” definition of valid states; different definitions may be appropriate for different tasks. In the context of SHIFT attacks, our goal is to maintain stealthiness to humans with domain knowledge of the relevant task. Defining valid states based on a narrower, task-specific world model enhances stealthiness, as humans are more likely to detect out-of-context scenarios (e.g., “an astronaut riding a horse on the moon” in a driving task) as attacks, while SHIFT-generated perturbations may remain undetected. On the other hand, broader definitions of valid states may be beneficial for tasks that require greater diversity, such as generative environments.

We hope our further clarifications can address your concerns.

Sincerely,

Authors of Submission 9986