We are truly grateful for your valuable suggestions. Below are our responses to the questions raised.

Response to Weekness 1 & Q4: Real-world applicability of 3D SemanticAEs

In our supplementary material, we include a simulation video showing adversarial attacks under continuous camera rotation around the 3D object, demonstrating strong multi-view robustness. Following the reviewer's suggestion, we conducted a Blender-based physical simulation using sunlight illumination. The tested model is the forklift model in the supplementary video. Under the viewpoints, material and lighting settings different from the training stage, the ResNet50 classification accuracy on clean samples was 59.41%, while on adversarial samples it dropped to 20.0%. Crucially, the two models are visually indistinguishable, indicating that the generated 3D adversarial objects maintain natural appearance while achieving strong attack performance. Due to NeurIPS policy restrictions, we are unable to include visualizations, but we will release the simulation results in the camera-ready version.

Additionally, a concurrent work at ICCV 2025 [1] (preprinted after our initial submission) demonstrates that adversarial examples in 3D Gaussian representations can effectively enable physical-world attacks. Our semantic-level control advances this direction. We discuss the relationship between that work and ours below. LLM-related application is discussed in our response to Reviewer t5zq.

Response to Weakness 2: Use of Trellis may reduce perceived originality

We clarify that our framework is not limited to Trellis and can be adapted to other diffusion-based 3D generation models—similar to how Hugging Face Diffusers supports multiple architectures. Trellis serves as one implementation backend, and our design choices prioritize compatibility with existing open-source tools. From the perspective of AI safety governance, we argue that effectively leveraging widely adopted, open models and frameworks is not only realistic but essential. It addresses the economic and scalability challenges in adversarial robustness research, enabling broader access and faster iteration.

Moreover, to the best of our knowledge, no prior work had applied 3D Gaussian Splatting to adversarial example generation. We note that a concurrent ICCV 2025 paper [1] also adopts 3D Gaussian rendering and reports improved physical-world attack performance. While we omit extensive discussion of 3D reconstruction mechanics due to space constraints in the main paper, we believe that since 3D Gaussian Splatting projects 2D gradient feedback onto spherical harmonic bases, it effectively reduces optimization uncertainty and enables efficient inverse optimization of adversarial patterns. By integrating this with ResAdv-DDIM, we establish a multidimensional uncertainty reduction framework, stabilizing both the semantic (via residual approximation) and geometric (via 3D Gaussian representation) optimization. We believe this paradigm provides valuable guidance for scaling red-teaming frameworks to complex tasks.

Response to Weakness 3: Perceived incremental nature of some components

The core innovation of this paper lies in constructing an integrated robust attack framework through Instruction Uncertainty Reduction. Regarding component-level contributions:

1. By combining residual approximation and adaptive optimization, we establish a new technical solution for transferability enhancement, while unlocking the potential of multi-step diffusion models in improving adversarial transferability and robustness.

2. Motivated by scenario adaptability, we design an effective knowledge injection mechanism for ResAdv-DDIM from both internal sampling model and external guidance perspectives, supporting cross-task applicability.

3. To address ambiguity in evaluation, we propose an exemplar-based metric that alleviates the metric robustness problem posed by Clip-VQA-style assessment in the SemanticAE research. Detailed discussions are provided in Appendix B.3 and Appendix D.4.

Regarding the use of masked guidance, we have discussed its role in Appendix A.1. Masked guidance has been primarily used in diffusion-based image editing to constrain the edited regions, i.e., constraining $x$, with limited exploration on the guidance model $\epsilon$. As stated in Appendix A.1, in our 2D generation task, we take a further step by modeling the interaction between conditional and unconditional guidance, achieving the new function of re-distributing the spatial strength of the semantic constraint. This approach has not yet been explored in the adversarial attack domain.

Response to Question 1

Thanks for the suggestion. We will provide a more detailed explanation in the camera-ready version. Specifically, at diffusion step $t$, the adversarial update $\Delta x_t$ is computed as:

where $\hat{x}_0 \sim \mathrm{ResApprox.}[x_t]$. In principle, the higher the number of internal sampling steps in the $\mathrm{ResApprox.}$ process, the more accurate the estimation of $\hat{x}_0$, leading to lower variance $\mathrm{Var}[\Delta x_t]$. This results in more stable optimization, as reflected by fewer abrupt drops in attack strength between consecutive denoising steps. We verify this phenomenon in the following supplementary experiments.

Furthermore, our contribution goes beyond improving $\hat{x}_0$ estimation: by enabling stronger adversarial optimization in early denoising steps via ResAdv-DDIM, we enhance the transferability of the final adversarial sample. Since early steps determine the global structure of the generated image, focusing on attack optimization in early steps leads to more structural and semantically stable features, which are more likely to transfer across models.

We do not modify the momentum mechanism used in prior adversarial optimization methods. To isolate the effect of our design, we keep the momentum parameter identical to the baseline (VENOM). A brief review of momentum-based transfer attacks and their mechanisms is provided in Appendix A.3, and detailed implementation is described in Appendix B.1. The choice of parameter $\xi$ is discussed in our response to Reviewer rjVJ’s Question 2.

Response to Question 2

Thanks for the suggestion. We analyze convergence stability by measuring the mean and fluctuation of the estimated confidence of successful attack across different denoising stages of the diffusion model. Due to NeurIPS policy restrictions, we are unable to include visual plots. The quantitative results are presented below. When $k=0$, the sampler degrades to DDIM.

Response to Question 3

Our language instructions are formally defined in the evaluation methodology section (Section 3.4, Eq. 10). The specific label configurations are listed in the Appendix. Regarding instruction diversity, we conduct experiments under different levels of referring diversity to evaluate robustness. The results are shown below.

Supplementary Experiment

We evaluate the impact of different referencing diversity levels and varying residual approximation steps $k$. The surrogate model is ResNet50, and the diffusion model is stabilityai/stable-diffusion-2-1-base. We define four prompt sets with decreasing referencing diversity:

• P0: a dog on the floor
• P1: a dog sitting on the floor
• P2: a brown Labrador sitting on the wooden floor, facing and gazing us directly

The experiment is conducted 10 trials for each setting with random seeds from 0 to 9. To measure optimization stability, we define Fluctuations as the cumulative drop in attack success rate (ASR) during the denoising process $t = [T, T-1, ..., 0]$:

Additionally, we compute the Estimated ASR—the predicted success rate after removing adversarial optimization in later steps—as a measure of the effectiveness of the attack at step $t$.

The experimental results are summarized in the table below.

Overall, as $k$ increases, fluctuations decrease and Estimated ASR improves across denoising stages. Prompts with higher specificity induce smaller fluctuations, while more ambiguous prompts lead to larger fluctuations, especially when $k=0$. Due to stronger semantic constraints, more precise prompts may result in lower average ASR. We will incorporate these analysis, along with visualization results, into the main text in the camera-ready version.

[1] Lou T, Jia X, Liang S, et al. 3D Gaussian Splatting Driven Multi-View Robust Physical Adversarial Camouflage Generation[J]. arXiv preprint arXiv:2507.01367, 2025.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to evaluating our work. We have carefully addressed all the comments and feedback provided in the review, and the corresponding responses are detailed in the rebuttal section above. We hope that our clarifications and revisions have adequately resolved the concerns raised.

Given the approaching deadlines for the author-reviewer discussion, we would greatly appreciate your prompt feedback on any remaining questions or concerns. Timely communication will ensure we can effectively address all issues and improve the quality of our work.

Thank you once again for your valuable insights and guidance!

Best regards,
Submission 2210 Authors.

We sincerely appreciate your recognition of our work's technical rigor and innovation. Below are our responses to the question raised.

Q1: Regarding the ResAdv-DDIM sampler (Section 3.2): Could you provide more intuition on the choice of $k$ (the number of coarse prediction steps)? Is there a principle for selecting the optimal $k$, or is it purely an empirical hyperparameter?**

R1: In principle, within a certain range, a higher $k$ leads to stronger adversarial optimization in the early denoising steps of diffusion, resulting in higher attack transferability—but at the cost of increased computational time. We provide a detailed ablation study analyzing this trade-off in Table 3 of the main paper (partial results shown below). In the original table, Iter$_\mathrm{max}$ corresponds to the maximum number of coarse prediction steps. The specific scheduling strategy for $k$, i.e., how many prediction steps to use at each denoising step based on remaining timesteps, is described in Appendix B.1 (in the supplementary material), with $K$ being the maximum allowed prediction step.

Further mechanistic analysis on the impact of $k$ on optimization stability is provided in our response to Reviewer vVyk. We believe that the choice of $K$ should be made based on the specific application scenario, balancing effectiveness and efficiency. In practice, $K$ can be selected adaptively by integrating with AutoML frameworks, using example instructions $\mathrm{Text}$ and stability metrics to determine an appropriate $K$ for the deployment environment.

Q2: Regarding 3D generation (Table 2): The results are extremely impressive, with ASR jumping to 92.2. The non-adversarial baseline already has a classification accuracy of 21.5 on the target model. Does this suggest that classifiers are inherently less robust to rendered 3D objects, making them an easier target for adversarial attacks compared to real 2D images?

R2: We believe this is due to the nature of the task. Nevertheless, our results sufficiently validate the effectiveness of the proposed method. Specifically,

1. Although the average clean accuracy across the dataset is low, for certain labels, the generated 3D objects achieve high clean accuracy under multi-view sampling. On these labels, our method still successfully generates effective 3D adversarial examples (visualized in Figure 11 of the main paper and the supplementary videos).
2. Our $ASR_\mathrm{relative}$ metric is designed precisely to address this issue: it excludes samples where the clean version is already misclassified, eliminating the influence of inherently hard-to-recognize objects. $ASR_\mathrm{relative}$ on clean samples could be regarded as 0%.
3. Ablation studies show that ResAdv-DDIM significantly improves attack performance, indicating this is a non-trivial problem.

Q3: Regarding the relationship between referring diversity and descriptive incompleteness. The paper neatly separates these concepts. However, could there be an interplay? For example, if an instruction is incomplete ("image of a car"), does this increase the diversity of valid references, making the optimization problem harder? How does your framework handle this potential interaction?

R3: We agree with the reviewer that these concepts are interrelated. As the reviewer points out, we view referring diversity as having a direct impact on optimization, while descriptive incompleteness has an indirect effect. From a design perspective, we believe it is reasonable to first develop optimization methods targeting referring diversity, and then define contextural knowledge embedding methods that account for issues arising from descriptive incompleteness. Specifically, referring diversity characterizes the optimization challenge faced by transfer attack algorithms, while descriptive incompleteness relates to how the optimization objective is defined. InSUR provides a collaborative design solution for these two issues. We will add further clarification on this methodological design principle at the beginning of the methodology section to better address this point.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to evaluating our work. We have carefully addressed all the comments and feedback provided in the review, and the corresponding responses are detailed in the rebuttal section above. We hope that our clarifications and revisions have adequately resolved the concerns raised.

Given the approaching deadlines for the author-reviewer discussion, we would greatly appreciate your prompt feedback on any remaining questions or concerns. Timely communication will ensure we can effectively address all issues and improve the quality of our work.

Thank you once again for your valuable insights and guidance!

Best regards,
Submission 2210 Authors.

We sincerely appreciate the reviewer's positive feedback on our approach and their meticulous proofreading to improve the readability. Below are our responses to the question raised.

Response to Weakness 1: limited evaluation on foundation models.

We thank the reviewer for the comment. While our primary evaluation focuses on standard architectures, we acknowledge the importance of testing on foundation models, given the paper’s emphasis on semantic alignment. To address this, we have conducted additional experiments on VQA tasks, as detailed in our response to Reviewer t5zq, with results presented at the end of that reply. Furthermore, in Appendix D.4 (provided in the supplementary material), we show that our generated adversarial examples effectively perturb the CLIP-VQA evaluation metric itself, indicating that semantic assessment based on CLIP is also vulnerable to such attacks. This highlights a key challenge: evaluating adversarial examples becomes difficult when the evaluator model is not robust. To mitigate this dependency, we propose the $ASR_\mathrm{relative}$ metric, which reduces reliance on absolute semantic evaluation by using relative performance comparison. We believe these findings are valuable for research on alignment and even super-alignment, as they reveal vulnerabilities not only in perception models but also in the metrics used to assess them.

Response to Weakness 2: freeze the structured latent (z₀^slat) is heuristic

Our choice of freezing the structured latent is not purely heuristic. The reasons are:

1. $z_0^{slat}$ encodes the 3D positioning data, while the accurate positioning is also rectified by $z_0$ in the Trellis framework. We could achieve the positioning disturbance also by optimizing $z_0$.
2. Estimating the adversarial feedback from the surrogate model to $z_0^{slat}$ is not computationally efficient, since it shall undergo the entire denoising diffusion process.
3. The computation from $z_0^{slat}$ to $\mathbf{pos}$ contains a quantization operation, which is non-differentiable.

Furthermore, $z_0^{slat}$ is an internal data structure of Trellis, and the optimization method customized specifically for this structure will be constrained by the Trellis framework. Besides, even with z₀^slat frozen, we still achieve favorable optimization results. We believe such optimization shall be developed together with harder attack tasks.

Response to Weakness 3: limited evaluation on prompt-based attacks.

Thanks for the comment. As stated in Section 3.1, our work focuses on a fixed instruction: how to generate an image that satisfies the semantic constraints of that instruction while being adversarial. We acknowledge that perturbing the instruction itself (i.e., prompt attacks) can also lead to adversarial effects. However, most existing prompt attacks are not designed for tasks similar to SemanticAE.

Our baseline, SD-NAE, performs attacks by perturbing the prompt embedding, making it approximately equivalent to prompt attack methods, but more aligned with our task objective. We will include a detailed description of SD-NAE in the Appendix, and ensure it is appended to the main paper in the camera-ready version.

Moreover, we discuss this distinction in Appendix A.2. Specifically, generating hard samples from language can be approached via two paradigms:
(1) Perturbing the input text without altering the language-guided generation model. (2) Altering the generation process by introducing adversarial capabilities into the model.
Our work focuses on the latter. In practice, these two approaches can be combined into a unified pipeline. Our evaluation could be regarded as assessing the effectiveness of our method as a module within the red-team framework.

We further tested a recent method that uses prompt optimization to generate adversarial images [1]. In our local single-GPU setup, it took several hours to find a single adversarial prompt, which is thousands of times slower than our method and the baselines we compare against. This performance gap significantly limits its applicability in scalable red-teaming systems, and it shall be deployed in different parts of the system compared to our method.

[1] Zhu, Xiaopei, et al. "Natural language induced adversarial images." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

Q1: In Eq. 4, how sensitive is the ResAdv-DDIM sampler to the choice of $k$?

R1: We conducted ablation studies comparing different $k$ value settings (Iter$_\mathrm{max}$ in Table 3 of the main paper). We will clarify it to make it clearer. As $k$ increases, attack effectiveness gradually improves, but so does computational cost. In practice, $k$ can be adjusted based on the specific application. Furthermore, following the suggestion from Reviewer vVyk, we provide additional analysis on the stability of adversarial optimization under semantic guidance instructions with varying ambiguity.

Q2: In Eq. 6, could you clarify how the thresholds $\xi_1 = 0.1$ and $\xi_2 = 0.01$ were chosen? Were these hyperparameters tuned for attack performance, or are they fixed across all tasks/models?

R2: They are fixed across all tasks and models. In our framework, $\mathrm{Confidence} < \xi_1$ indicates that the "residual approximation" predicts the attack performance will be sufficiently strong, so the inner optimization at the current denoising step is terminated and the process proceeds to the next denoising step. When $\mathrm{Confidence} < \xi_2$, it means the predicted attack performance already exceeds the task requirement—further optimization is unnecessary, and even if subsequent denoising steps improve naturalness at the cost of reduced attack strength, this is acceptable. These conditions reduce the expected number and lower bound of optimization steps. In Appendix B.1, we detail analyze the benefits of this design on generation efficiency (Proposition B.1 and related discussions).

We chose these threshold values heuristically and used them consistently across all tasks without manual tuning for attack performance or computational speed.

Below, we present results using ResNet50 as the surrogate model on the Abstract Label Evasion Task. In white-box attacks, all tested threshold values achieve strong performance (accuracy after attack < 3%). In black-box attacks, smaller $\xi_1$ values yield higher transferability, because lower $\xi_1$ leads to more aggressive adversarial optimization in early denoising steps—optimization that tends to be more transferable (this is consistent with the design principle of ResAdv-DDIM, which aims to enhance early-step optimization). However, this also increases computational cost. A practical deployment could balance effectiveness and efficiency.

Q3: In Line 192–193, the authors mention “M is the guidance masking that regularizes the ....". It is unclear how M is defined.

R3: The definition of $M$ is explained in Appendix B.1 (in the supplementary material). We will include the appendix after the main text in the camera-ready version and add a reference to it after this sentence. Specifically,

In the main experiments, we set $M_\mathrm{mid} = 3.0$ and $M_\mathrm{edge} = 0.3$. We further test $M_\mathrm{edge} = \{0.0, 0.3, 3.0\}$ in the ablation study.

Q4: In Table 2, given that generation times vary widely (e.g., up to 59s), is most of this cost from the Gaussian splatting renderer, diffusion steps, or EoT sampling? Can you attribute bottlenecks quantitatively?

R4: The variation in generation time arises from the adaptive number of optimization steps and differences in the size of sparse tensors in the Trellis framework. Attributed to the proposed early stopping mechanism, easier attack cases require fewer iterations and thus less time. The large variance in generation time reflects significant differences in attack difficulty across tasks, which highlights the effectiveness of our adaptive step design.

We further conducted a timing analysis on a single GPU using 100 labels from the Imagenet-Label dataset. The results are shown below. Gradient computation and residual approximation account for the majority of the computational cost. Meanwhile, EoT sampling is implemented via multi-view rendering using the Gaussian splatting renderer, while the rendering parameter computation does not occupy CUDA execution time.

For the single adversarial optimization iteration:

• Denoise Sampling: 123.321 (std=36.878)ms
• Residual Approximation: 147.256 (std=76.243)ms
• Gaussian Decoding: 21.547 (std=11.868)ms
• Gaussian Rendering: 28.556 (std=7.706)ms
• Surrogate Model: 6.777 (std=0.745)ms
• Backward Process: 208.471 (std=78.039)ms

The fluctuation in computation speed is primarily caused by inconsistent data sizes in the Sparse Tensor.

For the entire adversarial optimization:

• Denoise Sampling: 2994.933 (std=269.841)ms
• Residual Approximation: 7040.238 (std=8915.112)ms
• Gaussian Decoding: 1029.394 (std=1185.636)ms
• Gaussian Rendering: 1364.221 (std=1732.996)ms
• Target Model: 323.782 (std=423.503)ms
• Backward Process: 6261.577 (std=12119.871)ms

The fluctuation in computation speed is also attributed to the adaptive optimization steps. Specifically, the average count of optimization steps for the inner optimization is: 47.810 (std=63.566).

Hi Authors,

Thank you for your hard work on the rebuttal. All concerns are addressed and the reviewer has updated the rating to "Accept". Best of luck!

Regards,
Reviewer rjVJ

We sincerely appreciate your constructive feedback. Below are our responses to the questions raised.

Q1: How to control the abstracting level of the antonyms of the Text. Cat is an antonym of dog, and car is also an antonym of dog, but they are different-level antonyms.

R1: We acknowledge that the term antonyms in the original manuscript (Section 3.1) was imprecisely used and could lead to misunderstanding. Specifically, $A_{\text{Text}}$ refers to output types that are conceptually different from the original text, not necessarily strict linguistic antonyms. In fact, in the untargeted attack scenario, these should be understood as non-synonyms (e.g., cat is not an antonym of dog, but it is a non-synonym). As in prior transfer-based attack works, our experiments focus on the untargeted setting to evaluate effectiveness.

Our proposed benchmark already addressed the evaluation bias caused by overly fine-grained semantic abstraction. We agree with the reviewer that different abstraction levels inherently lead to different attack difficulties, as supported by prior work [1]. To account for this, we define the abstraction level of $A_{\text{Text}}$ via the scope of synonym coverage: broader synonym sets imply more distant "antonyms." In our experiments, we evaluate across both Abstracted Labels and Refined Labels—representing different abstraction levels—to comprehensively assess the performance of SemanticAE methods under varying semantic granularity.

The reviewer’s suggestion, “Different abstraction levels should be discussed”, can be further addressed through finer-grained evaluation, which we support. We also agree that VQA provides a more general and flexible framework for multi-level evaluation, potentially extending to alignment studies. We present VQA results under different abstraction levels in our response to Q3.

Q2: We know that diffusion models can generate realistic images, but some of the details are unreal. Your approach utilizes the diffusion models, so how to guarantee the naturalness?

R2: While generating natural adversarial examples has been studied by many prior works, our core contribution lies not in naturalness, but in exploring and constructing adversarial examples in a broader semantic space, with a focus on key properties such as transferability and robustness. Hence, our title emphasizes Semantically Constrained Adversarial Examples, not Natural ones.

In our technology contribution, diffusion models are primarily used for on-manifold regularization to enhance transferability. Nevertheless, we do incorporate naturalness constraints by adapting existing techniques to ResAdv-DDIM. Specifically, we constrain the difference between non-adversarial and adversarial denoising steps during diffusion sampling (as shown in Eq. 7):

Both visualizations and quantitative results confirm that our generated samples exhibit sufficient naturalness.

Moreover, we argue that the loose realistic constraint does not challenge the significance of the study, specifically:

1. From an attack system perspective, if our method is used as a module in a real system, generated samples can be filtered using semantic plausibility checks. If both generator $A$ and discriminator $B$ (not under attack) deem an image $X$ semantically valid, we have strong justification to treat $X$ as natural. We formalize this in Appendix B.3 via a proposition linking our evaluation metric to practical utility. In real attack scenarios—whether physical attacks or jailbreaks—the targets are often automated systems; our evaluation aligns well with modeling such defenses. We further analyze attack performance under defenses in Appendix D.
2. For adversarial training, recent work [2] (ICML 2025, oral) shows that diffusion-generated samples—even with minor imperfections—can significantly improve training efficiency and model performance, enhancing scaling laws. Thus, minor realism flaws do not undermine their utility. We detailed this work in our response to Q3.

The realism of generated images is more central to diffusion model alignment research. Our framework may also inform Test-Time Scaling of Diffusion Model Alignment. While recent works[3] use reward models to refine generation, we explore the inverse: what optimization structures induce robustness failures in such reward models?

Q3: Is the semantic AE possible in the real world or does the semantic AE have real world application? I understand that the adversarial examples can be used for single object evaluation, but can you extend the approach for more complex scenarios in general VQA tasks, e.g., the image contains multiple objects? Can your adversarial examples be able to generate such adversarial images.

R3: Attributed to the technical improvement on the key problems, our framework has broad applications:

1. Adversarial training: Our framework can adapt to a concurrent ICML 2025 [2] work proposing diffusion-based data generation for training, which uses the target model $f_\phi$’s prediction entropy $H$ to guide diffusion sampling to generate challenging samples: $\textrm{Guidance Signal} \propto \omega \nabla_{x_t} H(f_\phi(\hat{x}_{0,t}))$ Our ResAdv-DDIM can enhance this pipeline by stabilizing multi-step adversarial directions through a refined estimation of $\hat{x}\_{0,t}$. This improves sampling accuracy and speed, while offering new insights into transferability. Since high-transferability adversarial examples expose non-robust features shared across models, they may offer advantages in adversarial training. We will include this discussion in the revised manuscript.

2. Jailbreak and physical attacks: We have already demonstrated effectiveness in cross-domain settings (2D → 3D). In complex systems, inaccurate adversarial feedback is a key challenge. The success of our method in 3D scenarios—where multi-view feedback is often incomplete—shows it can handle such uncertainty. We elaborate further in our response to Reviewer vVyk.

3. Extension to complex VQA tasks: Both the generator and the surrogate model are replaceable in our framework. Our 3D generation experiment validates the cross-task extension capability. For images with multiple objects, one can simply include multiple object descriptions in the diffusion model’s prompt. For applications in autonomous driving, our framework could guide the adversarial data generation techniques developed based on the scene generation diffusion models in the simulation pipeline.

We conducted additional VQA experiments using OpenAI CLIP models {RN50, ViT-B/32, RN50x16} (with DeCoWA) as surrogates, generating adversarial examples under three different abstraction levels. For each answer option, InSUR generates samples satisfying the corresponding semantic constraint $\mathcal{S}_\text{Text}$ ($\text{Text} \in \{\text{Options}\}$), using stabilityai/stable-diffusion-2-1-base. We evaluate transferability across OpenAI CLIP variants (RN50, RN101, RN50x4, RN50x16, ViT-B/32, ViT-B/16) and large VLMs (LLaVA, Qwen 7B). Results show:

• InSUR achieves strong cross-model transferability and enables weak surrogates to attack stronger models via diffusion assistance, though transfer weakens with greater semantic distance.
• Due to the widespread adoption of ViT architectures in modern VLMs, ViT-based surrogates show stronger transfer to LLaVA and Qwen.

We believe these findings are valuable for both alignment and super-alignment research. We will add the discussion to the camera-ready version.

Experimental Results:

Question: "What brand is this car?", Options: ["Audi", "Mercedes", "Ferrari", "Toyota"]

Accuracy(%) After Attack ($\downarrow$)

Question: "What is in the picture:", Options: ["Police car", "Ambulance", "Taxi", "School Bus"]

Accuracy(%) After Attack ($\downarrow$)

Question: "What is in the picture:", Options: ["Air Liner", "Car", "Helicopter", "Cruise Ship"]

Accuracy(%) After Attack ($\downarrow$)

Overall Accuracy After Attack(%)($\downarrow$)

[1] Wang, Jiakai, et al. "Generate transferable adversarial physical camouflages via triplet attention suppression." International Journal of Computer Vision 132.11 (2024): 5084-5100.

[2] Askari-Hemmat, Reyhane, et al. "Improving the Scaling Laws of Synthetic Data with Deliberate Practice." arXiv preprint arXiv:2502.15588 (2025).

[3] Jain, Vineet, et al. "Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models." arXiv preprint arXiv:2506.20701 (2025).

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to evaluating our work. We have carefully addressed all the comments and feedback provided in the review, and the corresponding responses are detailed in the rebuttal section. We hope that our clarifications and revisions have adequately resolved the concerns raised.

Given the approaching deadlines for the author-reviewer discussion, we would greatly appreciate your prompt feedback on any remaining questions or concerns. Timely communication will ensure we can effectively address all issues and improve the quality of our work.

Thank you once again for your valuable insights and guidance!

Best regards,
Submission 2210 Authors.