Semantic Representation Attack against Aligned Large Language Models

Dear Reviewer uz6Q,

Thank you for your positive feedback on our paper's presentation, theoretical analysis, and comprehensive experiments. We appreciate your thoughtful comments and your main concern regarding the significance of our work, which we will address below.

---

W1: My main and only concern is the significance of this paper. The LLMs used in the experiments has already been proven, by many previous studies, to be very fragile. I have conducted PAIR and AutoDAN on Llama2-7B and Vicuna-7B and -13B. This two methods reach >90% ASR. In fact, it is not hard to jailbreak these models even using DAN-like adversarial prompts. I suggest the authors to challenge (in a safe circumstances) the SOTA close-source LLM like GPT, Claude, and Deepseek for better evaluations.

A1: We appreciate the reviewer’s thoughtful concern regarding the significance of our work and the well-documented fragility of many open-source LLMs. We acknowledge that prior studies, including your experiments with PAIR and AutoDAN on Llama2-7B and Vicuna-7B/13B, have demonstrated high attack success rates (ASR) on these models. However, we believe that this observed fragility is partly a consequence of early evaluation methodologies that relied on simple keyword-based criteria to judge attack success. Such approaches may overestimate the vulnerability of LLMs by failing to capture the true semantic intent of model outputs.

To address this limitation, our work adopts HarmBench, a more advanced evaluation framework that is better aligned with human values, as evidenced by the agreement rates with human judgment shown in the table below.

We further highlight the gap (ASR) between traditional keyword-based evaluation (e.g., AdvBench) and human-value-aligned LLM assessment (HarmBench), demonstrating that the choice of evaluation framework significantly impacts the perceived robustness of LLMs.

In our experiments, we evaluate our method on 18 LLMs, including state-of-the-art models such as DeepSeek and Llama 3, to comprehensively demonstrate its effectiveness. Furthermore, we supplement transfer attack results on closed-source models like GPT-3.5 Turbo and GPT-4 Turbo, as shown in the table below, providing concrete evidence that our approach remains effective even in challenging black-box scenarios.

In summary, while we recognize the fragility of certain LLMs, our work advances the field by employing more rigorous, human-aligned evaluation criteria and demonstrating strong attack performance across a broad spectrum of both open- and closed-source models.

---

Q1&Q2: What is the transferability of the proposed method? As for the "unknown distribution" issue, it is commonly believed that the models within the same family with different scale (e.g., Qwen2.5-0.5B and Qwen2.5-70B) share a similar output distribution (The effectiveness of speculative inference has validated this belief in some sense). I am very curious about the transferability of the proposed method across such models. My second question, which might seem similar to my first one, would be: Could the authors please design and conduct an auxiliary experiment to validate my conjecture (i.e., the proposed method would transfer better within models in a same family)?

A1&A2: As the table below summarizes, we conducted experiments to directly test the reviewer's conjecture on transferability within and across model families. The results show that our method achieves substantially higher transfer attack success rates, and models with similar architectures and training data distributions exhibit greater transferability. Specifically:

(1) Transferring attacks from Vicuna 7B to Vicuna 13B yields 19.23% (BEAST) and 87.88% (Ours) ASR, while transferring from Guanaco 7B to Vicuna 13B yields 8.46% (BEAST) and 79.62% (Ours).

(2) Transferring attacks from Vicuna 13B to Vicuna 7B achieves 49.81% (BEAST) and 94.42% (Ours) ASR, whereas transferring from Guanaco 7B to Vicuna 7B achieves 28.08% (BEAST) and 91.73% (Ours).

These findings support the hypothesis that models with similar architectures and training data distributions exhibit greater transferability. However, we note an important caveat: for a fair comparison, the models being compared should ideally possess similar levels of adversarial robustness. In practice, it is challenging to quantify or ensure that any two LLMs—even within the same family—exhibit identical robustness to adversarial prompts. This limitation should be carefully considered when interpreting transferability results across models, both within and across families.

---

Once again, we thank you for your valuable feedback. We believe that by clarifying these points and conducting the suggested experiment, we can better highlight the significance and contribution of our work.

Sincerely,

The Authors

Reference

[1] Universal and transferable adversarial attacks on aligned language models https://arxiv.org/pdf/2307.15043

[2] Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts https://arxiv.org/pdf/2309.10253

[3] "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models https://arxiv.org/abs/2308.03825

[4] Purple llama cyberseceval: A secure coding benchmark for language models https://arxiv.org/pdf/2312.04724

[5] Jailbreaking black box large language models in twenty queries https://arxiv.org/abs/2310.08419

[6] Harmbench: A standardized evaluation framework for automated red teaming and robust refusal https://arxiv.org/pdf/2402.04249

Dear Reviewer uz6Q,

Thank you sincerely for your Mandatory Acknowledgement.

We would be most grateful if you could kindly share your evaluation comments when convenient—whether our responses have sufficiently addressed your concerns or if any aspects would benefit from further clarification. Your thoughtful feedback is highly valued and essential to ensure a fair and thorough review process.

Thank you very much for your time, expertise, and guidance.

Best regards,
The Authors

Dear Reviewer ETbV,

Thank you for your exceptionally thorough and positive review. We are delighted you found our work technically solid, rigorous, and impactful. Your detailed questions are highly insightful, and we appreciate the opportunity to provide further clarification and detail on our work.

---

W1-1&Q1: White‑box assumption: SRHS needs token probabilities; closed models (e.g. ChatGPT) are not tested.

A1-1: We appreciate the reviewer’s thoughtful concern regarding the white-box assumption. Our method does require access to token probabilities. To address this, we leverage proxy models like DeepSeek and Llama 3 to perform a semantic representation attack. Our experimental results show that these prompts can be directly transferred to black-box models, achieving state-of-the-art attack success rates without access to internal probabilities. This transferability underscores the practical risk posed by our method in real-world black-box scenarios.

---

W1-2&Q2: Semantic classifier dependency: success hinges on the HarmBench‑fine‑tuned Llama‑2‑13B; generality to other semantics detectors is untested.

A1-2: We acknowledge the reviewer’s concern regarding our reliance on the HarmBench-fine-tuned Llama-2-13B [6] as the semantic classifier. We selected this model because, as shown in the table below, it achieves the highest agreement with human judgments among all evaluated frameworks, ensuring reliable and valid semantic equivalence assessments. Furthermore, the high transferability of our adversarial prompts across diverse model families suggests that our approach captures fundamental semantic properties rather than overfitting to a specific classifier. We agree that exploring alternative semantic detectors is a valuable direction for future work and have clarified this in the revised manuscript.

---

W2: Dense notation and long proofs re‑state obvious steps; could be shortened. The link between Theorem 3.1 and practical pruning thresholds (τ) is buried in appendix.

A2: We appreciate the reviewer’s feedback on the density of notation and the length of the proofs. We agree that some steps in the theoretical analysis could be presented more concisely and have revised the main text to streamline the exposition. Additionally, we acknowledge that the connection between Theorem 3.1 and the practical choice of pruning thresholds (τ) was not sufficiently emphasized. In the revised manuscript, we have explicitly clarified how Theorem 3.1 provides a theoretical basis for setting τ.

---

W3: Impact on real‑world closed‑source LLMs is speculative; without such evidence, practical risk remains uncertain.

A3: We appreciate the reviewer’s concern about the practical risk to real-world closed-source LLMs. As detailed in our response to W1-1 (A1-1), we have demonstrated that adversarial prompts generated via proxy models can be effectively transferred to closed-source models (e.g., GPT-3.5/4 Turbo), achieving SOTA attack success rates. This provides concrete evidence of the practical risk and real-world impact of our method.

---

W4: Use of perplexity as a coherence proxy is known; the novelty lies mainly in combining it with semantic‑class targeting.

A4: We agree and appreciate the reviewer’s recognition that our main novelty is the integration of semantic-class targeting with perplexity-based filtering.

---

Q3: Perplexity threshold τ – Table 3 shows τ = 20 performs well, but τ must be chosen per model. Can you propose a heuristic for selecting τ without manual sweeps?

A3: We propose a heuristic search strategy for the perplexity threshold τ: starting from an initial τ, we iteratively adjust τ based on attack performance on a small validation set, using the observed success rate as a heuristic function. This process automatically guides the search toward an optimal τ, efficiently balancing attack efficacy and training stability without exhaustive manual tuning.

---

Q4: Prompt length vs. ASR – What is the average token length of successful prompts, and does further shortening sacrifice ASR? This matters for detectability.

A4: We report the average token length (with standard deviation) of 300 successful prompts (100% ASR) in the table below. Our method incrementally searches for candidate tokens based on the original malicious prompt, resulting in significantly shorter prompts than prior methods. This compactness highlights both the efficiency and effectiveness of our approach.

---

Q5: Defence suggestions – Beyond lowering perplexity thresholds, which defences could specifically break semantic‑representation convergence? Clarifying this would aid the safety community and could influence my significance score.

A5: We recommend a multi-dimensional defense strategy that combines the strengths of different approaches to effectively mitigate semantic-representation-based jailbreak attacks. (1) At the system level, adversarial prompt purification techniques could be applied before inputs reach the LLM. (2) At the model level, further improving the model’s ability to distinguish harmful from benign content—by aligning more closely with human values—remains essential. (3) Additionally, output filtering mechanisms could be employed to detect and block harmful responses at the system level. Integrating these system-level and model-level defenses can provide robust protection against advanced jailbreak attacks.

---

Q6: Criteria for score change: Demonstrating SRHS effectiveness under black‑box settings or showing robustness across different semantic classifiers would raise quality and significance to excellent.

A6: Thank you for outlining these important criteria. As discussed in our responses to A1-1 and A1-2, we have provided empirical evidence demonstrating our method’s effectiveness in black-box settings and the advantages of our chosen semantic classifiers. We hope these results directly address your concerns and further strengthen the quality and significance of our work.

---

Once again, thank you for your constructive feedback and for helping us improve the paper.

Sincerely,

The Authors

Reference

[1] Universal and transferable adversarial attacks on aligned language models https://arxiv.org/pdf/2307.15043

[2] Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts https://arxiv.org/pdf/2309.10253

[3] "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models https://arxiv.org/abs/2308.03825

[4] Purple llama cyberseceval: A secure coding benchmark for language models https://arxiv.org/pdf/2312.04724

[5] Jailbreaking black box large language models in twenty queries https://arxiv.org/abs/2310.08419

[6] Harmbench: A standardized evaluation framework for automated red teaming and robust refusal https://arxiv.org/pdf/2402.04249

[7] Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks https://arxiv.org/abs/2404.02151

[8] AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models https://arxiv.org/abs/2310.04451

Dear Reviewer ETbV,

Thank you very much for submitting the Mandatory Acknowledgement.

We would be truly grateful if you could kindly share your evaluation comments—whether our responses have adequately addressed your concerns, or if any points require further clarification. Your thoughtful feedback is invaluable to us and to the integrity of the review process.

Thank you sincerely for your time and guidance.

Best regards,

The Authors

Dear Reviewer 6jQY,

Thank you for your positive assessment and insightful feedback on our paper. We are encouraged that you found our work a novel advancement in adversarial NLP with strong empirical results and high efficiency. We appreciate the opportunity to address the weaknesses you've identified.

---

W1: The attack assumes access to model logits/probabilities, which is infeasible for most closed-source or API-only LLMs.

A1: Thank you for highlighting this important limitation. In our new transfer attack experiments, we generate adversarial prompts using our semantic representation attack on proxy models (e.g., DeepSeek, Llama 3) and apply them directly to closed-source/API-only LLMs. As shown in the table below, our method achieves strong attack success rates (ASR) on both GPT-3.5 Turbo and GPT-4 Turbo, even without access to their logits or internal probabilities. This demonstrates that our approach is not only effective in white-box settings but also highly transferable to black-box scenarios—a key strength that reveals the vulnerability of SOTA LLMs to semantic representation-based attacks, even with API-only access. We have explicitly discussed these findings and their implications in the revised manuscript.

This high transferability stems from our focus on optimizing the semantic representation space rather than specific textual patterns, making our adversarial prompts more robust and generalizable across model architectures. We further believe that incorporating closed-source/API-only LLMs into the optimization pipeline via proxy models could further enhance attack performance by better tailoring prompts to the target models’ internal representations.

---

W2: Requires defining target semantic representations, which could be non-trivial or subjective in complex or ambiguous tasks.

A2: Thank you for raising this important issue. A semantic representation is an abstract, language-independent meaning that can be expressed by multiple surface forms with equivalent propositional content. Specifically, for a given harmful query, we identify the set of responses that share the same semantic representation—i.e., they fulfill the malicious intent regardless of lexical or syntactic variation. Formally, as described in our methodology, we denote the semantic representation space as $\Omega$, and for each target harmful behavior, we define a representation $\Phi \in \Omega$ such that the set of compliant responses $\mathcal{Y}_\Phi$ all satisfy $\mathcal{R}(\boldsymbol{y}) = \Phi$. This approach allows us to optimize for a broad equivalence class of harmful outputs, rather than a single fixed response, making the attack both more general and more robust. We have clarified this definition and its practical implementation in the revised manuscript, and discuss the challenges and future directions for defining semantic targets in more complex or ambiguous scenarios.

---

W3: Relies on external models (e.g., fine-tuned Llama) to define semantic equivalence, which may vary in accuracy across tasks or domains.

A3: We appreciate your concern about relying on an external model (e.g., fine-tuned Llama) to define semantic equivalence, as its accuracy may vary across tasks or domains. We selected this model because, as shown in the table below, it achieves superior alignment with human values compared to other evaluation frameworks. This strong alignment ensures that the semantic representations used in our attack are consistent with human judgment, increasing the reliability and validity of our results. Additionally, the high transferability of our prompts across diverse model families suggests that our approach captures fundamental semantic properties rather than overfitting to the guidance model. We have clarified this rationale and its implications in the revised manuscript.

---

Once again, thank you for your valuable feedback and for recognizing the potential of our work. We are confident that the paper has been significantly strengthened by addressing these points.

Sincerely,

The Authors

Reference

[1] Universal and transferable adversarial attacks on aligned language models https://arxiv.org/pdf/2307.15043

[2] Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts https://arxiv.org/pdf/2309.10253

[3] "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models https://arxiv.org/abs/2308.03825

[4] Purple llama cyberseceval: A secure coding benchmark for language models https://arxiv.org/pdf/2312.04724

[5] Jailbreaking black box large language models in twenty queries https://arxiv.org/abs/2310.08419

[6] Harmbench: A standardized evaluation framework for automated red teaming and robust refusal https://arxiv.org/pdf/2402.04249

Dear Reviewer oJrc,

Thank you for your detailed review and constructive feedback. We appreciate your recognition of our methodological rigor, comprehensive evaluation across 18 LLMs, and strong results. We also value your comments on the paper’s weaknesses, which we have addressed point by point below.

---

W1: Lack of novelty. There are multiple jailbreaking methods [1, 2, 3] currently in existence that manipulate prompts and retain semantic coherence. A Comparison with them and an explanation in the related work section will be helpful. The paper could benefit from a more thorough literature survey.

A1: We thank you for your insightful feedback and for highlighting these important references. We agree that a direct comparison is essential and have updated the "Related Work" section and experiments to clarify our contributions.

While previous methods generate semantically coherent prompts by optimizing for specific textual patterns (e.g., "Sure, here is..."), our Semantic Representation Attack (SRA) fundamentally differs by directly optimizing for the target semantic representation of harmful responses. This strategy overcomes the trade-off between prompt naturalness and attack efficacy that constrains prior work. Extensive experiments show that SRA achieves superior attack efficacy, transferability, naturalness, and efficiency compared to existing methods.

Specifically, our work differs from [1] in three fundamental ways:

1. Optimization Goals: Method [1] optimizes for specific textual patterns (e.g., "Sure, here is...") as its objective. In contrast, our approach directly targets the underlying malicious semantic representation. By optimizing for a broad class of semantically equivalent outputs rather than narrow surface forms, our method overcomes the trade-off between attack efficacy and prompt naturalness that limits previous work.

2. Prompt Generation Framework: Method [1] relies on a manual template with random search suffixes, resulting in long prompts (~480 tokens), while our method is fully automatic and produces much shorter prompts (<10 tokens), as summarized in the table below.

3. Evaluation Framework: Method [1] evaluates attack success using a vanilla LLM (GPT-4) [9] and a rule-based keyword list [5]. In contrast, we use the more advanced HarmBench framework [10], which achieves significantly higher agreement with human judgments on harmfulness, as shown in the table below.

TAP [2] employs an attacker LLM to iteratively refine candidate prompts until one successfully jailbreaks the target model or the maximum tree depth is reached. The comparative experimental results are shown below.

Work [3] shows that aligned LLMs can be vulnerable to malicious questions generated by other aligned LLMs when given a malicious response, forming a "Question-Answer" and "Answer-Question" loop. However, these prompts are generated without attack optimization, which fundamentally distinguishes our approach.

---

W2: Needs more Evaluations in the presence of defenses. The paper does not evaluate the jailbreaking capabilities in the presence of defenses other than PPL-based defenses, which is necessary to understand the robustness. Defenses like [4] could be effective against this attack.

A2: We fully agree that evaluating our attack against a broader range of defenses is essential. Thank you for recommending work [4] as a relevant defense. To address this, we selected three strong baseline defenses from [4] and conducted adaptive attack evaluations on the same dataset. The results (ASR) show that our method consistently identifies effective adversarial prompts in the semantic representation space, even when prompts are perturbed:

These results are notable. While these defenses substantially reduce the ASR of strong baselines like PAIR and AutoDAN (by up to 34 points), our attack remains highly effective—and in some cases, ASR even increases under these defenses. This counterintuitive outcome arises because (1) prior methods rely on specific keywords, making them vulnerable to prompt perturbations, whereas our approach targets the underlying semantic space; and (2) perturbed prompts may fall outside the distribution of safety alignment training, diminishing the effectiveness of model-level defenses.

---

W3: Perplexity as a measure of coherence. The paper extensively uses perplexity as a measure of coherence. Log-Perplexity, calculating the certainty of occurrence of words in the context, does not directly capture the coherence/naturalness of the text. Moreover, the algorithm itself uses perplexity to generate the jailbreaks; hence, the same measure should not be used as a metric.

A3: We appreciate your thoughtful comments on the use of perplexity as a metric. While perplexity mainly measures grammatical fluency rather than true semantic coherence, it remains a widely used and reasonable proxy for naturalness in adversarial text generation [11,12], as it quantifies how "expected" or "natural" text appears to a language model.

You also raise an important point about the potential issue of using perplexity both as an optimization objective and an evaluation metric. Prior methods often struggle with a trade-off between low perplexity (natural prompts) and high attack success. In contrast, our semantic representation-based approach overcomes this limitation, achieving both high attack success rates and low perplexity, thus generating natural-looking prompts without sacrificing efficacy.

To further address your concern, we conducted a human evaluation of 300 generated prompts across three representative LLMs. Independent evaluators rated the naturalness of these prompts, and the results (all >95%) confirm that our method consistently produces human-like, coherent text while maintaining strong attack performance.

---

Once again, thank you for your invaluable feedback, which has been crucial to improving our paper. We hope our responses address your concerns and are confident that these revisions make our work more compelling and its contributions clearer.

Sincerely,

The Authors

Reference

[1] Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks https://arxiv.org/abs/2404.02151

[2] Tree of Attacks: Jailbreaking Black-Box LLMs Automatically https://arxiv.org/abs/2312.02119

[3] Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? https://arxiv.org/abs/2412.03235

[4] Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing https://arxiv.org/abs/2402.16192

[5] Universal and transferable adversarial attacks on aligned language models https://arxiv.org/pdf/2307.15043

[6] Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts https://arxiv.org/pdf/2309.10253

[7] "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models https://arxiv.org/abs/2308.03825

[8] Purple llama cyberseceval: A secure coding benchmark for language models https://arxiv.org/pdf/2312.04724

[9] Jailbreaking black box large language models in twenty queries https://arxiv.org/abs/2310.08419

[10] Harmbench: A standardized evaluation framework for automated red teaming and robust refusal https://arxiv.org/pdf/2402.04249

[11] AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models https://arxiv.org/abs/2310.04451

[12] Fast Adversarial Attacks on Language Models In One GPU Minute https://arxiv.org/abs/2402.15570

Dear Reviewer oJrc,

Thank you for your feedback and for confirming that W2 and W3 are resolved. We apologize for the earlier misunderstanding regarding W1 and appreciate the opportunity to clarify the novelty of our work. We have incorporated these clarifications into the revised manuscript.

We agree with the reviewer that a comprehensive comparison with recent related work is essential to situate our contributions. To clarify the novelty of our Semantic Representation Attack (SRA), we first categorize prior jailbreaking methods based on their core optimization objective. As shown in our comparison table, existing approaches largely fall into three groups: (1) those that optimize for specific textual patterns (e.g., "Sure, here is..."), (2) those that rely on manually designed prompts, and (3) those that optimize for a score from a judge LLM.

Our approach differs significantly from all three. Unlike methods targeting textual patterns, we directly optimize for the target semantic representation of harmful responses. This avoids constraining the model to a narrow surface form and overcomes the trade-off between prompt naturalness and attack efficacy that limits prior work. Compared to manual methods, our approach is fully automated and more generalizable. Finally, while methods like TAP and PAIR optimize for a judge's score, our SRA targets the underlying semantic space directly, which is a more fundamental and robust objective than a single scalar score. This conceptual shift allows SRA to achieve superior attack efficacy, transferability, naturalness, and efficiency, as demonstrated in our experiments.

This clarification has been included in the revised version. We appreciate your understanding regarding the current system limitations for uploading the revision.

We look forward to your response.

Best regards,

The Authors

References

[1] Universal and Transferable Adversarial Attacks on Aligned Language Models https://arxiv.org/pdf/2307.15043

[2] AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models https://arxiv.org/pdf/2310.04451

[3] Fast Adversarial Attacks on Language Models In One GPU Minute https://arxiv.org/pdf/2402.15570

[4] Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks https://arxiv.org/abs/2404.02151

[5] AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs https://arxiv.org/abs/2410.05295

[6] BadRobot: Jailbreaking LLM-based Embodied AI in the Physical World https://arxiv.org/abs/2407.20242

[7] Does Refusal Training in LLMs Generalize to the Past Tense? https://arxiv.org/abs/2407.11969

[8] Tree of Attacks: Jailbreaking Black-Box LLMs Automatically https://arxiv.org/abs/2312.02119

[9] Jailbreaking Black Box Large Language Models in Twenty Queries https://arxiv.org/abs/2310.08419

[10] Red Teaming Language Models with Language Models https://arxiv.org/pdf/2202.03286

[11] Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? https://arxiv.org/abs/2412.03235