We thank Reviewer MajM for their thoughtful feedback and for highlighting the clarity of our work. We are especially grateful for the recognition that “the authors did a good job at connecting it to traditional adversarial attack formulations in computer vision,” as this connection was central to our motivation. We also appreciate the acknowledgment of our human study. Please find our detailed responses to the insightful questions below.

Q1: "The experiment is only on multi-choice problems, whereas in practice open-ended problems are more common."

A1: First, we clarify that our setting is already an open-ended MCQA generation task, where the model is prompted to select an incorrect answer choice followed by a hallucinated open-ended explanation.

Next, we designed our problem along two axes of difficulty: (1) open-ended MCQA vs. free-form open-ended generation, and (2) constrained vs. unconstrained optimization. While existing work may tackle unconstrained attacks in more complex free-form tasks, we emphasize that our current constrained optimization setting for open-ended MCQA generation task is already highly challenging and underexplored. No existing attack method is capable of operating under these constraints, making our formulation a meaningful and non-trivial benchmark.

Our choice of MMLU is well-justified for the following reasons:

• Hallucination attacks versus Jailbreaking: This work focuses on eliciting hallucinations from LLMs, rather than bypassing safety mechanisms (jailbreaking). While our method can be viewed as an adversarial attack, we emphasize throughout the paper that our goal is fundamentally different from jailbreaking. Accordingly, we choose MMLU to better evaluate hallucination behaviors in LLMs.

• Novel constrained optimization setting: To the best of our knowledge, we are the first to explicitly formulate hallucination attacks under semantic equivalence and coherence constraints (see Equation (2), Line 138). This presents a significant challenge, as existing SOTA attack methods are not designed to handle such constraints (see Section 2 and our response to Q1 from Reviewer 6rsw for further discussion). MMLU offers a clean and structured evaluation space where semantic equivalence is critical and constraint violations are easily observable. Even within the MMLU task, it was previously unclear whether generating semantically equivalent and linguistically coherent adversarial prompts under constrained optimization was feasible—highlighting the novelty of our setting.

Given these reasons, we focus on open-ended MCQA generation tasks rather than free-form open-ended generation. We agree that extending our method to free-form generation, potentially by optimizing over entire responses rather than token-level objectives, is a promising direction for future work.

Q2: "The prompt template set up can be problematic: In the example in figure 5 (Appendix B), it seems that the LLM is prompted to first give an answer (from "A", "B", "C", "D") and then explain the reasoning. However, in practice, it is more reasonable to first reason and then give the final answer. It is unclear whether the method can still work under this more practical setting."; "Does the method still work in a setting where the LLM is prompted to first reason through the question before selecting a multiple-choice answer?"

A2: Answer-first settings are common in adversarial attacks in LLMs. In optimization-based attacks such as Greedy Coordinate Gradient (GCG), it is standard to prompt the model to begin with a target string, e.g., “Sure, here is how to make a bomb”, so that the log-likelihood objective can be defined over the initial confirmative tokens, which are more likely to induce harmful continuations. Such methods do not easily extend to reasoning-first settings, where objective formulation is challenging. Inspired by this, we design our hallucination targets to begin with one of ‘A’, ‘B’, ‘C’, or ‘D’, which effectively elicits follow-up hallucinated responses, as demonstrated in our paper. While we also experimented with reasoning-first prompts, their unpredictable reasoning length makes it difficult to determine where the answer choice will appear, complicating the definition of a principled objective (e.g., log-likelihood over a fixed span).

Besides that, we aim to highlight the distinction between LLM explanations and reasoning. As shown in Figure 5 of Appendix B, our intention is to illustrate how the LLM explains its predicted answer post hoc. In contrast, reasoning refers to an inference-time computation strategy intended to guide or improve the model's prediction. While we agree that extending our method to reasoning-first prompting is a valuable direction for future work, we emphasize that the central research problem we are addressing is how to generate semantically equivalent and coherent prompts to elicit hallucinations. While reasoning-first prompts might be more resistant to SECA prompts, this is an open problem that we reserve for future work.

To date, no existing attack method is capable of generating semantically equivalent and coherent adversarial prompts, even for the answer-first prompting. This makes our current answer-first setting both meaningful and non-trivial. In fact, prior to our work, it was unclear whether such attacks were feasible at all under constrained optimization, even in the current setting. We therefore view our work as a crucial first step in establishing the feasibility and methodology for constrained hallucination attacks. Once this foundation is solid, extending to reasoning-first prompting or even reasoning models becomes a natural and promising direction for future research.

We also note that in adversarial machine learning, it is common for early attack methods to be challenged by subsequent defense mechanisms. For example, the well-known Greedy Coordinate Gradient (GCG) attack was later shown to be vulnerable to techniques such as randomized smoothing. Similarly, reasoning-first prompting and reasoning models may act as implicit defenses against hallucination. However, this does not diminish the value or strength of our proposed attack. Rather, it highlights the importance of establishing strong baselines and well-defined constrained settings before evaluating robustness under more complex model behaviors.

Q3: "The main weakness is the over-simplified experimental setting."

A3: We respectfully disagree with this statement.

As discussed in our responses to Q1 and Q2, the proposed constrained optimization framework, applied even to open-ended MCQA generation tasks with an answer-option-first prompting format, is already highly challenging and underexplored. Prior to our work, it was unclear whether one could generate semantically equivalent and linguistically coherent adversarial prompts under such constraints. Our setting is both novel and non-trivial, as acknowledged by Reviewers LvAw and 6rsw. We believe it deserves focused investigation before extending to more complex scenarios such as open-ended generation or reasoning-augmented prompting/models.

We also note that in adversarial machine learning, it is common for early attack methodologies to begin with simplified but principled settings. For example, the well-known Greedy Coordinate Gradient (GCG) attack uses prompts like "Sure, here is how to make a bomb" as optimization targets—despite the fact that, in real-world scenarios, models often instruct to respond with intermediate reasoning before producing a final answer. Nevertheless, no jailbreak attack against reasoning models or multi-step prompting can exist without first establishing effective methods under more controlled settings.

Similarly, in adversarial attack in computer vision, the foundational works on $\ell_p$​-norm attacks (e.g., FGSM, PGD) helped define robustness evaluation benchmarks, even though such perturbations do not fully align with human perceptual similarity. These simplified attack settings were important first steps that later led to more advanced attacks better aligned with human perception like perceptual attack.

In this context, we view our constrained and well-defined setting as a necessary and meaningful first step toward building rigorous and extensible adversarial benchmarks for LLMs.

We sincerely thank the reviewer again for their valuable feedback and hope that our clarifications help reinforce the contributions of our work.

We thank Reviewer LvAw for recognizing the quality, clarity, significance, and originality of our work. We sincerely appreciate the reviewer’s acknowledgment that our method is novel, simple, and effective; that our empirical evaluation is comprehensive; and that the paper is well-written, well-organized, and easy to reproduce. Please find our detailed responses to the insightful questions below.

Q1: "The method is only evaluated in the multiple-choice QA setting (MMLU). It remains unclear how well SECA generalizes to open-ended generation tasks such as summarization or dialogue."

A1: First, we clarify that our setting is already an open-ended MCQA generation task, where the model is prompted to select an incorrect answer choice followed by a hallucinated open-ended explanation.

Next, we designed our problem along two axes of difficulty: (1) open-ended MCQA vs. free-form open-ended generation, and (2) constrained vs. unconstrained optimization. While existing work may tackle unconstrained attacks in more complex free-form tasks, we emphasize that our current constrained optimization setting for open-ended MCQA generation task is already highly challenging and underexplored. No existing attack method is capable of operating under these constraints, making our formulation a meaningful and non-trivial benchmark.

Our choice of MMLU is well-justified for the following reasons:

• Hallucination attacks versus Jailbreaking: This work focuses on eliciting hallucinations from LLMs, rather than bypassing safety mechanisms (jailbreaking). While our method can be viewed as an adversarial attack, we emphasize throughout the paper that our goal is fundamentally different from jailbreaking. Accordingly, we choose MMLU to better evaluate hallucination behaviors in LLMs.

• Novel constrained optimization setting: To the best of our knowledge, we are the first to explicitly formulate hallucination attacks under semantic equivalence and coherence constraints (see Equation (2), Line 138). This presents a significant challenge, as existing SOTA attack methods are not designed to handle such constraints (see Section 2 and our response to Q1 from Reviewer 6rsw for further discussion). MMLU offers a clean and structured evaluation space where semantic equivalence is critical and constraint violations are easily observable. Even within the MMLU task, it was previously unclear whether generating semantically equivalent and linguistically coherent adversarial prompts under constrained optimization was feasible—highlighting the novelty of our setting.

Given these reasons, we focus on open-ended MCQA generation tasks rather than free-form open-ended generation. We agree that extending our method to free-form generation, potentially by optimizing over entire responses rather than token-level objectives, is a promising direction for future work.

Q2: "While SECA is effective in eliciting hallucinations, the paper lacks analysis on whether models trained to resist SECA prompts would generalize to other types of realistic adversarial inputs."

A2: While we agree that leveraging SECA prompts to improve the robustness of existing LLMs is an important and promising direction, we emphasize training LLM on SECA prompts is not the central research problem in our paper.

Based on our understanding and preliminary assessment, we believe that fine-tuning LLMs on SECA prompts can enhance robustness against certain realistic adversarial inputs, such as jailbreaks involving paraphrasing or stylistic modifications or backdoor attacks using paraphrasing-based triggers. However, as noted in Q1, our proposed constrained optimization framework is novel and merits a focused study. We thus leave the exploration of improving model robustness through training/finetuning as an exciting direction for future work.

Q3: "Can the proposed method be applied to other types attacks against LLMs (e.g. adversarial attacks, backdoor attacks)?"

A3: While the primary goal of our work is to generate semantically equivalent and coherent adversarial prompts to elicit hallucinations from LLMs—rather than to bypass safety mechanisms—the answer is yes: SECA can be viewed as a general framework that may be extended to other types of attacks on LLMs.

For example, in jailbreaking or backdoor attacks, one could similarly impose semantic equivalence and coherence constraints to craft stealthier adversarial prompts. A prompt such as “How to build a bomb?” might be rephrased as “How does one go about assembling an explosive device?”, retaining malicious intent while potentially bypassing safety filters. This would effectively constitute a semantically equivalent jailbreak or backdoor trigger.

However, we do not recommend using SECA for such tasks, for the following reasons:

• In jailbreaking, the objective is to circumvent safety mechanisms. Arbitrary prompts, including intent-shifting, storytelling, or even gibberish, are acceptable. Imposing semantic constraints is unnecessary and often counterproductive.

• In backdoor attacks, the goal is typically to embed a trigger that activates harmful behavior under specific conditions. Semantic equivalence to a base prompt is not required, so enforcing equivalence adds unnecessary overhead.

• In hallucination attacks, by contrast, the goal is to induce hallucinations without changing the task intent. Semantic equivalence is therefore essential to ensure the adversarial prompt remains a faithful and meaningful variant of the original query.

  To draw an analogy from adversarial attacks in computer vision: if the goal is to fool a classifier into misclassifying a cat as a dog, the perturbation must be small enough that the image still looks like a cat to a human. Allowing arbitrary changes (e.g., turning a cat into a dog) trivializes the attack, as one could simply change the image into a dog. Similarly, hallucination attacks without semantic equivalence constraints either become trivial or cause intent shift (see Table 1 in the paper and our response to Q1 from Reviewer 6rsw).

Conclusion: While SECA is broadly applicable, we believe its strength lies in hallucination attacks where semantic equivalence constraints are both meaningful and necessary. Thus, we recommend applying SECA primarily to settings where preserving task intent is essential.

We sincerely thank the reviewer again for their valuable feedback and hope that our clarifications help to further highlight the contributions of our work.

We thank Reviewer 6rsw for recognizing the quality, clarity, significance, and originality of our work. We appreciate the acknowledgment that our method sheds light on the reasoning capabilities of LLMs, suggesting that their behavior may be driven more by grammatical and syntactic patterns than by true semantic understanding. We also thank the reviewer for highlighting the effectiveness and efficiency of our proposed approach. Please find our detailed responses to the insightful questions below.

Q1: Evaluation is only against Raw and GCG. Compare with other jailbreaking methods. Explain "if there any other experiments comparing with different methods?"

A1: The algorithms you mentioned are indeed interesting; we will include appropriate citations, and they may help further improve the efficiency of our SECA method. We also examined several prompts generated by existing attack methods and found that, although they can generate successful adversarial prompts, none were able to produce semantically equivalent variants of the original prompts. This indicates that these methods cannot find feasible solutions under our newly proposed constrained optimization formulation (Equation (2), Line 138). We provide several illustrative examples below as an extended version of Table 1 in our paper.

Note: The COLD attack may produce prompts that are semantically similar but not equivalent (see Section 2 in our paper for a discussion on semantic similarity vs. equivalence). The example prompt appears topically related but leads to a different solution, constituting an intent shift attack

We also hope to emphasize that it is essential for distinguishing jailbreaking and hallucination attacks:

• In jailbreaking, the goal is to bypass safety mechanisms. Arbitrary prompts—such as intent-hiding, storytelling, or gibberish—are acceptable.

• In hallucination attacks, the goal is to induce factual errors while preserving the original task intent. Therefore, enforcing semantic equivalence is essential to ensure meaningful attacks.

  To draw an analogy from adversarial attacks in computer vision: if the goal is to fool a classifier into misclassifying a cat as a dog, the perturbation must be small enough that the image still looks like a cat to a human. Allowing arbitrary changes (e.g., turning a cat into a dog) trivializes the attack, as one could simply change the image into a dog. Similarly, hallucination attacks that do not enforce semantic constraints either become trivial or result in intent shift (see Table 1 in the paper and the table above in our response).

  Since none of the existing attack methods can handle our constrained formulation, we compare only with the most representative prior work, GCG—which, however, also leads to infeasible solutions.

Q2: Experiments on larger models

A2: Please see the tables below for new results on larger models. SECA remains effective on larger models, showing strong performance relative to raw ASR.

ASR@30/10/1 (Raw)

ASR@30/10/1 (SECA)

Objective Difference (SECA increases the objective significantly compared to Raw)

Note: Due to time constraints, GPT-3.5-Turbo (gpt-3.5-turbo-0125) and GPT-4 (gpt-4-0613) were evaluated on a 30% subset.

Q3: "The evaluation requires more explanation, as the only reported metrics are new "; "Could authors explain if there any metrics relevant to the use"

A3: As mentioned in Section 4.1, the metrics used in our paper include: Best-of-K Attack Success Rate (ASR@K), constraint violations $v_{SE}$ (semantic equivalence) and $v_C$ (coherence), and Type Token Ratio (TTR). ASR@K and TTR are standard metrics widely adopted in the trustworthy LLM community.

The main nonstandard metrics are the constraint violations $v_{SE}$ and $v_C$. This is because, to the best of our knowledge, we are the first to introduce the constrained optimization formulation for hallucination attacks (see Equation (2), Line 138), where the goal is to generate semantically equivalent and coherent adversarial prompts that induce hallucinations in LLMs. As a result, it is necessary to evaluate whether the generated prompts violate these constraints—hence the use of $v_{SE}$ and $v_C$.

While such constraint-based metrics may be uncommon in prior LLM adversarial attack literature, they are well-established in constrained optimization and adversarial attacks in computer vision—such as $\ell_p$-bounded or perceptual attacks. This justifies their adoption in our work.

Q4: Experiments on reasoning models/chain-of-thought methods

A4: While extending to open-ended QA and reasoning models is an interesting direction for future work, we emphasize that our current constrained optimization setting—despite not involving reasoning models—is already highly challenging and underexplored. To the best of our knowledge, no existing attack method can effectively operate under the these constraints, making our setting a meaningful and non-trivial benchmark. Prior to our work, it was unclear whether such attacks were feasible at all under constrained optimization.

We view our work as a crucial first step toward principled constrained hallucination attacks. Once this foundation is established, extending to reasoning models and chain-of-thought prompting becomes a natural next step.

As in adversarial ML, early attacks often precede the development of stronger defenses. E.g., Greedy Coordinate Gradient (GCG) attack was later shown to be vulnerable to techniques such as randomized smoothing. Similarly, reasoning models/chain-of-thought methods may eventually serve as implicit defenses, but this only underscores the need for strong, well-defined attack baselines like ours.

Q5: "How does the framework mitigate hallucinations created by the auxiliary LLMs?"

A5: While we agree that any LLMs, including our auxiliary models, may introduce hallucinations, we have performed additional experiments to demonstrate that it does not happen frequently and invalidate our results. To this end, we designed a two-stage verification process, as described in Lines 6–7 of Algorithm 1.

1. Semantic Equivalence Proposal:
   We first use an efficient and cost-effective model (GPT-4.1-nano) to generate semantically equivalent candidate prompts. Empirically, this step yields approximately 70% strictly semantically equivalent prompts.

2. Feasibility Verification:
   To further reduce hallucinations, we verify each candidate using a stronger feasibility checker LLM (GPT-4.1-mini). This model verifies whether the proposed prompt is semantically equivalent to the original. As shown in Table 3 in the paper, this verifier achieves 80–85% accuracy in identifying equivalence.

By evaluating each candidate twice—once for generation and once for verification—we significantly reduce hallucination risks introduced by the auxiliary LLMs.

3. Final Hallucination Evaluation:
   For measuring hallucination in our results, we use GPT-4.1, among the best publicly available models at the time of submission. According to Table 3, it achieves 90–95% accuracy in hallucination detection, providing a strong upper bound on evaluation reliability.

We sincerely thank the reviewer again for their valuable feedback and hope that our clarifications help reinforce the contributions of our work.

Thank you to the authors for answering my questions. Most of my concerns have been resolved, however I feel that the Chain-of-Thought or reasoning methodologies evaluation could be critical for the novelty of the work, given that most of the research area is headed towards LRMs, which there are certain of them that are also open source right now (e.g. OpenThinker). I will retain my rating.

We thank Reviewer ibLV for their thoughtful and constructive feedback, as well as for recognizing the originality of our work. We appreciate the acknowledgment that our paper addresses a practical gap between existing jailbreaking research and real-world usage scenarios, and that our training-free attack method offers easy deployment and testing. Below, we provide detailed responses and additional experimental results addressing the reviewer’s insightful questions.

Q1: Exact numbers in Figure 3 are not provided; Additional experiments on GPT-3.5-Turbo & GPT-4; Ablation study on stricter query budgets.

A1: Below is the raw data of the requested experiments. SECA remains effective on larger models, showing strong performance relative to raw ASR. Higher budgets lead to better success rates, but SECA still improves raw with limited queries.

ASR@30/10/1 (Raw)

ASR@30/10/1 (SECA)

Objective Difference (SECA increases the objective significantly compared to Raw)

Note: Due to time constraints, GPT-3.5-Turbo (gpt-3.5-turbo-0125) and GPT-4 (gpt-4-0613) were evaluated on a 30% subset.

Q2: Table 2 only show results on 3 models rather than 7

A2: We evaluated SECA and Raw on all seven models, as shown in Figure 3 and detailed in our response to Q1. As noted in Appendix K (Lines 635–637), we excluded GCG on 14B models due to memory constraints—our machine supports GCG only on 7B models. GPT models were also omitted, as GCG requires white-box access.

Q3: MMLU is a less common benchmark; testing on open-ended QA benchmarks (AdvBench or JBB-Behaviors); attacking reasoning models

A3: We did not test SECA on AdvBench or JBB-Behaviors because these datasets are designed for jailbreak attacks, whereas our work focuses on hallucination attacks, which are fundamentally different from jailbreaking (see our response to Q1 from Reviewer 6rsw). Also, we clarify that our setting is already an open-ended MCQA generation task, where the model is prompted to select an incorrect answer choice followed by a hallucinated open-ended explanation.

Due to space limitations, we kindly refer you to our response to Q1 from Reviewer LvAW for detailed discussion related to MMLU and open-ended QA, and to Q4 from Reviewer 6rsw for reasoning model–specific questions.

Q4: Compare with other attack methods

A4: The algorithms you mentioned are indeed interesting; we will include appropriate citations, and they may help further improve the efficiency of our SECA method. Due to space constraints in this rebuttal, we kindly refer you to our response to Q1 from Reviewer 6rsw for comparisons between SECA and the referenced attack methods.

Q5: Efficiency of SECA

A5: In our experiments, generating one candidate prompt takes approximately 0.5–1 second on a single NVIDIA A5000 GPU.

We sincerely thank the reviewer again for their valuable feedback and hope that our clarifications help reinforce the contributions of our work.

Dear Reviewer,

Thank you for your valuable questions and suggestions. Please let us know if you have any other concerns you'd like us to address. If our responses have been satisfactory and addressed all of your concerns, we would appreciate it if you would reconsider your evaluation.

Best,

The Authors