Gandalf the Red: Adaptive Security for LLMs

Thank you for taking the time to review our paper.

We appreciate the reviewer’s interest in the safety policy used in RedCrowd. We have now clarified our ethics, privacy and safety policy in the manuscript by adding the following to the introduction: “RedCrowd is a white-hat red-teaming system designed to identify vulnerabilities in commercial LLMs before they can be exploited maliciously. It does not promote or expose users to unsafe content. The password extraction use case focuses on identifying security weaknesses, not on generating or disseminating harmful outputs.”

Our focus was narrowly scoped to password extraction, which does not involve the generation of harmful or unsafe content. As such, we did not conduct an explicit analysis of broader safety violations committed by players. A more comprehensive taxonomy and analysis of safety violations is an important direction for future work, particularly as we extend RedCrowd to cover a wider range of misuse scenarios.

Thank you for taking the time to review our paper.

We group your comments and questions below and respond to each separately.

Generalizability of the results

• The D-SEC framework is not specific to any one application and generalizes beyond the experimental “password extraction” setup presented in the paper. Practitioners can apply security and utility metrics tailored to their use case. While the evaluation focuses on the narrow use case of password extraction, the findings provide generalizable insights as we observe consistent patterns across different target LLMs. For example, selecting a more relevant benign distribution has a consistent and measurable impact on the security-utility tradeoff across models.
• We appreciate your comment regarding the use of embeddings in measuring utility drop beyond over-refusal. We would like to clarify that our method does use embeddings to compute cosine similarity, as described in Figure 4. [We are happy to make this clearer in the main text if helpful.]
• To acknowledge the limitations of the task, we will add the following sentence to the Discussion section: “The insights from RedCrowd are limited by the narrow application to password extraction, and a broader empirical analysis is needed to draw application-agnostic conclusions.”

Actionable insights from the security-utility trade-off

• An advantage of Eq 1 is that it can be explicitly optimized, providing practitioners with a principled method for selecting defenses based on their specific goals.
• In the “Defense in depth” section, we show how optimizing the aggregation of multiple defenses for different $\lambda$ yields a strictly better tradeoff than naive “or” or “and” strategies.
• Beyond its practical optimization benefits, the framework highlights the often subtle but important ways in which security interventions can affect user utility, something we believe deserves more attention in the field.

Experiments with newer models

We limited our game design and analysis to three target LLMs. We selected models based on two criteria: (a) widespread use in real-world applications, and (b) variation in quality and size. GPT-4 was chosen over GPT-4o, which was not yet available when we designed the game protocol. Additionally, we were concerned that a stronger model might make it too hard for players to succeed, leading them to give up early, though we didn’t test this extensively. Expanding to a larger set of models remains an important direction for future work.

We appreciate the suggestion to resubmit prompts to newer models. It is important to note that we cannot replicate our full setup: the adaptive, multi-turn nature of gameplay is lost, and password reveals (central to our success metric) are difficult to detect automatically, especially for obfuscated outputs (see Appendix G). However, two parts of our study can be re-run: the defense-in-depth and utility experiments.

We re-ran both on GPT-4o and will include full figures and tables in the appendix. We provide a summary of findings below.

Defense-in-depth: GPT-4o shows a higher “unblocked” rate (see Figure 5), indicating more frequent circumvention of explicit refusals. Many responses avoid giving the password without triggering simple refusal heuristics, reinforcing the need for stronger semantic detection.

Utility: The table below reports SCR-derived false positive rates for GPT-4o alongside GPT-4 and GPT-4o-mini, including upper bounds of 95% confidence intervals. GPT-4o is consistently less likely to refuse benign inputs, particularly under strong system prompt defenses (C3).

We also compare utility metrics based on relative prompt length and cosine similarity to undefended responses. GPT-4o outperforms GPT-4 across all settings except prompt length under the weakest defense (B). As with other models, we observe a drop in utility with stronger defenses, but GPT-4o exhibits smaller degradation overall.

Thank you for taking the time to review our paper.

We have addressed both of your suggested improvements in the manuscript and respond explicitly to the comments below.

Additional citation and discussion on real-world applications

• We will make the implications of RedCrowd for real world deployments clearer by adding the following in the discussion: “While our experiments use RedCrowd in a controlled setting, the D-SEC framework is designed for real-world deployment. Developers can apply the core strategies—domain restriction, defense aggregation, and adaptive defenses—using their own user data, without needing to replicate RedCrowd in full. While large-scale crowdsourcing remains the gold standard for red teaming, it is resource-intensive; advancing highly capable automated attackers is an important direction for making these evaluations more practical at scale.”

Goals and capabilities of actors

• We appreciate the suggestion to incorporate adversary capabilities and goals to the threat model as discussed in [ https://arxiv.org/abs/2407.14937]. We will add the following sentence to the Threat Model section: “Following the framework in [1], we restrict the attacker’s capabilities to sending direct inputs to the target LLM via user messages. While indirect attacks via data (e.g., prompt injections in retrieved content) are also compatible with D-SEC, we focus on the direct setting for simplicity. Attacker goals are broad and application-specific.”

Thank you for taking the time to review our paper.

Below, we respond to all of the concerns and questions in your review. Please let us know if we missed anything or should expand on our responses.

Comments regarding the significance and novelty of the D-SEC framework

• We appreciate the reviewer’s observation and agree that prior work on over-refusals is highly relevant. We will incorporate an explicit reference to the suggested papers and add the following sentence to the main text: “User utility has previously been discussed in the context of model over-refusals [1, 2], a key way in which security interventions can degrade model usability. We analyze this effect in depth in the experimental section of this paper.”
• We would also like to clarify that the notion of utility in D-SEC is broader than over-refusals alone. D-SEC facilitates simple modifications to the utility function to account for general types of user experience degradation. We propose such example utility metrics in Figure 4, where we measure changes in response content independently of over refusals. We expect these effects to be even more pronounced in agent deployments, where the security layer can meaningfully impact the program’s execution flow.

Question about Eq (1)

• You are correct that for Eq (1) to make sense, both security and utility should be measured in comparable units. While this might appear limiting at first glance, it actually only means that the metrics for security and utility need to be appropriately (nonlinearly) transformed before analyzing the security-utility trade-off. We opted for this representation for two reasons: (i) It makes it easier for readers to understand the security-utility tradeoff. (ii) It encourages practitioners to select an appropriate transformation of their desired metric for which units have a comparable meaning. That said, as you allude to, it may not always be as easy as with AFR and SCR to transform metrics onto a comparable scale. We extended footnote 3 by adding: “Both $\mathcal{Q}_{\mathcal{M}}$ and $\mathcal{R}_{\mathcal{M}}$ are assumed to have been transformed appropriately so they have comparable units.”

Suggestions on improving paper structure

• List research questions and findings in experiments: We agree that making our research questions and key findings more explicit will improve the paper’s clarity. We will revise the introduction to (a) clearly state our goal of using large-scale crowdsourced red teaming to evaluate the security and utility trade-offs of leading commercial LLMs, and (b) connect it to the main experimental findings regarding the three strategies consistently improve the trade-off: restricting the application domain, aggregating multiple defenses in a defense-in-depth manner, and using adaptive defenses.
• In the current manuscript, we use “dynamic” to describe systems or assumptions that evolve over time (e.g., the threat model) and “adaptive” to describe mechanisms that incorporate past observations to modify behavior (e.g., attacks or defenses that respond to prior outcomes). We double-checked all occurrences and made sure they align with these definitions.
• Forward references: We agree that introducing later sections early will provide clarity. For example, we will revise the text to include: “Section 4.1 summarizes how utility metrics vary with the choice of benign user distribution and highlights that utility degradation can extend beyond over-refusals.”