On Evaluating the Durability of Safeguards for Open-Weight LLMs

We thank the reviewer for acknowledging the thoroughness and comprehensiveness of our study, as well as for recognizing the valuable insights it provides. We also greatly appreciate the constructive feedback for improvement. In response, we have carefully revised our work and provided clarifications below, which we hope will address the reviewer’s remaining concerns.

1. The problem of durably safeguarding open-weight LLMs is left unresolved because it is hard and still remains an open problem.

The paper highlights the challenges of creating durable safeguards but does not propose new methods or solutions...

We thank the reviewer for bringing up this issue. Below is our clarification on this point:

• The point of this paper is not to attempt to solve the problem but to establish baseline evaluation principles for future work that attempts to make progress. A painful lesson we learned from AI security research over the last decade is that many defenses failed to withstand rigorous scrutiny due to flawed or incomplete evaluations (e.g., [1,2]). This highlights the importance of establishing standard baseline evaluation principles for evaluating defenses, as exemplified by [3,4]. Only if we first address the meta-problem of evaluating defenses can we genuinely track progress in building defenses and confidently draw security conclusions.

  However, this evaluation issue has not received sufficient attention in recent work (such as RepNoise and TAR, which we examined in this paper) that attempts to produce durable safeguards for open-weight LLMs. These studies initially made strong claims about the effectiveness of their defenses, but upon re-evaluation, we found that they can be easily bypassed and that their evaluations contain various pitfalls, leading to overestimation of their defenses' effectiveness. This motivated us to conduct our study. By using these two defenses as case studies, we highlight common evaluation pitfalls in this area, and we hope that our findings can serve as important evaluation principles for future work attempting to make progress in this problem.

• Why didn't we explore alternative solutions? The reviewer is correct that this work did not explore alternative approaches for durably safeguarding open-weight LLMs. We didn't do so because safeguarding an open-weight model when the adversary can arbitrarily modify the model's weights is a difficult open problem. The goal of our paper is not to develop a new solution, but to ensure that any exploration of solutions to this challenging problem properly evaluates the solutions and avoids pitfalls.

  To appreciate its difficulty, consider that so far, we do not even have a solution for adversarial examples, where the adversary only "fine-tunes" the input to the model --- and it is already 2024, we still see researchers try very hard to solve the problem but the solution was then quickly shown to fail in openreview comments (https://openreview.net/forum?id=IHRQif8VQC).

  Now, we have a much harder problem: for open-weight LLMs, how can we defend against a much stronger adversary who can directly "fine-tune" the model's weights, not just the input, as in adversarial examples? It is even unclear under what conditions durably safeguarding open-weight models is a feasible goal to pursue under such a strong threat model. All these remain open research questions to date.

• Another point of this paper is to inform practitioners of the difficulty of safeguarding open-weight LLMs with current technologies. Not all practitioners in AI safety and security are fully aware of the actual effectiveness of current defenses. Our study provides solid evidence to help stakeholders calibrate their expectations regarding the feasibility of durably safeguarding open-weight LLMs with existing technologies.

  For example, a recent public interest comment (https://www.mercatus.org/research/public-interest-comments/comment-nist/aisi-ai-800-1-expanding-guidelines-open-source-ai) suggested that the U.S. National Institute of Standards and Technology (NIST) include approaches like TAR in their guidelines for managing open-source foundation models. Our evaluation offers an evidence-based counterpoint, indicating that users and policymakers should be cautious about relying on these methods in their current state.

[1] Carlini, Nicholas, and David Wagner. "Defensive distillation is not robust to adversarial examples." arXiv preprint arXiv:1607.04311 (2016).

[2] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." International conference on machine learning. PMLR, 2018.

[3] Carlini, Nicholas, et al. "On evaluating adversarial robustness." arXiv preprint arXiv:1902.06705 (2019).

[4] Tramer, Florian, et al. "On adaptive attacks to adversarial example defenses." Advances in neural information processing systems 33 (2020): 1633-1645.

We are delighted that the reviewer finds our work clearly motivated and our replication studies detail-oriented and of high importance. We also thank the reviewer for the constructive feedback and suggestions for improvement. We hope our following revision and clarifications can address the reviewer’s remaining concerns:

1. We have revised our paper to address several ambiguities pointed out by the reviewer in our presentation.

W1a: This work uses ambiguous language for technical concepts at times. Formalising some concepts using math could help to greately disambiguate concepts that are vague in text. For instance, L136 "RepNoise trains a model to push its representations of HarmfulQA data points at each layer toward random noise." For readers not familiar with prior work, this statement is hardly comprehensible. Formalising objectives can help to clarify.

Following the reviewer's suggestion, we have added a new Appendix C to provide a more formal introduction to RepNoise and TAR. There, we have mathematically formalized the training objectives of the two approaches for better clarity and precision. We also added reference links to the main text of this appendix, where these concepts are introduced.

W1b: The paper is sometimes a too ambiguous in language when presenting results. For instance, "the model with RepNoise does not respond to HarmfulQA questions more than a moderate amount" (L138). What is a moderate amount?

Following the reviewer's suggestions, we have tightened our language here to avoid ambiguity. We revised the initial sentence the model with RepNoise does not respond to HarmfulQA questions more than a moderate amount to the model with RepNoise can still consistently refuse over 90% of HarmfulQA questions from the test set.

W1c: The specific details of attacks and threat models are not always entirely clear. For instance, "An attacker attempts to recover this knowledge through fine-tuning." (L163) is very vague. What data is this fine-tuning performed on? Fine-tuning on a high quality dataset of weaponization knowledge will always improve performance and unlearning cannot prevent that.

As the reviewer also pointed out, for the threat model of TAR, there isn't a clear restriction on what data the attack can use for fine-tuning. We would like to clarify that this is not an ambiguity in our writing but rather an accurate representation of the strong and broad threat model faced by TAR. To clarify, TAR is designed to introduce durable safeguards into open-weight LLMs, ensuring that adversaries cannot recover weaponization knowledge even after fine-tuning the model for several thousand steps. Our critique in the paper emphasizes that this threat model may be overly strong and broad; evaluating defenses against fine-tuning attacks on a single dataset is insufficient to assess their overall effectiveness.

2. We have added a checklist of evaluation pitfalls to make our learned lessons more explicit and actionable.

W2: The paper could be more explicit about the lessons learned from failing to replicate results reliably. The findings could be more actionable. For instance, providing a checklist for future evaluations might be useful.

We thank the reviewer for this suggestion. In our rebuttal revision, we added Appendix A to provide a more explicit and actionable checklist of items to consider for future evaluations of defenses to avoid similar pitfalls we identified in our paper. The checklist includes tips for:

(1) Checking the robustness of the defense against attacks with different random seeds;

(2) Employing widely used and thoroughly tested attack implementations for defense evaluation;

(3) Ensuring the defense either restricts its threat model to scenarios it can reliably address or undergoes comprehensive evaluation against a wide range of possible attacks within the defined threat model;

(4) including comprehensive common benchmark tests to address potential side effects;

(5) exercising caution when claiming "unlearning", and some suggested tests for verifying unlearning claims.

We also note that this checklist is by no means exhaustive. The problem of evaluating defenses for open-weight LLMs is hard and is still at an early stage of research. We hope our checklist can be a starting point for future work to rule out at least some basic mistakes in evaluation.

We are delighted that the reviewer finds our paper clear and well-written, and the problem we illustrate in the paper is important. We also appreciate the reviewer's constructive feedback and suggestions for improvement. We hope our following revision and clarifications can address the reviewer’s remaining concerns:

1. Clarifications regarding our evaluation setups for multiple-choice datasets

It’s a standard practice to evaluate models on multiple-choice datasets by checking the highest logits of the letters corresponding to the answer options. So evaluating on a full answer by using humans and LLM judges sounds a bit of an unfair comparison. I agree that the practice of evaluating the logits instead of the full answer might be insufficient to properly compute the performance, but this sounds more a problem of the standard evaluation setting than the problem itself. Probably a more fair comparison would be to check the logits after adding the chat template.

We agree with the reviewer that it's very common to evaluate models on multiple-choice datasets by checking the highest logits of the letters corresponding to the answer options. This approach is appropriate for utility performance tasks, such as those in the MMLU benchmark. High accuracy under this setup is a meaningful metric correlated with the model's task capability, making it a suitable basis for fair model-to-model comparisons.

However, in our paper, for the evaluation case where we use a full answer + humans and LLM judges, the context is very different --- our goal here is not to compare model capabilities fairly in a multiple-choice question (MCQ) task but rather to challenge whether the standard MCQ evaluation setup is even a reliable way to measure a model's unlearning of any specific knowledge. More specifically, The rationale here is as follows:

• Here, we evaluate models on the WMDP benchmark [1], which is to measure whether the model has unlearned certain categories of weaponization knowledge. The promise of this benchmark is that --- if a model has unlearned the weaponization knowledge, it should not be able to answer the multiple-choice questions (MCQ) in the benchmark (consisting of weaponization knowledge questions) correctly, and the accuracy should be low. The defense such as TAR studied in this paper shows that the model with the safeguard indeed has a low accuracy on this benchmark and therefore claims the effectiveness of the defense.

• The challenge we raise is --- is a low accuracy on the WMDP benchmark under a certain evaluation setup (e.g., checking the highest logits of the letters corresponding to the answer options) really a reliable indicator that the model has unlearned the weaponization knowledge? We show that it is not necessarily the case --- by simply adding a prompt template and evaluating the answer in free-form text, the model's performance can be significantly better on this weaponization knowledge benchmark, and the model definitely still retains the weaponization knowledge to correctly answer the questions. Thus, a low accuracy on WMDP under the standard MCQ evaluation setup does not necessarily mean the model has unlearned the weaponization knowledge.

Regarding the reviewer’s suggestion that "a more fair comparison would be to check the logits after adding the chat template": we agree and have already included such an evaluation in Figure 4. However, as shown, even with a chat template, the logits-based evaluation still indicates random accuracy on the WMDP benchmark, incorrectly suggesting the model may have unlearned the knowledge. In contrast, free-form text responses evaluated by human and LLM judges reveal higher accuracy, countering this conclusion.

[1] Li, Nathaniel, et al. "The wmdp benchmark: Measuring and reducing malicious use with unlearning." arXiv preprint arXiv:2403.03218 (2024).

In evaluating other multiple-choice datasets, did you use human evaluation + LLM judge as well?

No, for any other evaluation of multiple-choice datasets (except for the evaluation in Figure 4), we just follow the standard evaluation setup. This ensures that our reproduced studies follow the same setup as the original studies.

Again, the use of the full answer + human evaluation / LLM judge setup in Figure 4 solely demonstrates that the standard MCQ evaluation is not always reliable for assessing unlearning.

We sincerely thank the reviewer for the thoughtful engagement with our work. We are pleased to know that our revisions and clarifications have addressed the reviewer’s initial questions.

Regarding the model’s performance drop after TAR, the reviewer’s observations are correct. Indeed, our independent evaluation shows that the model’s performance on multiple benchmarks can drop to concerningly low levels post-TAR. This is an important finding we highlight in our paper. Our goal is to caution researchers in this domain about such potential side effects and call for comprehensive evaluations to identify or rule out such side effects. This can make any proposed mitigation approach more convincing and reliable to use.

Regarding the reviewer’s follow-up question:

Using open-ended generation might provide better signal on the knowledge that a model has, however this can be hard to evaluate practically. For example, an LLM judge evaluating for biosecurity knowledge needs to know very well the topic in order to be able to classify whether an AI output is nonsensical or correct, and in addition might not be reliable anyway. And having humans judging AI outputs doesn't scale well and can be prohibitively costly, especially because one needs experts in a field to judge properly. Do you think there might be a good "compromise" between full MCQ evaluation and full open-ended?

The reviewer raises an excellent point regarding the trade-offs between open-ended-generation-based evaluation and the standard first-token-logits-based evaluation. We find this issue is moderate in the specific context we deal with in our paper. We clarify this point below.

In our paper, we are primarily only working with the MCQ questions for the unlearning evaluation — the WMDP benchmark is a set of MCQ questions. By “open-ended generation”, we essentially mean that we allow the model to output its choice among (A,B,C,D) in an open-ended generation as opposed to relying on the logits of the first output token to decide the model’s answer. In this context, the LLM judge is only used to extract the model’s selected choice (among A~D) from the generated output. This is a simple task for the LLM judge model — the judge is only tasked with parsing the model’s output to extract its MCQ answer instead of using any deep domain expertise (e.g., biosecurity knowledge) to assess the correctness of the answer. We hope this clarification addresses the reviewer’s concerns and demonstrates how our approach navigates the trade-offs between practical feasibility and rigorous evaluation.

More broadly, we agree with the reviewer’s point that scaling evaluations for open-ended generation tasks beyond MCQ-style questions is hard. It remains an ongoing challenge in the general LLM evaluation landscape, not just in the context of unlearning evaluation.

We appreciate the reviewer's recognition of the importance of the issue addressed in our paper, as well as the acknowledgment of the empirical rigor and the valuable practical implications of our work. We hope our following clarifications can address the reviewer’s remaining concerns:

1. Exploration of Alternative Solutions

We thank the reviewer for bringing up this issue. Below is our clarification on this point:

• The point of this paper is not to attempt to solve the problem but to establish baseline evaluation principles for any future work that attempts to make progress in this problem. A painful lesson we learned from AI security research over the last decade is that many defenses failed to withstand rigorous scrutiny due to flawed or incomplete evaluations (e.g., [1,2,3]). This highlights the importance of establishing standard baseline evaluation principles for evaluating defenses, as exemplified by [4,5]. Only if we first address the meta-problem of evaluating defenses can we genuinely track progress in building defenses and confidently draw security conclusions. However, we notice this evaluation issue has not received sufficient attention in recent work (such as RepNoise and TAR, which we examined in this paper) that attempts to produce durable safeguards for open-weight LLMs. These studies initially made strong claims about the effectiveness of their defenses, but upon re-evaluation, we found that they can be easily bypassed and that their evaluations contain various pitfalls, leading to overestimation of their defenses' effectiveness. This motivated us to conduct our study. By using these two recent defenses as case studies, we illustrate and highlight common evaluation pitfalls in this problem area, and we hope that our findings can serve as important evaluation principles for future work attempting to make progress in this problem.
• Why didn't we explore alternative solutions? The reviewer is correct that this work did not explore alternative approaches for durably safeguarding open-weight LLMs. We didn't do so because safeguarding an open-weight model when the adversary can arbitrarily modify the model's weights is a difficult open problem. The goal of our paper is not to develop a new solution, but to ensure that any exploration of solutions to this challenging problem properly evaluates the solutions and avoids pitfalls. To appreciate its difficulty, consider that so far, we do not even have a solution for adversarial examples, where the adversary only "fine-tunes" the input to the model --- and it is already 2024, we still see researchers try very hard to solve the problem but the solution was then quickly shown to fail in openreview comments (https://openreview.net/forum?id=IHRQif8VQC). Now, we have a much harder problem: for open-weight LLMs, how can we defend against a much stronger adversary who can directly "fine-tune" the model's weights, not just the input, as in adversarial examples? It is even unclear under what conditions durably safeguarding open-weight models is a feasible goal to pursue under such a strong threat model. All these remain open research questions to date.
• Another point of this paper is to inform practitioners of the difficulty of safeguarding open-weight LLMs with current technologies. Not all practitioners in AI safety and security are fully aware of the actual effectiveness of current defense approaches. Our study provides solid evaluation evidence to help stakeholders calibrate their expectations regarding the feasibility of durably safeguarding open-weight LLMs with existing technologies. For example, a recent public interest comment (https://www.mercatus.org/research/public-interest-comments/comment-nist/aisi-ai-800-1-expanding-guidelines-open-source-ai) suggested that the U.S. National Institute of Standards and Technology (NIST) include approaches like TAR in their guidelines for managing open-source foundation model misuse risks. Our evaluation offers an evidence-based counterpoint, indicating that users and policymakers should be cautious about relying on these approaches in their current state.

[1] Carlini, Nicholas, and David Wagner. "Defensive distillation is not robust to adversarial examples." arXiv preprint arXiv:1607.04311 (2016).

[2] Carlini, Nicholas, and David Wagner. "Magnet and" efficient defenses against adversarial attacks" are not robust to adversarial examples." arXiv preprint arXiv:1711.08478 (2017).

[3] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." International conference on machine learning. PMLR, 2018.

[4] Carlini, Nicholas, et al. "On evaluating adversarial robustness." arXiv preprint arXiv:1902.06705 (2019).

[5] Tramer, Florian, et al. "On adaptive attacks to adversarial example defenses." Advances in neural information processing systems 33 (2020): 1633-1645.

We thank the reviewer for the thoughtful engagement and detailed feedback on our work. We are pleased to know that our revisions and clarifications have made the objective of this paper clearer. We also appreciate the reviewer’s recognition of the value of the actionable checklist we provide, which aims to guide future evaluations and address the pitfalls highlighted in the paper. We hope the following clarifications can help address the reviewer’s remaining concerns:

Research roadmap and pessimism in the progress. We wish to acknowledge upfront that this paper indeed does not propose a practical roadmap for solving the problem of durably safeguarding open-weight LLMs. As discussed in our earlier rebuttal, this is because this problem remains a fundamentally hard and open problem. Particularly, this problem is expected to be more difficult than addressing adversarial examples — and it’s known that even for addressing adversarial examples, a clear roadmap remains absent despite over a decade of research efforts. So, we hope the reviewer can appreciate that developing such a roadmap for durably safeguarding open-weight LLMs is a highly non-trivial agenda, one that warrants a dedicated line of work separate from the scope of this paper. From a positive perspective, we do believe these defense mechanisms (including future methods evolving from RepNoise and TAR) could potentially increase the cost to adversaries, even if they don’t provide high worst-case security guarantees, but evaluations should accurately and rigorously reflect the added costs for adversaries. Otherwise, an incomprehensive evaluation may neglect the potential side effects brought by the defense. For instance, the original evaluation of RepNoise and TAR failed to capture the high variance of the model’s robustness against fine-tunining attack due to the lack of evaluating with different random seeds. A key contribution of our work is to provide constructive recommendations to avoid such biased evaluation in our paper.

We also understand the reviewer’s concern that the paper may project a pessimistic view on the progress of developing defenses. However, we believe that this paper will have a positive impact on advancing the field. The baseline evaluation checklist we propose establishes a more rigorous protocol for assessing the correctness and effectiveness of defenses. This will enable future research to better track progress, avoid common pitfalls, and mitigate the risk of pursuing directions that yield seemingly promising results but are ultimately artifacts of misevaluation. This dynamic has been observed in the previous study of adversarial examples, where a series of critical adaptive attack studies [1,2,3,4,5] have played a pivotal role in identifying which defenses are effective and which are not.

[1] Carlini, Nicholas, and David Wagner. "Defensive distillation is not robust to adversarial examples." arXiv preprint arXiv:1607.04311 (2016).

[2] Carlini, Nicholas, and David Wagner. "Adversarial examples are not easily detected: Bypassing ten detection methods." Proceedings of the 10th ACM workshop on artificial intelligence and security. 2017.

[3] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." International conference on machine learning. PMLR, 2018.

[4] Carlini, Nicholas, et al. "On evaluating adversarial robustness." arXiv preprint arXiv:1902.06705 (2019).

[5] Tramer, Florian, et al. "On adaptive attacks to adversarial example defenses." Advances in neural information processing systems 33 (2020): 1633-1645.