Tamper-Resistant Safeguards for Open-Weight LLMs

Thank you for your careful analysis of our work. We are glad you found our work to be of great significance, and we hope the following response addresses your concerns.

Definition of metrics.

There are to many proxy objectives, like "safety_metric", "capabilities_metric". These pre-defined metrics will limit the significance and universality of the proposed method. Unless the author can prove that TAR is still working under other "safety_metric" or "capabilities_metric"

In our method description, we only have two proxy objectives: one for safety and one for capabilities. These are the training losses that we use to improve the test-time safety and capabilities metrics. In other words, each application of our method only needs to consider two losses and two test-time metrics, which is fairly standard for the field.

Are there any experiments to validate TAR on proxy metrics has a certain degree of generalization on other safety scenes or benchmarks?

Yes, we do have these experiments! Each of the two domains that we describe in section 3.2 uses different specific safety metrics, so we already have results demonstrating that TAR works under multiple different safety and capabilities metrics. We will clarify these points in the updated paper. Thank you for your feedback.

TAR training cost.

How about the training cost of TAR. Is it larger than original training?

Each TAR fine-tuning run required 18 hours on 8 NVIDIA A100 80GB GPUs. This is approximately 8 orders of magnitude less compute than original pre-training for the Llama-3-8B model. For details, please see below.

Regarding run-time overhead, for each inner loop step, both an adversary gradient and meta-gradient is computed for the adversary’s loss and tamper-resistance loss, respectively. The outer loop also computes one additional gradient for the retain-set, which is added to the accumulated meta-gradient. However, via the subsampling trick discussed in Appendix B.3, we compute meta-gradients every 4 adversary steps, which we found to speed up training by a constant factor without sacrificing robustness. In total, there are (750 retain gradients + 750*64 adversary gradients + 750*64/4 meta-gradients) = 60,750 forward/backward passes computed during our method with a batch size of 64.

This is multiple orders of magnitude fewer forward/backward passes than used in original model pre-training, which is reported to be 1,200,000 optimization steps with a batch size of 8,000,000 for Llama-3-8B [1]. This comes out to a factor of around 10^8 fewer forward/backward passes for our TAR fine-tuning method compared to Llama-3-8B pre-training.

Regarding memory overhead, one might consider memory to be the primary concern for running TAR on large models. However, since we perform first-order meta-learning, as we explain in Appendix B.4, meta-gradients can be accumulated in-place in a separate data structure. Even with common model sharding algorithms such as FSDP, this can be performed without additional distributed communication, requiring additional memory equivalent to only one extra copy of model gradients per device. Thus, our method is quite memory-efficient as it scales memory usage by only a constant factor with sharded model size, which means TAR has the same memory complexity as standard training asymptotically.

Thank you for bringing this up, we will include more discussion of these costs and our mitigations in our revision.

[1] Llama Team, AI @ Meta. The llama 3 herd of models, 2024.

Thanks the authors for the comprehensive response. The response addressed some of my concerns. However, for the core idea, although authors propose their method compute adversarial gradients on model parameters instead of model inputs. I still find there are previous works [1] have consider the adversarial weight perturbation. The core contribution of this paper is not convincing for me. Considering the authors make a comprehensive study on Weight Tampering Attack, I have decided to keep my original rating.

[1] Adversarial Weight Perturbation Helps Robust Generalization. Dongxian Wu, Shu-tao Xia, Yisen Wang

Thank you for your careful analysis of our work. We are glad you found our experiments to support the performance of our method. We hope the following response addresses your concerns.

Our main contribution is to show that tamper-resistance is tractable.

Insufficient sustainability … this effectiveness actually depends on the diversity of attack types included in the training data … the resilience of the proposed mechanism may be superficial and does not guarantee the security of open-weight LLMs. Furthermore, the authors do not provide corresponding theoretical analysis or proof.

In the paper, we clearly state that our method does not guarantee robustness across arbitrary adversaries. Obtaining theoretical guarantees for tamper-resistance is quite difficult at this stage, given that research on the problem has only just begun. To address this, we emphasize in the paper that extensive red teaming of methods is critical to gain confidence in their empirical robustness.

For this reason, we stress-test our method with 28 diverse attacks, finding that some do succeed. Our main contribution is to show that tamper-resistance is a tractable problem in the first place, which we demonstrate for the first time in LLMs.

Clarifying technical contributions.

Incremental technical contributions. Although the paper is expressed clearly, its innovation is not evident in terms of both technical aspects and application scenarios. Specifically, the proposed solutions are based on existing widely used methods, and the authors have not clearly articulated their unique contributions.

Please see our above description of how our main contribution is to show that tamper-resistance for LLMs is tractable in the first place. This itself is a significant technical contribution that we stress throughout the paper.

Our technical methods indeed build on prior work. As we mention in the paper, our TAR method builds on MLAC and first-order MAML. However, we make numerous significant technical contributions on top of these works as follows:

Other technical contributions include:

• Developing a tamper-resistance evaluation pipeline for LLMs, including an extensive red teaming suite. This did not exist prior to our work.
• Demonstrating that MLAC does not scale to LLMs
• Developing a 2-stage method with an initial unlearning/alignment stage that doesn’t use tamper-resistance training, which greatly outperforms prior work
• Demonstrating that input-based adversarial training (e.g., R2D2) does not generalize to provide robustness to weight tampering attacks. This had not previously been shown.
• Demonstrating that concurrent work (e.g., RepNoise) lacks robustness against a broader set of attacks
• Identifying a significantly improved outer loop loss for tamper-resistance training
• Introduced methods for minimizing memory overhead during meta-learning, making our adversarial meta-learning procedure possible.

Given these contributions that can be found in our paper, we kindly disagree that the technical and experimental contributions of the paper are incremental. However, we agree that the contributions could be made more clear. We will add an explicit contributions list to the updated paper. Thank you for your suggestion.

We significantly improve resilience compared to prior work.

The performance of the proposed mechanism is closely related to the adversarial training ... which means its resilience remains a significant issue.

We agree that resilience to attacks is an important issue. As previously mentioned, our goal is to show that progress is possible in contrast with prior work, since the best existing methods are not robust to fine-tuning attacks. Thus, while our method does not provide full resilience, it does improve resilience.

Clarifying Figure 5.

The presentation of the performance comparison between TAR and existing mechanisms in Figure 5 is unclear and potentially confusing. The authors should provide further analysis of this result, explaining why the performance of TAR shows a significant change as the step size increases.

Please note that the optimizer step-size does not increase in Figure 5. Rather, the x-axis shows the number of optimizer steps. Also note that TAR keeps the adversary’s loss high (above the recovery region) across all 1000 steps, indicating that TAR’s performance is strong compared to the baseline.

You may be asking about significant changes in the adversary’s loss shown in Figure 5. We partly explain these changes in Section 5.3. However, we agree that further analysis could improve clarity for readers, which we will add to the updated paper. Namely, we will explain that the model has implemented the entropy maximization by sharply increasing the cross-entropy loss at the beginning of fine-tuning. Thank you for your suggestion.

Thank you again for your comments. If we have addressed the thrust of your concerns, we kindly ask that you consider raising your score.

Thank you for your careful analysis of our work. We are glad that you found our paper to be well-written, and our experiments to be comprehensive! We hope the following response addresses your concerns/questions

Time and memory complexity of TAR / our efficiency optimizations.

The time complexity analysis is important but not included … I suggest the authors provide computation analysis either empirical or theoretical.

Your concern about computational efficiency for large models is valid, as sampling fine-tuning attacks and performing meta-learning naively may increase both runtime and memory overhead. However, we address both of these concerns in our paper, introducing several efficiency tricks in Appendix B.3 and B.4 to make first-order meta-learning possible and efficient for billion-parameter LLMs. We provide additional analysis for each of your concerns below for understanding the costs and how we mitigate them, as discussed in our paper.

Both adversarial training and meta-learning are time-consuming. The proposed method can be expensive especially when the model size is increasing.

Our method, TAR, involves an inner-loop/outer-loop training setup, for conducting first-order meta-learning as described in both Algorithm 1 and Equation 1. For outer loop steps $N$ and inner loop steps $K$, the runtime complexity of the method is $O(NK)$. Generally, $K<<N$, as we observe empirically in Figure 6 in our appendix that inner-loop lengths of $K=64$ with an outer loop length of $N=750$, can already preserve a high adversary loss for much more than $K$ steps.

Regarding run-time overhead, for each inner loop step, both an adversary gradient and meta-gradient is computed for the adversary’s loss and tamper-resistance loss, respectively. The outer loop also computes one additional gradient for the retain-set, which is added to the accumulated meta-gradient. However, via the subsampling trick discussed in Appendix B.3, we compute meta-gradients every 4 adversary steps, which we found to speed up training by a constant factor without sacrificing robustness. In total, there are (750 retain gradients + 750*64 adversary gradients + 750*64/4 meta-gradients) = 60,750 forward/backward passes computed during our method with a batch size of 64.

This is multiple orders of magnitude fewer forward/backward passes than used in original model pre-training, which is reported to be 1,200,000 forward/backward passes with a batch size of 8,000,000 for Llama-3-8B [1].

On 8 80GB A100 GPUs, our method with N=750 and K=64 can be run in 18 hours.

The proposed method can be expensive especially when the model size is increasing. This brings a concern about whether this method is practical for protecting large models.

Regarding memory overhead, one might consider memory to be the primary concern for running TAR on large models. However, since we perform first-order meta-learning, as we explain in Appendix B.4, meta-gradients can be accumulated in-place in a separate data structure. Even with common model sharding algorithms such as FSDP, this can be performed without additional distributed communication, requiring additional memory equivalent to only one extra copy of model gradients per device. Thus, our method is quite memory-efficient as it scales memory usage by only a constant factor with sharded model size, which means TAR has the same memory complexity as standard training asymptotically.

Thank you for bringing up this concern, we will include more discussion of these costs and our mitigations in our revision.

[1] Llama Team, AI @ Meta. The llama 3 herd of models, 2024.

Understanding the hyperparameters of TAR.

There are many hyperparameters in either objective in Eq (1) or optimizing it, such as the number of outer loops and coefficients before tamper-resistance loss and retain loss (lambda_TR and lambda_retain). How they influence the defending performance is not discussed, and I suggest including it.

We agree that discussing the effect of hyperparameters is important. In fact, we do include detailed discussion of important hyperparameters in our ablations in Appendix C.2 and C.4. In Figure 6 of Appendix C.2, we show the effect of increasing the inner loop steps $K$. In Table 6 of Appendix C.4, we actually already discuss the effect of varying $\lambda_{TR}$ as you suggest. We found that increasing $\lambda_{TR}$ from $1.0$ to $4.0$, while holding $\lambda_{\text{retain}}$ at $1.0$, greatly improved overall tamper-resistance, in line with our expectations.

Regarding the outer loop steps $N$, we ran our method until we observed convergence in our full TAR loss (Eqn. 1). The smoothed outer-loop TAR loss is shown as the blue curves in Figure 3. We mention in Section 5.1 that $N=750$ for the weaponization knowledge restriction setting, and in Section 5.2 that $N=100$ for harmful request refusal.

To clarify this for readers, we will add a list of all hyperparameters to the updated paper in one location for ease of reference, with recommended settings from our ablations. Thank you for your suggestion.

Sampling attacks during TAR.

I am curious about what kind of A_train in the objective in Eq(1) is required to have a good defense. For example, how diverse attacks need to be included; how many adversarial examples for each attack need to be included, etc.

This is a great question! We generally found that diversity in both dataset distribution and learning rate contribute significantly to robustness. For example, in Section 3.3, we mention that while developing TAR, we included new fine-tuning attacks that broke intermediate versions of the method. In one interesting case, we were concerned about an adversary that initially performs benign, retain-set fine-tuning, followed by forget-set fine-tuning (called the “R->F adversary”). We show in Table 4 in Appendix C.3 that targeted patching of vulnerabilities is possible with TAR, as sampling a reduced version of this R->F adversary actually did improve downstream robustness. This demonstrates that increasing the diversity of training attacks can improve downstream robustness.

Overall, the configuration of training adversaries listed in Table 7 and 8 obtained the best robustness in our experiments for the domains we consider, which we hope future work will improve upon.

If we have addressed the thrust of your concerns, we kindly ask that you consider raising your score.

Thank you for your careful analysis of our work. We hope the following response addresses your concerns.

Clarifying design motivation in Section 4.

The organization of Section 4 requires further refinement for improved readability. It would be beneficial to briefly outline the design motivation before delving into specific details, particularly regarding the content of Fig. 3.

We apologize for any lack of clarity in Section 4. To improve clarity, we will briefly outline the motivation for our tamper-resistance loss at the top of Section 4.2. Thank you for your suggestion.

We have added mathematical formulations for the tamper-resistance losses.

the mathematical formulation of the loss function used is currently absent.

We kindly point the reviewer to Appendix B.2, where we fully describe the tamper-resistance losses used during TAR. However, we agree that adding mathematical formulas in the main paper for the tamper-resistance losses would improve readability. We have added these to the updated paper. Thank you for your suggestion.

Improved caption for Figure 1.

The caption of figure 1 needs to explain the difference between the two branches more clearly. For example, what's the difference between the first nodes of the two branches.

Please see below for the improved Figure 1 caption. We agree that the original caption lacked clarity, and we think the new version will help readers understand the problem setting better. Thank you for your suggestion.

New caption: “An illustration comparing two approaches to LLM safety when subjected to adversarial fine-tuning. The top branch shows conventional safeguards (like refusal training), which can be easily bypassed when adversaries fine-tune the model weights to remove safety constraints. The bottom branch demonstrates our proposed TAR (Tamper-Resistant) method, which maintains robustness even when adversaries attempt to fine-tune the model to reintroduce harmful capabilities. While conventional safeguards can be circumvented by adversaries with access to model weights, TAR provides significantly more protection against attempts to weaponize open-weight LLMs through malicious fine-tuning.”

Clarifying 5000 step claim and attack configurations.

In Section 1, the authors assert that the proposed method can endure fine-tuning of up to 5000 steps … this claim does not intuitively convey the contribution of the paper.

The 5000-step attacks correspond to Adv. 1 and Adv. 2 in Table 9 and Figure 4, which utilize a fully held-out biology dataset. We agree that claims based on hyperparameters such as step-count could be misleading, and considering the hyperparameters of the attack is crucial. We show in Table 9 that the 5000-step adversaries use similar optimization hyperparameters as all other attacks in our red-teaming suite, with the learning rate for these 2 adversaries varied between $2\times10^{-5}$ and $4\times10^{-5}$. We believe our robustness results on these adversaries support our claims, in that our method is capable of withstanding much stronger optimization pressure compared to prior work. We will clarify this in the updated paper.

The details surrounding fine-tuning, such as batch size and learning rate, are unclear; these parameters significantly influence the number of fine-tuning steps.

We have already fully specified these hyperparameters in Tables 7, 8, 9, and 10 in our Appendix.

Secondly, a comparative analysis with the typical number of steps required to fine-tune models on downstream tasks is lacking.

In Table 9, we list out all of our attacks, which are performed on standard fine-tuning hyperparameters such as varying the learning between between $2\times10^{-6}$, $2\times10^{-5}$, and $4\times10^{-5}$, optimization steps of 1000 or 5000, and batch sizes of 32 or 64. We also consider LoRA training alongside full-parameter training. These are all representative of common fine-tuning settings for Llama-3-8B models.

Thanks for the authors' response that has addressed partial my doubts. Based on this, I have decided to modify my rating to 5.

Thank you for your careful analysis of our work and positive rating. We are glad you found our method intuitive and that our experimental results validate our claims. We hope the following response addresses your concerns:

Runtime and memory costs.

The authors do not discuss the cost of the experiments, including time cost and GPU memory cost. Section B.4 mentioned that the experiments use 8 A100 with 80GB GPU memory. What is the minimum requirement for the experiments?

Our method, TAR, involves an inner-loop/outer-loop training setup, for conducting first-order meta-learning as described in both Algorithm 1 and Equation 1. For outer loop steps $N$ and inner loop steps $K$, the runtime complexity of the method is $O(NK)$. Generally, $K<<N$, as we observe empirically in Figure 6 in Appendix C.2 that inner-loop lengths of $K=64$ with an outer loop length of $N=750$, can already preserve a high adversary loss much more than $K$ steps.

Regarding run-time overhead, for each inner loop step, both an adversary gradient and meta-gradient is computed for the adversary’s loss and tamper-resistance loss, respectively. The outer loop also computes one additional gradient for the retain-set, which is added to the accumulated meta-gradient. However, via the subsampling trick discussed in Appendix B.3, we compute meta-gradients every 4 adversary steps, which we found to speed up training by a constant factor without sacrificing robustness. In total, there are (750 retain gradients + 750*64 adversary gradients + 750*64/4 meta-gradients) = 60,750 forward/backward passes computed during our method with a batch size of 64.

This is multiple orders of magnitude fewer forward/backward passes than used in original model pre-training, which is reported to be 1,200,000 optimization steps with a batch size of 8,000,000 for Llama-3-8B [1].

On 8 80GB A100 GPUs, our method with N=750 and K=64 can be run in 18 hours.

Regarding memory overhead, one might consider memory to be the primary concern for running TAR on large models. However, since we perform first-order meta-learning, as we explain in Appendix B.4, meta-gradients can be accumulated in-place in a separate data structure. Even with common model sharding algorithms such as FSDP, this can be performed without additional distributed communication, requiring additional memory equivalent to only one extra copy of model gradients per device. Thus, our method is quite memory-efficient as it scales memory usage by only a constant factor with sharded model size, which means TAR has the same memory complexity as standard training asymptotically.

Thank you for bringing up this concern, we will include more discussion of these costs and our mitigations in our revision.

[1] Llama Team, AI @ Meta. The llama 3 herd of models, 2024.

Contributions.

I suggest including a statement of contribution to make this paper easier to follow.

Thank you for your suggestion. We will do so in our revision. Specifically, we will highlight the following 3 core contributions: (1) We develop a tamper-resistance evaluation pipeline for LLMs, including an extensive red-teaming suite. (2) We show that tamper-resistance is tractable for LLMs, and we introduce the first method to provide robustness for a wide range of adversaries. (3) We show that existing methods for adversarial training of LLMs, including concurrently released unlearning methods, are not robust.

Capabilities-robustness tradeoff.

In Figure 2, the difference between TAR and the baseline methods is significant. However, it seems that the capabilities of TAR are lower than the baseline methods. Is there a trade-off between capabilities and tamper resistance?

We do observe a trade-off between benign capabilities and tamper-resistance, which we discuss in Section 5 and Appendix C.4. This is a similarity between our setting and standard adversarial robustness, e.g., for image classification, where adversarial training defenses often induce a trade-off between robustness and benign accuracy. Improving this tradeoff to reduce the cost of tamper-resistance defenses would be an interesting direction for future work.

Improving clarity.

We apologize for any confusion in our explanation of experiments. We clarify each of your questions below, and will update the paper in the revision to improve clarity.

Many important parts of the paper are in Appendix, e.g. Random Mapping, attack categories, many crucial evaluations (see above). This makes the main paper less clear and prevents it from being self-sufficient.

We apologize for introducing any confusion. We wanted to prioritize discussion of the method and experimental results in the main text, which is limited to 10 pages. We will update our revision to clarify exactly where to look for further content in the Appendix, to improve the experience of the reader.

Was a single model trained to be tamper-resistant against all 3 weaponization domains, or were 3 different TAR models considered (e.g. in Table 1)? Was a single model trained for both weaponization restriction and harmful request refusal, or 2 different ones?

The weaponization restriction and harmful request refusal are 2 separate settings; 3 separate models were trained for weaponization restriction (one model for each weaponization domain) and 1 model for harmful request refusal. We include the full details for each training setting in Appendices D.1 and D.2.

Could a single model be defended against all considered harms? How would it affect benign capabilities?

This is a great question. For ease of evaluation and minimizing confounding variables, we opted to train different models for each setting. We were initially interested in seeing if achieving specificity of tamper-resistance to specific knowledge domains was possible, which we do observe in Table 1. It’s difficult to predict how benign capabilities would change as a result of defending against all harms in a single model. This would be a good direction to explore for future work.

Is the approach the same for baseline methods? It was not clear from the text for me. These details could be included in the experimental setup section, and you could include a diagram or table summarizing the model configurations used for different experiments.

Yes; we implemented/reproduced the baseline methods to also individually restrict harms for each of the considered knowledge domains. We will update our revision to include a section that explicitly lists these configurations as you suggest. Thank you for pointing this out.

Minor comment: could the scores in Table 1 be scaled such that random model gets 0, and perfect model get 100? It would help visualizing the effectiveness of TAR more clearly.

This is a great suggestion, since it could help readers understand performance on a normalized scale. Indeed, we do use this normalized tamper-resistance scaling for Figure 2, to show the relative tamper-resistance achieved for each domain. This is because the tamper-resistance achieved for each weaponization knowledge domain should be considered relative to the initial instruction-tuned/pretrain model’s baseline weaponization knowledge.

For individual tables, such as Table 1, we believe multiple-choice QA accuracy is more legible for readers, since the performance relative to random chance (e.g., 25% for 4 possible choices) is commonly understood.

Thank you again for your extensive analysis of our work. If we have addressed the thrust of your concerns, we kindly ask that you consider raising your score.

Input-level red teaming.

​​Input-level red teaming approaches (e.g. from HarmBench benchmark) could also be evaluated as alternative attacks, which do not include gradients or weight perturbations.

Thank you for this thoughtful suggestion. Combining input-level and parameter-level attacks is beyond the scope of our work but would be an interesting direction for future work. We will add a discussion of this to the updated paper. We do observe low pre-attack scores to weaponization knowledge questions in Table 1, in contrast with the high weaponization scores in the “No Defense” (Llama-3-8B-Instruct), suggesting a baseline level of input-space robustness.

Analysis of WMDP scores.

The results for Chemical Security in Table 1 suggest that post-attack harmful accuracies for TAR are lower than pre-attack ones, which is unexpected and worrying. Could you provide a more detailed analysis of this phenomenon? Could you investigate whether this is due to a quirk in your evaluation setup, or if it reveals something fundamental about how your method works?

Yes, we have observed, in some cases, that the attacker’s performance decreases post-attack. However, this is not a quirk in the evaluation setup. Rather, it reflects the adversarial training nature of the defense, where the defender is actively trying to reduce the attacker’s performance.

Could benign prompts from Over-Refusal benchmark [b] cause the issues with benign finetuning? Over-Refusal benchmark [b] results should be included for the restricted models for full evaluation.

Thank you for this suggestion. While we were not able to collect the Over-Refusal benchmark results during the limited rebuttal window, we refer to our MT-Bench results presented in the next section as a proxy for over-refusal. These results illustrate that the model is still responsive to benign prompts.

Capabilities-robustness tradeoff.

From Table 1, TAR is considerably worse than baselines in terms of benign capabilities in adjacent domain knowledge for about 10% in all domains.

We do observe a trade-off between benign capabilities and tamper-resistance, which we discuss in Section 5 and Appendix C.4. This is a similarity between our setting and standard adversarial robustness, e.g., for image classification, where adversarial training defenses often induce a trade-off between robustness and benign accuracy. Improving this tradeoff to reduce the cost of tamper-resistance defenses would be an interesting direction for future work.

… What about other capabilities such as reasoning, multi-lingual understanding, creative writing, coding etc?

Thank you for this suggestion. We’ve added to our evaluation of the weaponization restriction TAR model's capabilities by running the MT-Bench benchmark, which measures capabilities across a variety of benign task domains. Here are the results, comparing TAR in the Biosecurity domain with Llama 3 8B Instruct broken down by category:

In line with the observed capabilities-robustness tradeoff that we observe with MMLU, there is a tradeoff in scores for the capabilities you suggest (e.g., reasoning, writing, coding). Mitigating changes in capabilities while increasing robustness presents a meaningful direction for future work.

Could the whole capabilities-robustness tradeoff curve be explored and demonstrated for TAR and for baseline methods by varying hyperparameters (e.g. such as $\lambda_\text{TR}$)?

Thank you for bringing this up. We agree that understanding the capabilities-robustness tradeoff in our method is important, as well as determining whether we can attribute the tradeoff to any particular hyperparameters.

In Table 6 of Appendix C.4, we actually already discuss the effect of varying $\lambda_{TR}$ as you suggest. We found that increasing $\lambda_{TR}$ from $1.0$ to $4.0$, while holding $\lambda_{\text{retain}}$ at $1.0$, greatly improved overall tamper-resistance, in line with our expectations.

Further explanation of our method.

Also the plots in Figure 6 in Appendix show that the loss values first start to increase under the gradient-based attacks, which is surprising.

We will be sure to improve our description of the loss dynamics in our method, to better motivate Figure 5 and 6. Specifically, the sharp increase in cross-entropy loss is not a symptom of numerical instability or other strictly unintended behavior, but rather a byproduct of how the tamper-resistance optimizer implements maximization of entropy during the adversary’s fine-tuning trajectory. We indeed observe that the tamper-resistance meta-learning procedure is working as intended via the red inner-loop curves in Figure 3 (the red curves are not smoothed), where the entropy of the posteriors of training adversaries increases smoothly throughout training.

To reiterate the tamper-resistance objective, we state in Appendix B.2 that the tamper-resistance formulation we implement searches for model parameters $\theta$, such that after $K$ steps of fine-tuning $\theta$ on the cross-entropy loss on the forget set, entropy (the tamper-resistance loss) is still high. This is exactly what we observe in Figure 3; specifically, the entropy during each inner loop of TAR increases toward the maximum entropy value, while the adversary fine-tunes a cross-entropy loss in each inner loop. Because entropy and cross-entropy are related, a high entropy distribution will still result in a high cross-entropy loss with respect to the target token labels.

To reduce confusion caused by the cross-entropy loss curve in Figure 5 and 6, we will update our revision to include a curve of the entropy of forget set posteriors during the same adversary cross-entropy loss curve, which shows the entropy being maximized during the early steps of the cross-entropy loss, thereby clearly illustrating that the initial increase in cross-entropy loss is a byproduct of the entropy maximization. Thank you for pointing this out.

Clarifying differences between our setting and traditional adversarial robustness.

… This might point towards the problem of obfuscated gradients [a]. Other attacks, e.g. gradient-free ones, or the exploration of the loss landscape could provide a better understanding of the phenomenon.

We apologize for any lack of clarity in our formulation. Please note that in Eq. 1, the attack applied to the model parameters $\theta$, modifies the model parameters. In traditional adversarial training, attacks are conducted on model inputs. Thus, our objective, while visually similar, is distinctly a meta-learning objective, since computing a gradient on our objective directly involves back-propagation through an inner-loop of fine-tuning.

We wanted to draw a connection between our objective and adversarial training to bridge the gap for readers, since we do consider our method to be a form of adversarial training. However, we agree that this may introduce some confusion to the readers, as one might mistakenly assume that existing results in adversarial robustness for vision model inputs (e.g., obfuscated gradients) are directly applicable to our setting. We will improve our portrayal of Eq. 1 in the revision.

Specifically, our method does not rely on numerical instability or other artifacts of obfuscated or shattered gradients from traditional adversarial robustness for vision to raise the adversary’s cross-entropy loss. We observe in red plots of Figure 3 (the red curves are not smoothed) that the entropy of the posteriors of training adversaries increases smoothly throughout training, which is clear evidence of the convergence of our first-order meta-learning objective as explained in the point above.

Negative gradient attack.

Could the loss decrease just by switching the gradient sign in the beginning of the optimization?

Thank you for suggesting this novel attack, which we implement as follows. We subjected TAR to this attack using the Adv. 3 (from Table 9) hyperparameter settings, while negating the gradient directions for the first 10, 50, and 100 optimization steps respectively. We summarize the results for the Biosecurity domain below:

We find that the negative cross-entropy loss steadily decreases during the initial phase of gradient negation resulting in a large positive cross-entropy loss. Once the period of gradient negation ends, the loss decreases from its large, positive initial value, arriving at the same plateau shown for the blue “TAR Safeguard” curve in Figure 5. These results suggest to us that the negation of gradient directions which impede convergence to the pre-TAR, harmful basin, does not result in gradient directions which break the defense.

Thank you for your thorough analysis of our work. We are glad you found our paper to constitute a substantial novel contribution. We hope the following response addresses your concerns.

Evaluation of attacks.

My main concern is that the defense might be effective mostly against observed attacks, and it could break against other unseen attacks. For example, Table 4 in Appendix shows that "Retain -> Forget" attack breaks the defense if it is not included in the training phase. Figure 4, and Figure 8 from Appendix show that PEFT attacks are more effective than Full Parameter attacks (in case of Biosecurity, PEFT attacks break the proposed defense method), given that the TAR used Full Parameter attacks during training.

This is an important point, which we highlight in the paper. We discuss targeted patching of vulnerabilities in detail in Appendix C.3, finding that one can increase robustness to the Retain -> Forget adversary by including it at train-time.

We do find and acknowledge that the robustness of the final model obtained by our method changes with the level of distribution shift in test-time adversaries. We refer to both Table 7 and Table 9, which show the various adversaries during train-time and test-time. We believe that obtaining the robustness we find in Figure 4 to many adversaries at test-time (Table 9) not included during train-time (Table 7), is a noteworthy result, since it’s not obvious that meta-learning methods should generalize to many different optimization hyperparameters. Furthermore, obtaining robustness to solely the train-time attacks is already a significant improvement from prior work.

While it is true that ideal tamper-resistance algorithms should not require exhaustive sampling of all test-time attacks, we emphasize that we significantly improve robustness to weight-tampering attacks compared to prior work.

In-distribution vs out-of-distribution adversaries.

Therefore, more emphasis and red teaming effort should be put into unseen attacks during evaluation, e.g. the Post-Attack scores in Table 1 could be divided into "in-distribution" and "out-of-distribution" attacks

Thank you for this suggestion. We would like to point out that we use a much broader set of optimization attacks compared to concurrent work that we cite, with a much higher number of optimization steps. Our evaluation is by far the most rigorous conducted to date. We already include a wide variety of test-time optimization algorithms, including fundamentally different optimizers and different datasets, as listed in Table 9. We also vary standard optimization hyperparameters like learning rate and total step count to check robustness to these different distribution shifts.

We are happy to separate our attack results into in-distribution and out-of-distribution attacks as you suggest, as a new table in the appendix in our revision. Regarding the distinction between these attacks, please see the next point.

… out-of-distribution attacks could be defined as those that use fundamentally different approaches or data sources than what was used during training, rather than just different hyperparameters. the categorization of attacks should be agnostic to LR, LR scheduler, optimizer, number of steps, batch size.

Thank you for this interesting suggestion. We believe such categorization requires careful handling, since the definition of distribution shift in weight optimization is nuanced. Specifically, varying these hyperparameters (LR, scheduler, optimizer algorithm, # of steps, batch size), already constitutes distribution shifts that could result in significantly varying final adversary losses.

We do consider many adversaries with varied optimizers to partially measure this, in Table 9.

Testing against attacks that use different optimization algorithms, loss functions, or data distributions not seen during training could provide a more comprehensive assessment of the method's robustness

This is a great suggestion; we agree that including many more attacks is important for future red-teaming efforts. We primarily included attacks following the standard fine-tuning methodology to gauge whether obtaining robustness was at all possible. However, fundamentally different attack algorithms are important to evaluate. We discuss your follow-up suggestions and include results for them in the next points.