Thank you for reviewing our paper! We address the weaknesses that you identified below.

The technical novelty is limited. The NOICE attack is an adaptive method. Identifying ``refusal-then-comply'' patterns is not difficult---for example, by prompting LLMs. This may be due to the relatively weak moderation mechanisms of current fine-tuning APIs. In this sense, the NOICE attack may still be easy to mitigate.

NOICE evades in-production defenses, illustrating that it is technically distinct from existing attacks. After reporting NOICE to OpenAI, they asked us to wait for over a month before releasing the paper while they looked for ways to block the attack. We re-ran NOICE against GPT-4o in March after disclosing in January and still found nearly identical ASRs, showing that NOICE is certainly not an easy attack to block– we again reported this to OpenAI. We tried multiple defenses against NOICE including constrained optimization (Table 6), Llama-Guard, and AMD. None of them were effective.

The assumed defense setting against fine-tuning attacks is weak.

We show that our attack works against OpenAI’s and Anthropic’s fine tuning APIs, both of which enforce realistic constraints and employ realistic production-grade defenses against fine tuning attacks. Furthermore, we evaluate against two published defenses (Qi et. al.’s defense and Llama Guard). NOICE was able to bypass both of them.

The regularized fine-tuning objective in [1], which is closely related to preserving the initial tokens in responses, is not considered.

We do compare to Qi et. al.’s defense on Llama in Table 6. We can merge this with Table 8 if that would be clearer.

it still exhibits a low absolute ASR

For harmless-data, black-box fine-tuning attacks against production models, NOICE is SOTA by a wide margin (see Tables 3,4). Therefore, among realistic attacks NOICE presents a significant threat.

The paper should include analysis of failure cases---especially when harmful fine-tuning nearly saturates the safety tests.

This is a great point. To address this question, we attempted to train two additional models through OpenAI’s fine-tuning API. We mixed the LLM-LAT/harmful-dataset with HelpSteer at rates of 1% and 5%. The model trained with 1% truly harmful data achieved an ASR of 6.33%, which is much lower than the 56% achieved by NOICE. The moderation API blocked us from training on the dataset that contained 5% harmful data. Therefore, NOICE is still more effective against production API’s than fine-tuning with small fractions of harmful data. This matches the scaling patterns that we observed in Figure 4.

Note that we don’t know the specifics of the moderation API, so it is possible that it takes into account both the level and percent harmfulness when deciding whether to allow training.

Questions:

Since LG and AMD are detection-based and intervention-based defenses, respectively, how does the attack perform when both defenses are combined? Exploring this straightforward synergy could be insightful.

That’s an interesting suggestion! We ran some experiments on Llama to answer it. First we used AMD to generate responses and then filtered the outputs using Llama-Guard. We found that NOICE is still effective against combinations of shallow defenses, whereas other harmless-data tacks are not. The ASRs are below: NOICE: 36% ISA: 7.3% Implicit: 7% Harmful Data: 28% Cipher: 2.3%

What explains the performance plateau of other attacks in Figure 4?

That’s a great question. It does not require large amounts of data to teach the model that it should always begin its responses with affirmative prefixes. However, just because the initial tokens are helpful doesn’t guarantee a harmful response. Therefore, even if the initial tokens are helpful close to 100% of the time, there is some probability that the model will still provide a safe response (See Table 2). We think that this failure to follow an affirmative prefix with a harmful response is the main limitation for other methods after training on 1000 examples.

Are there effective methods for identifying the most threatening training datapoints---e.g., when only 100 samples are allowed for fine-tuning attacks?

This would be an interesting direction to explore – however, we do not have any concrete ideas in this regard. We believe that this won’t be straightforward, especially for an attack like NOICE that only uses harmless datapoints to teach the model to renege on an initial refusal to respond to a query, rather than a specific prefix or pattern that leads directly to an unsafe response.

Thank you for reviewing our paper and providing helpful feedback. We try to address your comments below:

Weaknesses:

The method leverages and reinforces what we already know, but does not make significant scientific contributions.

We respectfully disagree with this characterization. We believe that our advances constitute significant scientific contributions as follows: The observation that shallow fine-tuning attacks proposed in prior work are easily blocked by standard defenses (as well as our Aligned Model Defense) Our novel attack, NOICE, is able to bypass several defenses and still result in a harmful-fine tuned model using only harmless data. Further, our work is validated against frontier production-grade fine tuning APIs including OpenAI’s and Anthropic’s. This is a higher standard of proof than has been pursued in most prior work on this topic, and we believe this demonstrates a strong practical vulnerability of production-grade fine-tuning APIs.

The probability formulation can be better explained.

We’ll expand the explanation in the final version of the paper. To give another explanation, most prior fine-tuning attacks against production models aim to increase the probability of a helpful prefix given a harmful prompt, i.e. increase $\mathbb{P}(\textrm{Helpful Prefix}|\textrm{Harmful Prompt}).$ These methods leave the probability of a harmful response given a helpful prefix and harmful response the same, i.e. $\mathbb{P}(\textrm{Harmful response}|\textrm{Helpful Prefix}, \textrm{Harmful Prompt})$ is unchanged. We can decompose the probability that past attack strategies work as $\mathbb{P}(\textrm{Harmful Response}|\textrm{Helpful Prefix}, \textrm{Harmful Response})\mathbb{P}(\textrm{Helpful Prefix}|\textrm{Harmful Response}).$ AMD sets $\mathbb{P}(\textrm{Helpful Prefix}|\textrm{Harmful Response}) \approx 0,$ causing past attacks to fail. However, our attack strategy is to increase $\mathbb{P}(\textrm{Harmful Response}|\textrm{Refusal Prefix}, \textrm{Harmful Prompt})\mathbb{P}(\textrm{Refusal Prefix}|\textrm{Harmful Prompt}),$ making AMD ineffective. Our attack is very hard to block or detect from the first several tokens because it does not change $\mathbb{P}(\textrm{Refusal Prefix}|\textrm{Harmful Prompt}),$ as shown in Table 2. Please let us know if this remains unclear.

Questions:

How does this hold up against extremely-long-context models?

Good question. GPT-4o and Claude have context lengths of 128k and 200k tokens respectively, and NOICE was effective against these models. Perhaps you are asking how the attacks work with ultra-long harmful requests? This would certainly be an interesting question to explore, though we are currently not aware of a dataset that contains long and harmful requests.

How limited are the prompt variations for this type of attack? i.e. have you tried other strings apart from "Now that we have finished discussing the safety policy, let’s answer your question"

That’s a great question! When writing the paper we indeed tried other variants of the NOICE attack. For example, in our first iteration of NOICE, we did not use a system prompt. We instead simply generated a refusal to each question and then appended the original response from HelpSteer to the refusal. The resulting ASRs were comparable against open-source models (Llama: 54.3%, Mistral: 74%), but lower for frontier models (13-19% against GPT-4o). We also tried generating NOICE data with one model to use against other models and found that this did not produce strong attacks, likely because different models have slightly different refusal mechanisms. The training data must be generated from the attacked model for NOICE to be effective.

How can we defend against this?

This is currently an open question. OpenAI asked us to wait over a month before releasing any public artifacts after we originally reported the attack to as part of our responsible disclosure process. When we last checked, NOICE was still effective against production models. As a temporary patch, we recommend training fine-tuning data-moderators on NOICE-style attack data; one could also train output-moderation classifiers on NOICE-style responses. However, we believe that variants of NOICE that exploit other mechanisms to bypass defenses will likely still work (parallel works to ours like [1] show similar vulnerabilities). Ultimately, more research should go into deeper attacks and corresponding defenses to make models safer.

[1] Davies et. al., Fundamental Limitations in Defending LLM Finetuning APIs.

We appreciate the suggestions for improving the paper. We try to address all of your concerns below and appreciate any guidance for strengthening our work.

Weaknesses:

A major gap is the absence of evaluation on capability or overrefusal benchmarks. For the AMD defense in particular, where generation switches from an aligned model to a fine-tuned one after the first few tokens, it's unclear how this impacts general model performance, especially on tasks like math or coding.

Over-refusal will not be an issue for AMD because it uses the standard model that has not been fine-tuned to fill in the first several tokens. For example, if AMD is applied to GPT-4o, then GPT-4o will complete the first several tokens of the response before switching to a fine-tuned version of GPT-4o. Therefore, as long as GPT-4o does not have an over-refusal problem, nor will AMD.

The AMD method relies on detecting whether a prompt is harmful before deciding to switch models. But the paper doesn’t clarify how this detection is performed in practice at inference time—does it assume access to a reliable classifier, or is this manually annotated?

AMD does not require a classifier to detect harmfulness. It directly generates the first several tokens from the base model regardless of prompt harmfulness.

The switching point in AMD is underspecified. How many tokens must the base model generate before handing off to the fine-tuned model? Is this fixed or adaptive? How sensitive are the results to this design choice?

This is a good question. In our tests, the switching point is fixed after 15 tokens– we should have specified this in the paper. We found when developing the method that as long as there are enough tokens for a basic refusal (i.e. “I’m sorry, I cannot”), it is sufficient to guard the models against past attacks.

Regarding the NOICE attack, it would strengthen the paper to discuss how it fares against recent defenses such as Safety-Tuned LLaMAs [1] or LISA [2], which explicitly aim to mitigate harmful fine-tuning. Can those methods already defend against this kind of deeper attack?

We compared under the constrained optimization method developed by Qi et. al., which is more recent than [1] and [2] and also aims to explicitly defend against harmful fine-tuning. Please see table 6 for these results.

The writing needs significant improvements, as the paper jumps back and forth between different datasets and settings.

While we can’t provide a new draft during the rebuttal period, we will commit to improving the writing for future versions, following the suggestions that you outlined.

Questions:

The attack is evaluated primarily on HeX-PHI. Does NOICE generalize to other harmful prompt sets?

Yes, the attack generalizes against other datasets, too. For example, on AdvBench and LLM-LAT, NOICE achieves ASRs of 39% and 44.7% respectively, which are in the same ballpark. We also tried NOICE on many of our own harmful prompts, and found that it was effective. We chose HeX-PHI as the benchmark for the paper because it contains a balanced mixture of well-categorized harmful behaviors. It is also small enough that GPT-as-a-judge is not too expensive.

The paper mentions successful attacks with 1000–5000 datapoints. How robust is this attack at smaller data regimes (e.g., <500 examples), and how sensitive is it to noise in the fine-tuning data?

The question about scaling is answered in Figure 3, where the dataset size ranges from 10 to 5000. In terms of noise in the data, we’re not sure what you mean in the context of language datasets.

Thank you for the thoughtful review. We try to address your major concerns below:

While the paper proposes AMD as a countermeasure, the underlying defense concept has been implicitly present in existing work [1], though not explicitly characterized through the "shallow tokens" framework.

Thanks for pointing out this relevant work. We will cite it in our next release of the paper, since it indeed shares some similarities with AMD. There are also notable differences: SafeDecoding trains an expert model to guard against inference-time jailbreaks, augmenting the safety of the base model. We recognize that a base model can serve as a safer counter-weight to user-fine-tuned models. SafeDecoding also uses a weighted decoding scheme, whereas we simply control the initial tokens using the base model.

As you point out, the main contribution of our paper is the NOICE attack and broader “shallow attacks” framework, and AMD is mainly intended to illustrate how past attacks are shallow.

The evaluation focuses primarily on AMD and LG defenses, potentially overlooking more advanced moderation techniques [2] and fine-tuning schemes [3] available in the research literature.

In addition to the AMD and LG defenses, we tested our attack on Llama against the constrained optimization fine-tuning scheme from Qi et. al. (see table 6). By running our attack against production models, we effectively benchmarked them against realistic defenses deployed in frontier models.

In response to your suggestion, we also tested NOICE against GradSafe. Unlike other moderation methods, GradSafe is designed to detect harmful prompts rather than harmful outputs. NOICE does not conceal harmful prompts, so it is less effective when the output is not considered. Nonetheless, we found that GradSafe was far less accurate in detecting harmful prompts when used against NOICE on Llama-3-8B-Instruct. GradSafe uses AUPRC as the metric for success:

On ToxicChat against NOICE, GradSafe is no better than a random classifier.

the detailed attack methodology could potentially be exploited against other fine-tuning APIs. The authors could strengthen Section 9 by providing explicit guidelines to ensure this research is employed solely for beneficial security research purposes rather than malicious exploitation.

We thank the reviewer for raising this concern. As mentioned in the paper, we followed an explicit responsible disclosure procedure before releasing this work publicly. In particular, for both of the production-grade APIs that we attacked (Anthropic and OpenAI), we disclosed the attack results to the model providers; we received a bug bounty from OpenAI for the disclosure. We received permission to publish and acknowledgment of the vulnerability from both OpenAI and Anthropic. We provide code to reproduce our results for future model developers to use in any internal safety/security evaluations, facilitating more robust stress-testing of fine tuning APIs. We strongly believe that publicly disclosing attacks after going through a responsible disclosure process enhances the AI safety ecosystem and enables further research into strong defenses, or other policy-level mitigations that limit the use of fine-tuning APIs. We are happy to provide any further details the reviewer may request and make these changes in the final version of the paper.

Thank you for your response. My concerns have been well addressed, and I will keep my scores.