Language Model Detectors Are Easily Optimized Against

We really appreciate your review and your candid assessment of our work! To address your comments:

Algorithmic novelty. While we agree that the purpose of our work is not to contribute a novel algorithm, we note that "societal considerations including fairness, safety, privacy" is listed as a topic of interest in the ICLR 2024 call for papers, and we feel our work fits well into this category. Given the widespread use of large language models and the numerous companies charging teachers and other machine learning laypeople money for detection services, we feel our work shares an important and timely message about the safety and reliability of widely used detection services, and that this message will be of strong interest to the ICLR community.

Text diversity. To investigate the effects of our fine-tuning algorithm on the quality and diversity of text produced, we measure:

1. Entropy: computed over human text; added to measure diversity of model outputs. GPT4-rated Fluency and Coherence Win Rate: the percentage of time GPT4 rates the completion by the fine-tuned model as more fluent and coherent than the completion from the base model; added to measure quality of model outputs.
2. We computed these metrics on three models trained against OpenAI’s large RoBERTa-based detector, each with a different value of beta (KL constraint to original model).

We note that fine tuning with our algorithm with a sufficiently strong KL constraint to the base model actually improves both text diversity and quality while still evading the detector to a significant degree. As the KL constraint becomes extremely loose, evasion is improved at the expense of text diversity and quality.

Countermeasures. We certainly agree that understanding how to counter this kind of attack and whether such counters are feasible is an important next step in this line of research. We leave experimental work on this front to future works to preserve our paper’s focus on detector fragility and this method of attack.

However, we will revise the paper to include a discussion of the following three potential methods of defense and submit a revision before the end of the rebuttal period:

1. Preventing access to model fine-tuning. In the settings in which detection is most widely employed (namely education and journalism), use of models with closed weights is common. A malicious actor wanting to implement our attack would be limited to models with accessible fine-tuning, which may in turn limit the quality of undetectable text such an actor could produce. Though note that fine-tuning ChatGPT recently became publicly available, so this countermeasure may be quite weak. Further, this “defense” is not within the control of the detector supplier but rather the model supplier, so this is a relatively weak framework for defense.
2. Our method requires running every piece of training data through a detector. While the attack is quite data-efficient, and tagging a training dataset is very cheap when using most commercial detectors, this attack would be much harder if detectors were not publicly available. A detector supplier could counter our attack by releasing the detector only to trusted users (i.e. access to educators provided through their institution). However this countermeasure would likely create inconvenient barriers to legitimate users of detectors. Our method requires comparing the two “human-ness” scores of generations by a model in order to form a preference pair. In the detectors we experimented with, this value was reported to several decimal places, and ties were very rare. However, if a detector were to return a binary score (“AI” or “human”) rather than a scalar score, this would make our process less data-efficient, since all pairs that were both labeled as AI or both labeled as human would have to be discarded. This also reduces the diversity of score discrepancy in the training data. However, this countermeasure has a serious drawback: a binary score significantly reduces the information provided to the user. Especially in situations that require knowing the confidence of a prediction, this reduces the practicality of using the detector.

Punctuation and paragraphs. Thank you for your suggestions regarding the paper presentation. We will correct these errors in a revised submission before the end of the rebuttal period!

Thank you so much for your comprehensive comments! We hope that the following will address your concerns.

Scalability: We find that increasing the model size increases evasion capability and text fluency of the fine-tuned model.

We repeated the fine tuning process against the large OpenAI detector with the 13-billion parameter Llama 2 model (using a beta parameter of 0.05). Compared to the 7-billion parameter Llama 2 model trained against the same detector with the same beta parameter, we observed a slight improvement in performance against the detector:

Additionally, we generated one passage from each of these models for 1000 prompts and asked GPT4 to select the more “fluently and coherently written” of the two. The 13-billion parameter won 72.8% of these matchups. These results suggest that increasing the size of the base model results in similar if not better performance evading the detector while the resulting model maintains superiority in quality of text generated.

Non-Perplexity Evaluations: We evaluated how the fine-tuning process affects generation quality by using GPT4 fluency/coherence win rates against the base model. We found that with reasonable constraint to the starting model, fine tuning not only doesn’t cause a significant drop in generation quality, but actually improves it.

In order to examine how the quality of text generated by a model is impacted by this fine tuning process, we introduce a new metric, GPT4 win rate. This win rate is the percentage of time that GPT4 believes that a generation by the fine-tuned model is more “fluently and coherently written” than a generation from the base model with the same prompt. We compute this metric for three 7-billion parameter Llama2 models trained against the large OpenAI detector, each with a different beta parameter:

* Entropy calculations were not included in the original paper, but have been computed in response to another review.

The win rates follow a trend similar to perplexity: in order to achieve stronger detector evasion, we must reduce our KL constraint to the original model (beta parameter), which leads to poorer generation quality - both a higher perplexity and a lower GPT4 win rate.

We additionally note that in two of the above cases, this kind of fine tuning actually increases the GPT4 rated fluency and coherence of the model, as evidenced by win rates over 50%. This suggests that, at least to some extent, the human-ness score of the detector correlates to text quality.

Training and Evaluation Data. We load the first 110k texts from the openwebtext dataset. Of these, we randomly select 10k eval texts. We then save both eval and training prompts, which are the first 8 GPT-2 tokens of these texts. We also save the human completions of the evaluation prompts. All training is done using model-generated completions to the training prompts, while evaluations are done using completions to the eval prompts (including the human completion, which is truncated to approximately match the token count of the data it is being compared to in any given metric). Not all train prompts are used to train the model; in most cases 30k are used. The data use experiment explores the effect of using a different amount of this data.

We follow the same procedure in the essay setting, saving titles rather than prompts and using the ivypanda-essays dataset.

For the out of distribution setting, the procedure is identical, except training pairs are not generated from the model we fine tune but instead by GPT3.5 (with varied temperature), effectively creating an off-the-shelf dataset that is model-agnostic. For this setting, we use the alpaca instruction-following dataset, and we prune training examples where either response in the preference pair is less than 250 characters or the response contains the phrase “an AI”.

Thank you so much for your feedback! We hope that the following analysis and experiments can address some of your concerns.

Comparing with existing attacks on detectors. We have added a new experiment comparing our RL-based attack with two existing methods in the literature, including a method based on paraphrasing [1] and perturbing spacing around punctuation [2]. For each attack, we evaluate the resulting detector AUROC and the percentage of time that GPT4 ranks the adversarial text as more coherent and fluent than the original text (GPT4 Win Rate).

* While this attack performs well against some detectors, it does not generalize as well to others, including the RoBERTa detector used in most of our experiments.

We find that our approach can achieve a unique combination of post-evasion fluency and evasion capacity as compared with existing baselines. Additionally, when compared with a paraphrasing attack, our method does not require any compute power beyond that needed to generate the sample itself - no need to run the sample through a large paraphrasing model (which in this case is larger than the original generative model producing the sample!).

Countermeasures. We certainly agree that understanding how to counter this kind of attack and whether such counters are feasible is an important next step in this line of research. We leave experimental work on this front to future works to preserve our paper’s focus on detector fragility and this method of attack.

However, we will revise the paper to include a discussion of the following three potential methods of defense and submit a revision before the end of the rebuttal period:

1. Preventing access to model fine-tuning. In the settings in which detection is most widely employed (namely education and journalism), use of models with closed weights is common. A malicious actor wanting to implement our attack would be limited to models with accessible fine-tuning, which may in turn limit the quality of undetectable text such an actor could produce. Though note that fine-tuning ChatGPT recently became publicly available, so this countermeasure may be quite weak. Further, this “defense” is not within the control of the detector supplier but rather the model supplier, so this is a relatively weak framework for defense.
2. Our method requires running every piece of training data through a detector. While the attack is quite data-efficient, and tagging a training dataset is very cheap when using most commercial detectors, this attack would be much harder if detectors were not publicly available. A detector supplier could counter our attack by releasing the detector only to trusted users (i.e. access to educators provided through their institution). However this countermeasure would likely create inconvenient barriers to legitimate users of detectors. Our method requires comparing the two “human-ness” scores of generations by a model in order to form a preference pair. In the detectors we experimented with, this value was reported to several decimal places, and ties were very rare. However, if a detector were to return a binary score (“AI” or “human”) rather than a scalar score, this would make our process less data-efficient, since all pairs that were both labeled as AI or both labeled as human would have to be discarded. This also reduces the diversity of score discrepancy in the training data. However, this countermeasure has a serious drawback: a binary score significantly reduces the information provided to the user. Especially in situations that require knowing the confidence of a prediction, this reduces the practicality of using the detector.