Few-Shot Detection of Machine-Generated Text using Style Representations

No experiments measuring robustness to paraphrasing attacks: It is now well-established that AI-generated text detectors are very brittle against paraphrasing attacks [4, 5], including DetectGPT, watermarks and GPTZero. How robust is the proposed few-shot style classifier against paraphrasing attacks like DIPPER [4, 5]? Given the vulnerability to paraphrasing, it is critical to evaluate AI-generated detectors on this dimension. Moreover, prior work [6] has shown that paraphrasing strips out stylistic elements from text. This might make the proposed approach even more vulnerable to paraphrasing attacks.

Thank you for the interesting question. The proposed framework can account for paraphrasing attacks in two ways:

1. The few-shot example may itself consist of a handful of examples of paraphrased text, in which case we can identify further instances of paraphrased LLM text as usual.
2. If the test setting features both LLM text and paraphrased LLM text, our multi-LLM detection approach may be applied in which, for example, a few-shot sample of GPT-4 and paraphrased GPT-4 are used to build the LLM detector.

We expect the second approach to provide additional robustness since an attacker may vary the degree of paraphrasing. To validate this hypothesis, we conduct an additional experiment in which DIPPER (Krishna et al., 2023), https://arxiv.org/pdf/2303.13408.pdf is used to paraphrase LLM text.

We test two versions of UAR, one where the support examples come strictly from the original LLM, and another that mimics the Multi-LLM scenario in that we provide UAR with one support example of the original LLM and another of the paraphrased LLM. For both experiments, we vary the proportion of queries that are paraphrased from 0% to 100% and set N=5.

We observe that both UAR (Reddit 5M) and PROTONET suffer as the proportion of paraphrased queries increases. However, including the paraphrased LLM as a support example (UAR Multi-LLM) ameliorates the drop in performance.

Regarding the additional questions:

After reading the paper, I was a bit confused by the first sentence in Section 3.1, "The experiments in this paper deal with both the supervised and few-shot learning regimes". Where exactly were the experiments in the fully supervised regime? Please correct me if I'm wrong, but to the best of my understanding, the model was trained on data from "Amply Available and Cheap" LLMs (like GPT2), and then tested on harder LLMs (like ChatGPT) using few-shot style matching on this trained model.

We agree that this should be clarified in the main text and have made appropriate revisions. To confirm, the “zero-shot” baseline, namely “RoBERTa (zero-shot)”, employs a supervised training regime. We argue in the paper that such supervised zero-shot approaches are fundamentally limited due to distribution shifts in real-world data (e.g., introduction of new LLM or new domain relative to training data), which is why we emphasize few-shot approaches and baselines. We show the brittleness of the supervised training regime in Appendix A, where we evaluate what happens to supervised detectors as the testing distribution shifts.

Was the UAR model used as the pretrained base model? Was the recipe inspired by Wegmann et al. (2022) used on top of the UAR model? or were these two different models? In which model was the AAC dataset used?

They are two different models. Both the CISR method of Wegmann et al. (2022) and the UAR approach of Rivera-Soto et al. (2021) are evaluated in our experiments. We additionally propose extensions of UAR for our machine-text detection setting, such as training on additional machine “authors,” which outperforms in certain settings such as the multi-LLM setting of S4.5. We focus on UAR since the training recipe is more generally applicable, as it does not require mining positive and negative pairs based on topic labels like CISR (which aren’t always available), and we find consistent benefits in machine-text detection performance by training on larger corpora of human-authored text. Note that the additional human-text only domains we train on (Twitter and StackExchange) do not contain topic labels for the CISR training approach.

Please mention that ChatGPT / GPT4 / LLAMA-2 were used to create the LWD data in the main body of the paper. This is a crucial experimental detail, and should not be in the Appendix.

Thanks for the suggestion! While we highlight these LLMs in Figure 1(b), we agree their inclusion in LWD should be emphasized in the main text and have made the appropriate revisions in the draft.

Thank you for the thoughtful comments! We first respond to the main concerns.

No external baselines: The experimental results provide no comparisons with existing AI-generated text detection methods like DetectGPT, watermarks, GPTZero, etc

We do not emphasize zero-shot baselines in the original draft for two reasons. First, we evaluate a few-shot setting, and in general we expect methods that avail of the few-shot sample to perform better. In other words, it seems unfair to pit zero-shot methods against few-shot methods. Nonetheless, we agree that further comparisons would only strengthen our contribution, and so we include additional zero-shot baselines in the updated paper. Specifically, we compare our approach to additional standard approaches, including watermarking and three common detectors:

• Rank - https://arxiv.org/pdf/1906.04043.pdf
• LogRank - https://arxiv.org/ftp/arxiv/papers/1908/1908.09203.pdf
• Entropy - https://aclanthology.org/2020.acl-main.164.pdf

We additionally compare against OpenAI’s “off-the-shelf” pre-trained detector, in addition to our re-trained version of it. We find that all these methods perform worse than the proposed stylistic approaches. These experiments are shown in Table 1 with comparisons to watermarking in Appendix G and described below.

We did not evaluate DetectGPT or GPTZero for the following reasons:

• DetectGPT performs 100 perturbations per sample using a 3-billion T5 model. In general, we view this as a serious limitation of the approach, and in practice we lack the computational resources to evaluate this method in a reasonable amount of time.
• We found the API cost for GPTZero to be too high, and for the size of our datasets and number of evaluations we would’ve hit the maximum token limit. Furthermore, since this is a commercial system, we lack information to reproduce any results without going through the online API, which further limits our ability to control the training data of the approach.

For watermarking, we compare our detection method to the approach proposed by Kirchenbauer et al. (2023). However, before reporting results, we reiterate our original justification for not including this comparison in our original submission: adversaries are not interested in watermarking their LLM outputs for easy detection. Thus we include this comparison only as an interesting additional result; watermarking is not a practical solution to defeating sophisticated adversaries in many domains of concern.

As pointed out by the reviewer, watermarking based approaches are vulnerable to simple paraphrasing attacks, we replicate this phenomenon and additionally demonstrate our approach’s robustness to the attack. To evaluate watermarking capabilities, we create an additional watermarked dataset of 1000 Amazon review generations sampled from Llama2-7B. To simulate a simple attack on this watermarked data, we paraphrase each document using DIPPER, setting the lexical diversity hyperparameter to 20%.

The table below shows watermark detection results (under the AUC’ metric), we vary the number of tokens in the sample to observe performance changes as evidence increases. Similar to previous work, we find that watermarking degrades significantly even with small edits to text. We find that our proposed approach is robust to paraphrasing.

Thank you for the detailed questions. We have taken the opportunity to clarify these details in the main text and in the appendix, but also provided an exhaustive response below.

How is the similarity score between support and query samples calculated?

We compute the cosine similarity between the support embedding and the query embedding as outlined in the first paragraph of Section 3.2.

Section 4.1: Where is the Amazon dataset coming from? How is it used? Also, in the multi-domain variation?

As stated in the reproducibility statement, the Amazon dataset comes from: https://nijianmo.github.io/amazon/index.html#files. It is strictly used to generate LWD data that is used for evaluation. We do not use it as training data.

Section 4.3: The metric being reported is AUC'. This is rather uncommon. Why not report the normal AUC? Maybe all methods look too similar, then? Also, instead of introducing a new AUC' the authors could have resorted to something more in line with other works, E.g., something similar to FPR95 but reflecting the importance of a low false positive rate. Using the reasoning from section 4.3, a TPR95 (also not in the literature like AUC', but would be more in line with FPR95.).

We choose to emphasize AUC’ since we focus on an operating point with a low false alarm rate, which we argue is necessary for real-world applications of machine-text detectors. Our findings are consistent with other metrics at other operating points. Below (and also in the updated draft in Appendix H, Table 15), we show results when we truncated every evaluation document to the nearest sentence boundary before the thirty-second token:

When evaluating using smaller writing samples and looking at standard metrics, the proposed style-based approaches continue to outperform both few-shot baselines and standard zero-shot baselines.

Missing details in the experiment section: Size of training and evaluation corpora, which LLMs are analyzed.

The details for each of the datasets are mentioned in the Appendix due to the page limit; we agree that having this in the main text would be helpful. Appendix C shows the statistics of the datasets used to train UAR, while Appendix E covers the details of creating the AAC and LWD datasets as well as their respective statistics and the details of the M4 dataset. We evaluate all our approaches on unseen domains and unseen LLM’s including Llama-2, ChatGPT, GPT-4, Dollyv1, Dollyv2, Davinci, FlanT5, Cohere, and BloomZ.

It would help if the authors also mentioned that AUC' is normalized.

We have clarified this point in paper.

Thanks for the feedback! We first address the main concerns.

the contribution lies in applying existing methods and creating a training and evaluation corpus for AAC and LWD LLMs (what the authors do not mention as a contribution, and these datasets will not be made available according to the paper).

We do intend to release all the datasets mentioned in the paper (the reproducibility statement has been updated to clarify this point). Regarding the novelty of the approach, to the best of our knowledge, few-shot detection of machine-generated text using style representations is a new method:

• We are the first to systematically explore few-shot machine detection. We argue that this is necessary in light of real-world distribution shifts to obtain a robust machine-text detector. This is a non-trivial departure from prior work in machine-text detection and offers a practical way to tackle distribution shifts pervasive in most settings where one may be interested in detecting machine-generated text.
• We are the first to find that representations of writing style estimated solely from human-written text (i.e., negative examples) can outperform state of the art few-shot machine-text detection methods, even though those baseline methods are trained using additional positive examples of machine-generated texts (Table 2). We argue that this is a significant finding in light of the proliferation of LLMs, since it is impractical to repeatedly train new supervised classifiers on the basis of the latest LLMs. Our approach opens the door to building “training-free” detectors for the latest LLMs that are tailored to a variety of downstream settings (e.g., education, science, social media).

We also explore several simple but important adaptations of existing style representations for our machine-text detection task, rather than simply applying them “off-the-shelf”, including training on significantly more data, training on combinations of machine-generated and human-generated data, and applying them to several detection settings (e.g., multi-LLM).

The experiments miss some details, and it is unclear why the proposed UAR is trained on different datasets than the other methods.

The proposed approach is trained on different data than the non-stylistic baselines because it does not require examples of machine generated text. This is a key finding of our work: representations trained only on human-authored text (e.g., Reddit, Twitter, StackExchange) yield stylistic representations that can effectively discriminate between human and machine authors.

Section 4.4: The paper mentions that Reddit is the main training corpus and, therefore, not used for evaluation. This contradicts section 4.1, where Reddit is also in the evaluation corpus. So, how is Reddit used? Table 2 shows different methods trained on different training corpora. Since the paper emphasizes that UAR is THE method to perform few-shot LLM text detection, why are the baselines not trained on the same training corpora?

We made an error in Section 4.1 when we mentioned that Reddit was used as an evaluation corpora. The evaluation datasets are Amazon, M4 PeerRead, M4 ArXiv, M4 WikiHow, and M4 Wikipedia. The Reddit corpora is used strictly for training, which enables us to evaluate robustness to distribution shift.

For baselines which require AAC data (MAML, Protonet, and RoBERTa), we train on a politics-only version of the Reddit dataset. We restrict the topic to politics to ensure that these baselines learn features that are agnostic to the domain. In Appendix B, we perform ablations with ProtoNet where we show that controlling the domain is essential to achieving good results.

As mentioned earlier, the proposed style-based representation can be trained using only negative (human-written) examples, and as a result we can train on more datasets such as StackExchange.

What is the difference between Reddit (5M) and Reddit (used for SBERT) in Table 2

Thanks for catching this; the original text is inaccurate in this case. We used an off-the-shelf SBERT, specifically paraphrase-distilroberta-base-v1 which is actually trained on a variety of domains, including some of our domains intended to be held-out (such as Wikipedia). This is strictly to the advantage of the SBERT baseline since it suffers less from distribution shifts, while we control the training data of our stylistic representations to ensure the held-out evaluation domains do introduce distribution shifts. In other words, our baseline is given an unfair advantage relative to our proposed methods, yet we still outperform it, which we attribute to the stylistic training objective.

Why is Twitter from Table 2 not mentioned in Table 1?

Twitter is only used as a training corpus for the Multi-LLM UAR. This is the reason it was not mentioned in Table 1 in the original draft.

How do the authors explain that the proposed UAR (based on SBERT) is still much better than SBERT? The authors emphasize that SBERT is unsuitable as it is trained to capture semantics. So why not use a "normal" BERT in UAR?

The proposed UAR was trained to capture the stylistic differences that are agnostic to the topic being written about. SBERT was trained to capture the semantic similarity, hence its embedding intrinsically captures the topic of a piece of text. The stylistic features of LLMs remain quite stable across the various domains considered, hence the effectiveness of the UAR in capturing the differences in writing styles between different LLMs. The reason to initialize UAR with SBERT over BERT is simply that SBERT was also trained with a contrastive objective, although in practice we suspect that initializing from BERT would perform equally well since the UAR training data is very large (hundreds of millions of documents) and hence the initialization is less critical.

RoBERTa (zero shot) is not introduced. What is it exactly?

RoBERTa (Zero-Shot) shares the same architecture as the OpenAI Detector except that it is trained on the Reddit+AAC corpora. As a reminder, we do this to introduce a distribution shift at evaluation time, and the proposed stylistic representations are also limited to training on disjoint domains rather than the evaluation domains. Elsewhere in this response and in Table 1 in the updated draft, we additionally report results using the pre-trained OpenAI model, which was trained on a variety of domains, likely including our evaluation domains. Nonetheless, we find that it performs worse than our “homegrown” version.

Section 4.5: The text mentions "other UAR variants." Are the Multi-LLM and Multi-domain models UAR variants? This should be clarified. Also, again, the same problem as in the other sections: why not train the other methods with this setup?

Yes, they are UAR variants, which was mentioned in Section 3.2 of the original paper under the “Multi-domain” and “Multi-LLM” headers.

Thanks for reviewing our paper! We have prepared a detailed response which we hope address your questions and concerns. Please also refer to our general response for further experiments requested by other reviewers.

First of all, as mentioned in our response to all reviewers, we do intend to release the datasets we created and have clarified this point in our reproducibility statement. We note that such datasets are already amply available publically, and in fact, as mentioned in S4.1, the publicly available M4 dataset comprises a portion of our evaluation. For this reason we disagree with the reviewer’s assertion that AAC data may be unavailable in practice, unless the reviewer was referring to AAC data in low-resource languages, in which case we agree but would argue that there are fewer LLM of concern in such languages. In any case, for the sake of reproducing our results exactly, we agree that the datasets we generated must be released.

The work has strong limitations in training the style representation learning network. Basically, the writing style can also be changing over time. The existing writing style training dataset may not be powerful enough to cover the whole.

We suspect that there may be a misunderstanding concerning the nature of the writing style representations used in this work. Such representations are specifically trained to capture temporally-invariant aspects of writing style and their ability to generalize to future data has been validated in prior work (see e.g. https://arxiv.org/abs/1910.04979). For example, as shown in our experiments, such representations are robust to various distribution shifts arising from the introduction of new domains relative to the training data used to train the style representations. Furthermore, as illustrated in Figure 1, LLM-generated text, even when produced using prompts designed to elicit varied writing styles, exhibits a consistent stylistic marker. This suggests that LLM-generated text indeed contains temporally-invariant stylistic traits that enable our approach to successfully detect them, perhaps a consequence of the fact that the LLM parameters are frozen.

[...] the comparison to existing approaches is not sufficient. More baselines are supposed to be included. Here are the most recent surveys on it. Other representative works are suggested to be included, such as OpenAI's detector and other training-free approaches should be compared because they do not have domain adaptation issues.

Given our new few-shot setting for machine-text detection, it seemed appropriate for us to focus our comparison on few-shot baselines, since these use the same data as the proposed approach. Therefore, the original paper includes comparisons to ProtoNet and MAML, two standard few-shot learning methods, and several additional few-shot detectors based on pre-trained representations including SBERT and CISR. Nevertheless, as mentioned in our general response to all reviewers, we have now included an additional four “zero-shot” baseline models, including OpenAI’s detector and training-free approaches, in the experiment reported in Table 1 of the revised paper, as well as comparisons to watermarking in Appendix G (Figure 4). We find that the proposed approach significantly outperforms all other training-free approaches.

Regarding reporting additional standard metrics, we report additional experiments in Appendix I on smaller writing samples (making the detection problem harder), where we include standard metrics. These results are reproduced below for convenience.

Note that Rank, LogRank, and Entropy are standard methods from the literature:

• Rank - https://arxiv.org/pdf/1906.04043.pdf
• LogRank - https://arxiv.org/ftp/arxiv/papers/1908/1908.09203.pdf
• Entropy - https://aclanthology.org/2020.acl-main.164.pdf

The approach may not be effective against more advanced language models that can better mimic human authorship.

The reviewer’s concern about the robustness of our approach to future, more fluent language models was precisely the motivation for our research methodology, which simulates precisely this situation. Indeed, we chose to evaluate documents by LWD models, which produce more fluent text than the AAC models used at training time for some proposed variations of style representations, and were developed in the future relative to those models. We also note that the proposed variation of style representations not availing of any machine generated data still detects LWD models better than baseline models in many cases (see Table 1 in the revised paper, or the reproduced table above), some of which are trained with AAC.

Thanks for the feedback!

Our previous use of the AUC’ metric may have insinuated that the approach is only relevant for settings requiring low false positive rates. However, this is not the case, as demonstrated in the additional results reported in Appendix H and reproduced below. Here, we vary the writing sample size to yield a harder detection problem, and additionally report the standard AUC and FPR@95TPR metrics. Note that the proposed style representation methods outperform all competing methods for these standard metrics, including the additional zero-shot baselines.

Note that Rank, LogRank, and Entropy are standard methods from the literature:

• Rank - https://arxiv.org/pdf/1906.04043.pdf
• LogRank - https://arxiv.org/ftp/arxiv/papers/1908/1908.09203.pdf
• Entropy - https://aclanthology.org/2020.acl-main.164.pdf

We thank the reviewer for their keen observation about potential artifacts of our prompting scheme appearing in the LM-generated documents. This artifact occurs only with LM-generated documents in the Reddit domain, which were not used in any experiments except the one reported in Appendix A, where they were used for evaluation rather than training. Therefore the artifact does not help any model to detect machine-generated documents. We have clarified where this generated data is employed in the revised paper to avoid confusion. We also thank the reviewer for the references which we have included in the Related Work section of the revised paper.

Will the datasets be released as well to facilitate reproducibility?

Yes, the data will be released. We regret not making this clear in the original reproducibility statement, which has been revised.