BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens

Thanks for your valuable review and suggestions. Here are our point-by-point responses:

W1: Regarding the OOD evaluation, previous studies [1, 5, 7, 8] shifted either the data (cross-dataset) or the generative models (cross-model).. We strictly follow their settings in our OOD evaluation. Our comparison with baselines is fair because we retrained all the baselines (except RADAR which is not open-source) and BiScope using the same datasets. And the testing was performed on the same datasets too. Please let us know if we misunderstand your question.

W2: Thank you for your suggestions. We have included three more baselines in the Table 6 (in the submitted PDF file), including Binoculars [5], GhostBuster [1], and OpenAI Detector [6]. The results illustrate that BiScope outperforms all three new baselines in most cases across both the normal and paraphrased datasets. On the other hand, watermarking techniques allow the detection of generated text by limiting the generation within some special vocabulary. They require modifications to the decoding strategy of the generative model. This does not align with the application scenario of BiScope and the baselines as they do not require any modification to the LLM’s generation process and can work in a black-box setting. Although it is difficult to compare the two, we consider them complementary. We will cite [10-13] and include the above discussion.

W3: Thank you for your suggestion. We have implemented these two ablation experiments, and the results are shown in Table 7 and Table 8 in the uploaded PDF file, respectively.

Table 7 presents the ablation results with different segmentation strategies in the multi-point splitting of BiScope. We tested three strategies: splitting at every 50% text length, every 25% text length, and every 10% text length (as used in our paper). The results indicate that a more fine-grained splitting interval generally improves BiScope's performance. However, in a small number of cases, a smaller splitting interval may degrade performance. We chose to use 10% as it achieves the highest detection scores in most cases while reaching low degradation in corner cases.

Table 8 presents the comparison results when using and not using the completion prompt in BiScope. The results show that in 25 of 45 cases, using the completion prompt performs better. Additionally, the completion prompt is more compatible with the summary procedure. Thus, we chose to use the completion prompt in BiScope.

Q1: Thanks for the suggestion. We further present the AUC-ROC curve of our method, shown in Figure 9 (in the submitted PDF file). The results show that our BiScope reaches over 0.8 detection TPR on average when the FPR is only 0.01, outperforming all the baselines on all the five generative models’s data.

Q2: RADAR is officially trained on the Openwebtext dataset, which contains over 8 million human-written texts and over 8 million AI-generated texts crafted by RADAR’s authors. These texts are very similar to the data in the Yelp dataset. Since the RADAR’s authors do not provide their training code, we did not further fine-tune RADAR on our datasets, but when calculating the F1 score, we did search the best threshold for RADAR’s output on our datasets. Considering the substantial amount of RADAR's pre-training data and our in-distribution threshold searching, the comparison in our paper does not put RADAR in an unfair position.

Q3: In Table 1, we present the results of BiScope on the paraphrased datasets under both the in-distribution (seen) setting and OOD (unseen) setting. Under the in-distribution setting, BiScope and all the baselines are trained and tested on the paraphrased dataset, resulting in a very small performance drop for BiScope (as mentioned in line 300-301). However, under the OOD setting, BiScope and all the baselines are trained on the normal dataset and tested on the paraphrased dataset, as shown in the right-most column in Table 1. We observe a significant F1 score drop for all methods, including BiScope. For example, BiScope’s detection F1 score drops by around 0.1 on the Arxiv dataset. Notably, BiScope’s performance drop is the smallest among all the detectors in most cases.

Q4: Thank you for pointing out the potential influence caused by the text lengths. We note that the length bias can be clearly identified in the Yelp, Essay, and Creative datasets, where the human-written texts can be twice as long as the AI-generated texts. Thus, we re-ran BiScope on these three datasets, using the first K characters as the input, where K equals the average text length of each individual dataset as shown in Table 2. The results are presented in the following table. For the Yelp dataset, the average detection F1 score drops within 0.01, while for the Creative and Essay datasets, there is even a slight increase in the average detection F1 score. Therefore, the length bias in the dataset does not provide BiScope with unfair advantages.

Thank you for all the presentation suggestions. We will further modify our paper based on them.

Thanks for your appreciation and suggestions. Here are our point-by-point responses:

W1: Thank you for pointing out the problem. We generated our datasets using five of the latest commercial LLMs, following the generation methods outlined in previous studies [1, 7, 8]. Due to the page limit of the initial submission, we placed the details of the generated datasets (e.g., data amount, length statistics) in Appendix C. We will add more details to the main text in our next version.

W2: Thank you for your suggestion. Due to the time constraints, we were unable to generate non-native language data for our datasets to provide a fair comparison during the rebuttal period. However, the setting you recommended is certainly an important OOD setting for all detectors. We will include [9] in our paper’s discussion and provide more detailed results in the next version.

Q1: Thank you for the question. We presented a detailed ablation study using various open-source surrogate LLMs in Table 5 in Appendix E.1. According to the results, the performance of BiScope improves when using more surrogate detection models. Notably, BiScope can maintain over 98% of the best (ensembled) performance even when only using a single surrogate model (e.g., Llama2-7B, Mistral-7B). This consistent performance of BiScope across various open-source LLMs illustrates its scalability and compatibility with different open-source LLMs, providing more flexible options for users. There is no limit or threshold for the number of surrogate models, and BiScope can be compatible with any combination of surrogate models.

Q2: We presented a more detailed comparison of the FCE-only, BCE-only, and FCE+BCE versions of BiScope in Table 4 of Appendix E.2. The combination of FCE and BCE outperforms either FCE only version or BCE only version in 64% of cases, showing >0.35 and >0.09 maximal detection F1 score improvement, respectively. Such results demonstrate the necessity of this combination.

Q3: Thank you for your suggestion. For simplicity, we recommend using either Llama2-7B or Llama2-13B as the surrogate model, while the ensemble of more surrogate models is always welcomed. For the text split method, we recommend splitting the text at every 10% length, as used in our paper. We will open-source our code and datasets for future researchers.

Thanks for your insightful review. Here are our point-by-point responses:

W1: We have included three more baselines in the comparison in Table 6 (in the submitted PDF file): Binoculars [5], GhostBuster [1], and OpenAI Detector [6]. The results show that BiScope outperforms all three baselines on both the normal and paraphrased datasets.

W2: We have presented the experimental results in both cross-model and cross-dataset OOD settings, as well as the evaluation on unseen paraphrased data, in the right three columns of Table 1 in our main text. Additionally, more detailed OOD results are provided in Table 3 in the appendix. We observe that BiScope performs the best or the second best in more than 75% cases.

W3: In Figure 3, the first three steps are used in both the classifier training and testing periods to extract features. During the classifier training period, the extracted features are used to train a classifier in step 4. In contrast, during the testing period, the classifier in step 4 is fixed, and we use this trained classifier to make predictions based on the extracted test sample’s features. We will further modify Figure 3 to make this process clearer.

W4: Thank you for pointing out the OpenAI Detector. We have included RADAR [7] in our main text’s results. RADAR is a fine-tuned RoBERTa, outperforming the OpenAI Detector [6] in most cases. Therefore, we initially chose not to include the OpenAI Detector in our paper. To further address your concern, we have included the OpenAI Detector in Table 6 (in the submitted PDF file). The results show that BiScope outperforms the OpenAI Detector with more than 0.2 F1 score improvement on average on both the normal and paraphrased datasets.

W5: In our cross-model OOD evaluation (Section 4.2), we trained the classifier on human data and AI-generated data from one LLM and then tested the trained classifier on AI-generated texts from the other four LLMs and calculated the average F1 score. This process was repeated for all five generative LLMs, and the average scores are reported in Table 1 in the main text. More detailed results are presented in Table 3 in our appendix. Results show that BiScope performs the best or the second best in more than 75% cases. Specifically, when we trained BiScope on the oldest GPT-3.5-Turbo’s data and tested it on all the other four latest LLMs’ data, the detection F1 score exceeded 0.92 on average.

W6: Thanks for pointing out additional metrics. We use the F1 score in our paper since it is a commonly used metric in previous papers [1, 8], considering both precision and recall. We further present the TPR-FPR (ROC) curve of BiScope in Figure 9 (in the submitted PDF file). We observe that our BiScope reaches over 0.8 detection TPR on average when the FPR is only 0.01, outperforming all the baselines on all the five generative models’s data.

W7: Thanks for your recommendation. We will further polish our tables based on your suggestion.

W8: Thank you for pointing out the limitation problem. Due to the page limit of the submission, we had to place the limitation section in Appendix F in our initial submission. We will move it to the main text. Regarding the OOD evaluation, previous studies [1, 5, 7, 8] shifted either the data (cross-dataset) or the generative models (cross-model). We followed their settings to test our OOD performance.

Thanks for your insightful review. Here are our detailed point-by-point feedbacks for your questions:

W1: As mentioned in Section 3.3., the summary generation is not necessary. We have two designs for generating the completion prompt: one with a summary and the other without. The latter has a significantly shorter processing time. As illustrated in Figure 5, this adjustment allows BiScope to achieve over 8.6x shorter processing time per sample. As shown in Table 1, BiScope without summary only results in an average detection F1 score degradation of less than 0.015 and still substantially outperforms the baselines. Users can choose between these two designs to balance detection accuracy and efficiency according to their specific needs. We will further modify our main text to further clarify this.

W2: Thank you for suggesting the references and baselines. We present a comparison between BiScope and GhostBuster [1] in Table 6, demonstrating that BiScope outperforms GhostBuster with a 0.06 average F1 score increase on both the normal and paraphrased datasets. Additionally, GhostBuster is 3x slower than BiScope in processing a single sample and requires several hours to identify the optimal feature compositions. We did not compare with [2] due to the lack of open-source implementation. Instead, as mentioned in the global response, we compared with two other baselines, Binoculars [5] and OpenAI Detector [6], which have comparable or better results than [2-4]. We will reference [1-4] in our paper and discuss them in the related work.