Training-free LLM-generated Text Detection by Mining Token Probability Sequences

We appreciate the Reviewer LuhL for your invaluable suggestions to improve the quality of our work. We have addressed all concerns as below.

W1: We apologize for any confusion caused by our writing. Figure 2 illustrates the computational workflow or framework of Lastde, using a 7-token text as an example. While the global statistics calculation is straightforward, for local statistics, the sequence of token probabilities (TPS) with 7 tokens is first processed into 3 new sequences (as we set $\tau^{\prime}=3$). Here, $\tau_i\in{1,2,3}$ represents the scale factor, indicating that the current scale is derived from the arithmetic mean of $\tau_i$ adjacent elements in the original TPS. For each new sequence, we slide from left to right with a stride of 1 and window width of 3 (as we set $s=3$), vertically concatenating each sliding window segment into a matrix. We then calculate the cosine similarity between adjacent rows (segments) to obtain a similarity sequence. Finally, we divide [-1,1] into 10 subintervals (as we set $\varepsilon=10$) and compute the frequency or probability of elements from the similarity sequence in each subinterval (resulting in a $1\times 10$ vector), followed by calculating the sequence entropy to obtain the DE value. The final step in local statistics calculation involves using an aggregation operator $\text{Agg}:\mathbb{R}^{\tau^{\prime}}\rightarrow\mathbb{R}$ to summarize all DE values. The corresponding mathematical derivations can be found in Section 3.1, and we have updated the main text accordingly with changes marked in red.

W2: We appreciate this constructive comment. Previous studies such as DNA-GPT[1] (ICLR2024) and Fast-DetectGPT[2] (ICLR2024) indeed included GPTZero as a baseline. However, since our main focus was on training-free methods, we did not test any training-based detectors in the main text and instead concentrated on Fast-DetectGPT as our main comparison target. Nevertheless, we have addressed this concern by including results from two powerful baselines in Table R1 (in General Response): Binoculars[3] (ICML2024) and GPTZero. Here is a brief summary of the results (please refer to Table R1 for complete details).

W3: Indeed, the ability to achieve universal detection across different domains or subjects is a notable advantage of all training-free methods, including the proposed method. To support this claim, we have provided extensive experimental results in the paper and Table R1 (in General Response). Specifically, most experiments in the main text were conducted on four datasets: Xsum, SQuAD, WritingPrompts, and Reddit, covering news, Wikipedia context knowledge, story creation, and community Q&A scenarios. Notably, the Reddit dataset encompasses six subjects: biology, physics, chemistry, economics, law, and technique. Furthermore, we included German and Chinese datasets, with the Chinese dataset explicitly containing samples from three subjects: food, history, and economics. Table 1 also presents detection results on PubMedQA, an expert-level Q&A dataset focusing on biochemical knowledge. The results consistently demonstrate our method's superiority over other detectors, providing compelling evidences for our method's robustness to cross-domain detection.

W4: Thank you for the great suggestion. In the rebuttal, we have conducted extensive experiments to validate the robustness of our method, and summarized the results in Table R2 (in General Response). In brief, even when we explicitly instructed the LLM to mimic human writing styles, Lastde++ still achieved excellent detection performance. Moreover, Lastde++ demonstrates greater advantages over Fast-DetectGPT in terms of TPR and Accuracy, which are more practical metrics. Below are the TPR at 5% FPR results from Tables R2 (in General Response):

[1] DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text

[2] Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

[3] Binoculars: Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

We appreciate the reviewer Kqgm for your constructive suggestions, we will address all raised concerns point by point as below.

W1: Thank you for the insights. In the revision, we additionally report the TPR and TNR values to address this concern. More specifically, we consider human-written text as the negative class and LLM-generated text as the positive class. When fixing FPR = 5% (i.e., TNR = 95%), we observe the following TPR results:

These results demonstrate that Lastde++ achieves TPR values higher than 74% on both benchmark datasets, indicating that while correctly classifying 95% of human-written texts, it can also accurately identify at least 74% of LLM-generated texts. This performance significantly outperforms other training-free methods. Therefore, we believe Lastde++ demonstrates robust capability in discriminating both human-written and LLM-generated texts.

W2: We have selected two representative datasets and conducted additional tests using a paraphraser with larger parameter count. The details about this paraphraser and comprehensive detection results are reported in Table R3 (in General Response). In brief, even when texts are attacked using more powerful models, our method still achieves SOTA performance. Furthermore, Tables R2 - R6 contain additional robustness test results, which we believe will help address your concerns.

W3: We apologize for not elaborating on the threshold determination method in our paper. For training-free detectors, most studies use AUROC as the evaluation metric, which doesn't require fixing a specific threshold. However, as the reviewer pointed out, practical applications typically require decision-making at a specific threshold. We have examined two approaches:

• Fast-DetectGPT [1] (ICLR2024) uses an "offset" to estimate thresholds on specific datasets (e.g., XSum(GPT-4)) in their open-source code (https://github.com/baoguangsheng/fast-detect-gpt/blob/main/scripts/local_infer.py).

• Other related studies, such as MGTBench [2] (CORR2023) and RAIDAR [3] (ICLR2024), fit a univariate logistic regression model using both detection scores and labels for final classification.

We prefer the latter approach and have fitted logistic regression models on datasets (including Xsum, WritingPrompts, Reddit) generated by two closed-source models (GPT-4-Turbo, GPT-4o) and one open-source model (OPT-13B), reporting metrics on the test set (test size=0.2). Average AUROC/TPR metrics at 1%FPR on three datasets are reported below.

The fixed threshold here corresponds to the decision boundary of the logistic regression model. For closed-source models, Lastde++ shows comparable or better performance in both AUROC and TPR compared to Fast-DetectGPT, while significantly outperforming Fast-DetectGPT on open-source models. However, this threshold determination method still requires true labels, which we acknowledge as a limitation (not entirely training-free).

To resolve this issue, we propose a more reasonable modification: According to Figure 12 in our Appendix E, regardless of the source model, the Lastde++ score boundary between human-written and LLM-generated texts consistently appears around 2.0, which can be estimated more accurately with larger sample sizes. While setting the Lastde++ detection threshold directly to 2.0 might yield satisfactory results, a more robust approach would be to first define a neighborhood with radius $r$ centered at 2.0, then assign pseudo-labels to samples on either side (human for left, LLM for right), and finally fit a logistic regression model. Notably, this modified approach doesn't utilize any true labels, maintaining its training-free nature.

W4: We agree with the reviewer's insight and have constructed new datasets as suggested. These datasets combine texts generated by multiple source models, incorporating various mixing ratios and model numbers. Detailed experimental procedures, workflows, and metrics are presented in Tables R4-R6 (in General Response). In summary, both Lastde and Lastde++ demonstrate effective detection capabilities across different scenarios, whether combining 2 or 4 source models, or mixing with ratios of 50%+50% or 80%+20%. Their detection results (AUROC, TPR, Accuracy) consistently outperform existing training-free detectors like Fast-DetectGPT. We hope these results can help address your concerns.

[1] Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

[2] MGTBench: Benchmarking Machine-Generated Text Detection

[3] Raidar: geneRative AI Detection viA Rewriting

• Q1: We would like to clarify that both Fast-DetectGPT [1] and DetectGPT [2] share similar fundamental assumptions about human and LLM-generated texts: given a context, LLMs tend to use tokens with higher probabilities in the next-token prediction, while humans rely more on semantics rather than probabilities. Under this assumption, we believe it is challenging for humans to precisely mimic AI writing styles. During the rebuttal, while we have tried our best to find datasets containing human-written texts mimicking AI styles, we haven't found a suitble one (we would promptly include these experiments if the reviewer could help specify such datasets). Nevertheless, as a compensatory measure, we constructed LLM-generated texts mimicking human writing styles, with results shown in Table R2 (in General Response). These results demonstrate that Lastde++ is comparable or better than Fast-DetectGPT in AUROC, TPR, and Accuracy metrics.

• Q2: We have addressed this concern and provided additional experiments in the Detecting LLM/human mixted texts in Response to Concern II in our General Response. In brief, we simulated human modifications using a paraphraser to attack LLM-generated texts. These attacked texts can be viewed as human-LLM collaborative generations. Detection results on these attacked texts show that Lastde++ outperforms Fast-DetectGPT with stronger robustness (smaller performance degradation). Additionally, when using the paraphraser to attack both human and LLM-generated texts, which we consider as LLM-Revised texts, Lastde++ demonstrates superior performance and robustness compared to Fast-DetectGPT.

• Q3: Thank you for the insightful observation.

  • a): We notice some training-based studies that use token log probability as initial features (such as Sniffer [3], SeqXGPT [4] (EMNLP 2023), POGER (IJCAI 2024)). However, Lastde's final output, after aggregating global and local statistics, is a real number (similar to likelihood and rank metrics). From a feature extraction perspective (e.g., "probabilistic features" in Section 2), the dimensionality or information content might seem insufficient. Since our goal is to develop a universal detector with strong generalization capabilities (a characteristic strength of training-free methods), we prefer combining Lastde with plug-and-play lightweight modules like DALD [5] (NeurIPS 2024) and TOCSIN [6] (EMNLP 2024), enhancing detection performance while maintaining the significant cross-domain, cross-model, and cross-lingual advantages of training-free methods.

  • b): In the revision, we have combined Lastde, Lastde++, and existing baselines with the TOCSIN lightweight model, with results shown in Table R1 (in General Response). The results clearly demonstrate performance improvements across all detection methods when combined with TOCSIN. Lastde and Lastde++ showed average improvements of 3.7% and 1.1% respectively, achieving SOTA performance both before and after integration, which we believe strongly validates your observation. We apprecite your insights again, and we have included these results in the Appendix.

[1] Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

[2] DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature

[3] Sniffer: Origin Tracing and Detecting of LLMs

[4] SeqXGPT: Sentence-Level AI-Generated Text Detection

[5] POGER: Ten Words Only Still Help: Improving Black-Box AI-Generated Text Detection via Proxy-Guided Efficient Re-Sampling

[6] DALD: Improving Logits-based Detector without Logits from Black-box LLMs

[7] TOCSIN: Zero-Shot Detection of LLM-Generated Text using Token Cohesiveness

We appreciate the reviewer WrSh for recognizing the effectiveness of our method and we would like to address all raised concerns in detail.

• W1: As the reviewer pointed out, the selection of proxy models may significantly impacts the performance of black-box detection, which is a common challenge encountered by almost all training-free detectors. Recent work such as DALD [1] (NeurIPS 2024) has attempted to align proxy models with source models through distillation, partially alleviating this pain point. However, selecting more general and generalizable black-box proxy models in a principled may remains challenging. Our proposed method focuses on extracting inherent textual features and represents foundational work that can be readily integrated with recent detection methods (such as DALD). Moreover, as demonstrated with our extensive experiments, the proposed method can consistently achieve state-of-the-art performances across all datasets, including the black-box scenarios, even though the proxy model selection was not explicitly optimized. And we believe its detection effectiveness can be further improved using a more carefully selected proxy model.

• W2: We apologize for the confusion regarding the choice of proxy model.

  • a): We would like to clarify that, in Section 4.1 "Source models & proxy model" and Appendix B.3, we have detailed the proxy model selection criteria for all detection methods. Unless otherwise specified, all detection methods use GPT-J as the sole proxy model. Specifically, for "Sample-based" methods, we directly use GPT-J for scoring without additional processing. For "Distribution-based" methods, DetectGPT [2] (ICML 2023) and DetectNPR [3] (EMNLP 2023) first use T5-3B to generate perturbed samples, then use GPT-J to score the distribution formed by perturbed and original texts. For DNA-GPT [4] (ICLR 2024), we first truncate candidate texts, input the first half into GPT-J to generate rewritten samples, and finally use GPT-J to score the distribution formed by rewritten and original texts. Fast-DetectGPT [5] (ICLR 2024) and Lastde++ first use GPT-J for inference on candidate texts, then perform fast sampling on the resulting logits to obtain contrastive samples. Finally, GPT-J scores the distribution formed by contrastive and original texts.

  • b): In Fast-DetectGPT, the prior SOTA, it explored GPT-2, Neo-2.7B, and GPT-J as proxy models, with Neo-2.7B as the default scoring model and GPT-J as the sampling/rewriting model (if applicable), and T5-3B as the perturbation model (if applicable). Following this setup, we conducted experiments on GPT-2, Neo-2.7B, and GPT-J, finding that using GPT-J for scoring, sampling, and rewriting yielded optimal results on our datasets, hence our default configuration. However, as mentioned in the previous paragraph, selecting black-box proxy models remains extremely challenging. To the best of our knowledge, few studies explicitly guide proxy model selection, necessitating reliance on detection results and generalization experiments to infer more universally applicable proxy models. In Section 4.3 "Selection of proxy models," we discussed detection results across different proxy models for various source texts. While these results can guide proxy model selection to some extent, we agree that more theoretical and empirical analysis is needed to establish selection criteria.

• W3: As pointed out by the reviewer, detection performance decreases with shorter texts, a challenge faced by all detection methods (both training-free and training-based). While MPU [6] reformulates AI text detection as a Positive-Unlabeled (PU) learning problem and shows promising results on short-text datasets like HC3 and TweepFake, substantial room for improvement remains. This is parimarily because short texts contain limited information and often lack detectable tendencies (human or AI). Therefore, we believe that detecting short texts (e.g., in social media scenarios, AI-generated rumors, fake news) may require combining textual features (such as likelihood, rank, Lastde) with external tools. This represents both potential value in our method and a direction for our future exporation.

[1] DALD: Improving Logits-based Detector without Logits from Black-box LLMs

[2] DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature

[3] DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text

[4] DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text

[5] Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

[6] MPU: Multiscale Positive-Unlabeled Detection of AI-Generated Texts

We sincerely thank Reviewer WrSh for your thoughtful feedback, and we are encouraged that our rebuttal has alleviated your concerns. We have incorporated new results, analyses, and discussions into our revised manuscript, which we believe significantly enhanced the clarity and overall quality of our work.

We appreciate the Reviewer’s insightful observations regarding potential challenges in scenarios such as proxy model selection in black-box settings and short-text detection. We acknowledge that these are indeed important and promising research directions, both for our future work and for advancing this field more broadly.

Meanwhile, we would like to clarify that these challenges are long-standing and prevalent in the domain of AI text detection. Despite these inherent difficulties, as demonstrated in our work (and summarized in the General Response), our proposed method consistently achieves state-of-the-art performance among training-free approaches across diverse scenarios, including cross-domain, cross-model, and cross-lingual detection tasks, under both white-box and black-box settings.

Notably, as detailed in Table R1 of the General Response, our approach can be augmented with external tools to further improve our performance, even surpassing commercial detectors like GPTZero.

We hope this response addresses any remaining concerns and would greatly appreciate any additional suggestions or insights you may have.

Thank you again for your constructive feedback and support.

Sincerely,

The Authors

We thank Reviewer SbPp for your thorough review of our paper. We would like to address your concerns, raised in Weakness and Questions parts, as below.

• W1: We agree with your point. While AUROC may be be not a comprehensive metric, we followed the conventional practice from popular baseline methods, including Fast-DetectGPT[1] (ICLR2024), DetectGPT[2] (ICML2023), and DetectLLM[3] (EMNLP2023), which all used AUROC as their primary metric. Following the reviewer's constructive suggestions, we also supplement our results with TPR and Accuracy values (at 1% FPR) for selected datasets.

  • TPR at 1% FPR（White-box/Black-box）

    Datasets/Methods	Likelihood	LogRank	DetectLRR	Lastde	Fast-DetectGPT	Lastde++
    WritingPrompts(Llama2-13b)	11.33/1.33	18.67/2.00	40.00/19.33	64.67/30.00	63.33/10.67	84.67/50.00
    WritingPrompts(OPT-13b)	10.00/5.33	19.33/9.33	46.67/25.33	71.33/54.67	84.00/20.00	86.67/53.33

  • Accuracy at 1% FPR（White-box/Black-box）

    Datasets/Methods	Likelihood	LogRank	DetectLRR	Lastde	Fast-DetectGPT	Lastde++
    WritingPrompts(Llama2-13b)	55.17/50.17	58.83/50.50	69.50/59.17	81.83/64.50	81.17/54.83	91.83/74.50
    WritingPrompts(OPT-13b)	54.50/52.17	59.17/54.17	72.83/62.17	85.17/76.83	91.50/59.50	92.83/76.17

  The results show that while all detection methods perform modestly when FPR is fixed at 1%, Lastde and Lastde++ consistently outperform its baseline methods. Notably, in the black-box setting, Lastde surpasses Fast-DetectGPT's performance. Additionally, we present more metrics and results under a more relaxed setting (FPR=5%) in Tables R2-R6 (in our General Response), which consistently demonstrate that the proposed method can achieve the state-of-the-art performance.

• W2: We thank the Reviewer for raising this excellent question. We believe Lastde is fundamentally different from SeqXGPT[4] (EMNLP2023) for the following reasons:

  • a): We apologize for the confusion and we would like to clarify that SeqXGPT is not a time series based detection method strictly, rather, it frames sentence-level text detection as a Named Entity Recognition (NER) task. Its foundational features are directly set as the log-probability values of each token across multiple open-source proxy models (resulting in a matrix of dimensions: token count × number of proxy models, as detailed in Section 4.3 of their paper). Therefore, we believe this approach does not involve time series transformation and analysis methods, but rather focuses on task definition. In contrast, our method goes beyond simply using these log-probabilities by introducing time series analysis to extract dynamic patterns from token log-probability segments, emphasizing time series feature extraction and analysis. Thus, we consider our approach fundamentally different from SeqXGPT in terms of perspective and feature processing.

  • b): Another major difference is that SeqXGPT is essentially a training-based method, while our paper primarily focuses on training-free approaches. Given our extensive cross-domain corpus experiments, it might not be fair to directly comparing SeqXGPT with the proposed method on these cross-domain datasets.

• Q1: Thank you for this intriguing suggestion. Based on our understanding of this question, Lastde (ours), likelihood, rank, and lrr are all variants of f(.), and they are all training-free. We believe more training-free f() variants will be developed for LLM-generated text detection in the future. Regarding your question about "learning an optimal f(.) through a network," we believe this is entirely feasible. However, since our paper primarily focused on training-free methods, we did not extensively explored this direction, and we believe it could be a promising avenue for our future research.

• Q2: We agree that text generated by instruction/chat source models is generally more challenging to detect. In Appendix B.2 Table 5, we list the Hugging Face indices for all models, most of which are base models (except for closed-source models), consistent with Fast-DetectGPT's approach. In Table R1 (in General Response), we conducted detection on more chat/instruct source models, and the results demonstrate that our method still achieves the best performance.

• Q3: Our method does not involve any assumptions about "prompts or questions used to generate AI text," therefore we do not need to know such information during detection. Indeed, all experimental results related to chat/instruct source models presented in the main paper or Table R1 (in General Response) were conducted without any prompts or questions.

• W3: We would like to clarify that most of our SOTA results were achieved under the same parameter setting, and we only set different default values for Lastde and its variant, Lastde++. Based on our discussion and analysis on hyperparameters (Section 4.2), their selection does not affect Lastde or Lastde++'s ability to achieve SOTA performance among existing methods. Moreover, in Table R1 (in General Response), we unified the hyperparameter settings for Lastde and Lastde++, and the third-party data detection results still demonstrate our SOTA performance.

• W4: In fact, GPT-2, Neo-2.7, and OPT-2.7 appear exclusively in the decoding strategy section of our robustness analysis (Section 4.3). For other results, we conducted experiments primarily on larger (7B or 13B) and more prominent models, generally following and covering Fast-DetectGPT's (ICLR 2024) experimental settings. For certain model series, we also included the latest versions (such as Llama3-8B). Due to limitations in computing resources, we couldn't generate new experimental data on source models exceeding 20B parameters, but we incorporated alternatives like Gemma and Phi2 to cover a broader range of LLM products from various tech companies. Given our coverage of model types (open-source, closed-source, new versions, old versions), which is more comprehensive than many existing studies, we believe our results are more compelling now. Here are the parameter counts for some of our source models (>7B):

  models	OPT-13	Llama-13	Llama2-13	GPT-4-Turbo	GPT-4o	Claude-3-haiku
  parameters	13B	13B	13B	>>7B	>>7B	>>7B

Please also refer to Table R1 (in General Response) for experimental results covering more models with ≥20B parameters.

[1] Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

[2]DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature

[3]DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text

[4]SeqXGPT: Sentence-Level AI-Generated Text Detection

please add those to the final version