DNA-DetectLLM: Unveiling AI-Generated Text via a DNA-Inspired Mutation-Repair Paradigm

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript and providing valuable suggestions. In response to your feedback, we offer the following point-by-point replies:

R1 (to W1 & W3): On Dataset and Baseline Expansions

To further evaluate generalization performance, we have included results on PubMedQA, a more challenging biomedical short-text dataset, in Table 1.

Our experiments already incorporate several of the most recent and state-of-the-art baselines, such as Biscope (NeurIPS 2024), R-Detect (ICLR 2025), and Lastde++ (ICLR 2025). In response to your suggestion, we have additionally included several relevant recent methods (e.g., DNA-GPT and ImBD) in Table 1 for a more comprehensive comparison. Note that DNA-GPT uses the same scoring model (Falcon-7B-Instruct) as the other methods, and ImBD is pretrained on the HC3 dataset. As shown in Table 1, DNA-DetectLLM maintains strong detection performance on PubMedQA (AUROC = 97.08%), clearly demonstrating its superior generalization ability across diverse distributions.

Table 1. AUROC(%) on GPT-4-generated Texts.

R2 (to W2): On the Use of GPTZero as a Baseline

GPTZero is a popular commercial detection tool that was included as a baseline in earlier works such as Fast-DetectGPT. However, many subsequent studies have shown that GPTZero performs significantly less reliably than strong zero-shot methods. Recent state-of-the-art approaches (e.g., Biscope, Lastde++) have also opted not to include GPTZero in comparisons, instead focusing on benchmarking against more advanced methods. Additionally, the inference cost of GPTZero is relatively high. Given these considerations, we believe that including GPTZero as a baseline in our work is not essential.

R3 (to W4): On the Use of GPT-4 Turbo, Gemini 2.0 Flash, and Claude 3.7 Sonnet

The model IDs for the text generators used are as follows:

• GPT-4 Turbo: gpt-4-turbo-2024-04-09

• Gemini 2.0 Flash: gemini-2.0-flash-001

• Claude 3.7 Sonnet: claude-3-7-sonnet@20250219

All generations were performed using the default official parameter settings provided by the respective APIs. We will include further implementation details and code snippets in the Appendix of the final version to enhance the reproducibility of our results. We once again thank you for your constructive suggestions and hope that our responses have adequately addressed your concerns.

Best regards,

All authors

I appreciate the authors’ clarification and am glad to see that the misunderstanding regarding GPTZero has now been resolved. While I originally mentioned that considering cost issues, it would be acceptable not to compare with GPTZero, what is unacceptable are the authors’ unfounded or incorrect assertions, such as claiming that BiScope does not include GPTZero, or arguing that 'However, many subsequent studies have shown that GPTZero performs significantly less reliably than strong zero-shot methods.'

Furthermore, there are still some assertions in the authors’ reply that leave me very confused, for example: 'However, in real-world scenarios, users often seek more diverse and natural outputs and therefore do not reduce the default temperature of 1.0.'

I do not think this statement is accurate. First, for GPT models, a temperature of 1.0 is actually quite high. While a higher temperature may lead to more creative text, it can also increase hallucinations. In real-world scenarios, users who pursue high-quality text (e.g., with fewer hallucinations) may reduce the temperature. Considering the domains used in text detection—such as XSum and writing—higher creativity may be desirable. Thus, balancing text quality, naturalness, and creativity, a temperature around 0.7–0.8 is often more reasonable. In fact, this is also the temperature used in Fast-DetectGPT and DNA-GPT.

That said, some papers do use a temperature of 1, such as DetectGPT and DetectRL, which mention using it to “promote the generation of creative and unpredictable text.” So, if the intention is to increase the difficulty of detection, this is indeed reasonable.

However, making a direct assertion like 'in real-world scenarios, users often seek more diverse and natural outputs and therefore do not reduce the default temperature of 1.0' could be misleading. In fact, in this context, “natural” and a temperature of 1.0 seem somewhat contradictory, as lower temperatures often lead to more natural text.

The authors do not need to regenerate texts using other temperature settings for experiments. My intention here is simply to correct this statement, because such unfounded assertions can mislead readers. Given the broad real-world applications of text detection, your readers may include practitioners who focus on applications rather than research. In such cases, statements like this could easily lead to misunderstandings.

Lastly, I appreciate the authors’ paper mainly for its promising experimental results and relatively short time consumption. In fact, the idea of using DNA-inspired methods for detectors can indeed be traced back to DNA-GPT. However, DNA-GPT suffers from extremely long processing times and unsatisfactory results (as reported in many papers). In contrast, your results show both low time cost and effective performance, which is impressive and clearly beneficial for future research. By this, I mean that your approach can serve as a competitive baseline.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript and providing valuable suggestions. In response to your feedback, we offer the following point-by-point replies:

R1 (to Weakness): Training-free Methods Require No Threshold Training under AUROC Evaluation.

We understand your point regarding training-free methods not being entirely "training-free." For example, methods like GECScore [1], aside from using predefined thresholds, can also optimize a distribution-specific "optimal threshold" based on training data. Similarly, other training-free methods also require identifying an “optimal threshold” based on feature scores from training data to enhance classification performance in practical applications. However, it is equally important to emphasize that when these methods calculate statistical scores based on relative differences, they indeed do not require additional training. For instance, Fast-DetectGPT [2] calculates log conditional probability differences across multiple samples using sampling and scoring models (LLMs in evaluation mode).

In our experiments, we primarily adopt AUROC, a threshold-independent evaluation metric. Therefore, none of the training-free methods in our experiments involves training on data within the same distribution to identify an "optimal threshold." Instead, they calculate statistical scores for positive and negative samples and directly convert these scores into AUROC values, enabling a fair comparison across methods. Meanwhile, the training-based methods are trained on the general HC3 dataset [3] to simulate real-world out-of-distribution (OOD) scenarios (see Section 4.1), providing a relatively fair evaluation against the zero-shot methods that require no additional training. We hope this clarification addresses your concerns regarding our training and evaluation.

P.S. The computation of F1 scores presented in Table 2 is explained in Appendix D of the manuscript.

R2 (to Suggestion): Thank you for your suggestion and for highlighting additional related work. We will expand Section 2 to include a discussion of sequence-level revise or rewrite-based approaches, such as Revise-Detect [4] and GECScore, which assess textual differences through revision-based strategies. As shown in the supplementary Table 1 below, we evaluated the detection performance of Revise-Detect and GECScore on GPT-4-generated texts. Our results show that these methods display substantial performance variability across different data distributions and are considerably less reliable than DNA-DetectLLM. Interestingly, Revise-Detect performs well on the Arxiv dataset. We believe this may be due to the formal writing style of the dataset, where the LLM used for revision (GPT-4o) can more effectively capture the homology of GPT-4-generated texts. To more fully demonstrate the advantages of DNA-DetectLLM, we will include these experimental results in the Appendix of the final version.

Table 1. AUROC(%) on GPT-4-generated Texts.

R3 (to Question): The training-free methods do not require any additional training and compute AUROC scores directly based on statistical scoring. In contrast, training-based methods such as Biscope are trained on the HC3 dataset and do not incorporate any explicit handling of input text length. In the robustness experiment on text length, we only truncate the input text during the testing phase, without modifying the training process.

[1] Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore. COLING, 2025.

[2] Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature. ICLR, 2024.

[3] How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. 2023.

[4] Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT. EMNLP, 2023.

Best regards,

All authors

Thank you very much for your clarifications regarding the evaluation procedure with AUROC. I realize that my earlier comment concerning “training” may have been ambiguous—I was specifically referring to the process of selecting an optimal decision threshold for F1-score computation, rather than model training in the conventional sense. I apologize for any confusion this may have caused.

To further clarify my concerns:

1. Threshold Selection in Table 2

In Table 2, the reported F1-scores for each method require setting a classification threshold. Could you please elaborate on how this threshold was determined? Specifically, was it selected by optimizing on the test set, or by using a separate "train/statistics" dataset?

If the threshold was chosen directly on the test set (i.e., by selecting the value that yields the highest F1-score on the test data), this may lead to overly optimistic (and potentially overfitted) F1-scores, which could affect the fairness of the comparison, especially for methods that are statistical or “training-free.”

For a fair comparison, it would be preferable if all methods used a threshold selected via a "train/statistics" dataset, and then reported the F1-score on the test set using that threshold. If the threshold was indeed selected on the test set, I would recommend revising the evaluation to align with standard best practices; otherwise, the reported F1-scores might not fully reflect true performance in a real deployment scenario.

2. F1-score Reporting in Robustness/Ablation Experiments

I also noticed that the ablation (robustness) experiments currently report only AUROC. While AUROC is a useful metric, I am concerned that it may not fully reflect the practical robustness of the method in real-world applications.

In practice, after model training or statistical analysis and validation, a fixed decision threshold (e.g., optimized on a "train/statistics" dataset or determined by statistical criteria) is set, and this same threshold is used for classifying all new samples—including paraphrased/edited or length-altered examples.

It is important to note that a high AUROC under attack or distribution shift does not always guarantee robust performance with a fixed threshold. For example, under adversarial or distributional shifts, the score distributions may change in such a way that the fixed threshold results in much lower precision, recall, or F1-score, even if the AUROC remains high.

Therefore, I would kindly suggest reporting the F1-score (or other threshold-dependent metrics) in the ablation/robustness experiments as well, using a threshold chosen on a clean (in-distribution) "train/statistics" dataset, and applying it directly to the perturbed or OOD test sets. I believe this will provide a more accurate and practical assessment of your method’s robustness.

I appreciate the authors’ additional experiments and clarifications regarding my concerns, and I am glad to see that most of my issues with the paper have been addressed. I have therefore raised my score to 5.

In fact, I understand the original experimental approach taken by the authors, which is consistent with many previous excellent works such as DetectGPT and Fast-DetectGPT. However, in practical testing, these detectors’ performance is far from sufficient for real-world applications. I think the main reason is that the evaluation metrics and test settings make the experimental results appear overly optimistic. From the updated results provided by the authors, I believe this work already offers enough insight and justification for acceptance, including its interesting motivation, encouraging experimental results and robustness, as well as manageable computational overhead. Additionally, I very much appreciate the authors’ experimental design, especially the validation of the detector’s robustness on more challenging benchmarks such as DetectRL, which is highly instructive.

I hope the authors can adjust the experimental setup and update all experimental results according to the rebuttal (if this paper is accepted), as this will be important for readers who wish to follow up on your research. As I mentioned, I appreciate the experimental design of this paper and hope it can serve as a standard for future research.

Finally, the only slight shortcoming in the paper is that the performance of the supervised detectors is very poor, which seems unreasonable because the comparison is unfair. As you mentioned, all the statistical-based methods are trained on in-distribution datasets, but the supervised detectors are not. In fact, if these detectors were trained on data from the same distribution, their performance would be very good, even with only a small amount of data, in my experience. That said, I think the authors should clearly state in the paper that the settings for these supervised detectors are different from those for the statistical methods. This will help readers understand the out-of-distribution detection limitations of such detectors, rather than giving the impression of downplaying the performance of supervised detectors to highlight the advantages of statistical methods. Otherwise, this may mislead readers into thinking that supervised detectors are very poor, though in reality their practical applicability may be much higher than that of statistical methods, especially on in-distribution data.

Dear Reviewer,

We sincerely appreciate your patient response and valuable clarification. In response to your concerns, we provide the following detailed reply:

1. Adjusting Threshold Selection for Standard F1 Scores in Table 2

We agree with your comment that "the threshold chosen directly on the test set may lead to overly optimistic F1-scores." To ensure a fairer performance comparison, we select fixed thresholds for all methods based on scores computed on a separate clean dataset (e.g., DNA-DetectLLM: 0.6533, Binoculars: 0.9366, etc.). Subsequently, we recompute and report the F1 scores in Table 2 using these fixed thresholds.

P.S. The clean dataset used for threshold selection consists of over 3,000 samples generated by GPT-4, Gemini, and Claude based on human-written texts sourced from XSum, WritingPrompt, and Arxiv. While this dataset is different from the data employed in the robustness experiments and ablation studies, it remains within the same distribution.

Table 1-1 below presents the updated F1-scores from Table 2 after reselecting thresholds.

Table 1-1

2. Additional Reporting of F1-scores in Robustness/Ablation Experiments

Using the fixed thresholds selected as described above, we additionally report the F1-score performances for the primary robustness experiments (as shown in Tables 1-2). Notably, DNA-DetectLLM consistently exhibits superior robustness compared to other baselines in practical scenarios. For a more accurate and practical evaluation, we will supplement these F1-score results along with other robustness and ablation experiment results in the appendix of our final manuscript.

Interestingly, during our experiments, we observed that methods with stronger performance typically possess more stable thresholds (i.e., thresholds with smaller fluctuations). DNA-DetectLLM’s stable threshold further ensures its generalization capability across various data distributions.

Once again, we sincerely thank you for your constructive suggestions. We genuinely hope our responses adequately address your concerns and look forward to further interactions with you.

Table 1-2

Best regards,

All authors

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript and providing valuable suggestions. In response to your feedback, we offer the following point-by-point replies:

R1 (to W1): The ideal sequence is derived from the input text and corresponds one-to-one with it. When the ideal sequence exists independently as a text sequence, it may have a lower probability or deviate from grammatical rules due to independent selection. However, from a fine-grained perspective, each token in the ideal sequence represents the most likely token for the corresponding position in the input text, similar to how nucleotides on the template strand of DNA replication are healthy (non-mutated). In contrast, the tokens (those not of the maximum probability) in the original text may correspond to mutated nucleotides on the DNA replication strand. These inspired our thinking to introduce the mutation-repair paradigm, analogous to DNA, into the detection process.

R2 (to W2): Expand on the Binoculars score to further clarify and strengthen the paper’s readability. The mutation score in DNA-DetectLLM improves upon the Binoculars score because the latter (the ratio of log perplexity to cross-perplexity) is highly effective at capturing AI-generated text with high perplexity. A key condition for calculating the cross-perplexity from dual model perspectives is that the tokenizers for the LLMs must be identical. We will provide a detailed explanation of the above points in Section 3.1 (Preliminary) of the final version of the manuscript.

Best regards,

All authors

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript and providing valuable suggestions. We are truly delighted by your positive assessment of our work. In response to your feedback, please find our reply below:

Response to Weakness: Figure 6 in our manuscript illustrates the performance of DNA-DetectLLM using various base LLMs on the DetectRL dataset. We have identified that further improvements can be achieved through optimal LLM pairing. To better highlight and clarify these experimental findings, we will include an additional comparative table in the Appendix in the final version.

Best regards,

All authors