Multiscale Positive-Unlabeled Detection of AI-Generated Texts

Dear reviewer vhmU,

Thank you very much for your review. Here are our responses:

W1: The writing of this paper should be further polished.

A1: Sorry for the writing problems in the paper. We have further polished our paper according to your advice as follows: firstly, we corrected double quotation mark mistakes. Secondly, we merge small ablations tables into a single large one to make the paper layout neater. Thirdly, though having "Ethics & Repoducibility Statement" on the 10th page is allowed, we think there is much redundancy in the paper and we reduce unnecessary parts to put everything except reference within 9 pages.

W2: The review of related work can be further improved.

A2: Thanks for your advice. We have revised that part according to your advice.

Sincerely,

Authors

Dear reviewer scbq,

Thank you very much for your review. Here are our responses:

W1: The authors should consider discussing the practical implications of tuning parameters like $\gamma$, especially when dealing with different large language models.

A1: Thanks for your advice. The magnitude of $\gamma$ implies the amount of "unlabeledness" in short AI texts. We hold that under identical training settings, $\gamma$ need to be set at higher values when large language models are more powerful. The reason is that LLMs that are more capable could generate sentences that are more alike to human. The "unlabeled" property is thus manifested more on short texts.

W2: It would be more convincing if the authors conducted an analysis of the performance of general recurrent language models. This analysis could include examining how well such models align with ground-truth labels and providing case studies to illustrate their effectiveness.

A2: Thanks for your advice of investigating the general recurrent models. We are afraid that we could not examine the models' alignment with ground-truth labels, because the attribution of short texts is a latent property. As explained in the paper, short texts are partially unlabeled. We could not know the precise attribution of a certain piece of text, and thus the label alignment analyses on the recurrent model could not be conducted. However, we could demonstrate the effectiveness of these general recurrent models via the detector's performance on benchmarks. As shown in the table below, we have compared the detector's performance with or without the recurrent model on the HC3-English benchmark in order to demonstrate the effectiveness of the General recurrent model.

Q1: Why does the BERT-MPU perform better than Roberta-MPU on full text but worse on short text?

A1: We hold that this gap is caused by the difference in difficulty of the two tasks, i.e. long text detection is much easier than short ext detection. Both BERT-MPU and RoBERTa-MPU reaches the performance supreme of full-level long texts; the rest of the minor cases that are falsely classified by both could be too difficult for existing detectors. In contrast, short text detection is a tricky task with more room for improvement. The gap of detection capability between BERT and RoBERTa is thus demonstrated.

Q2: Would the model's performance on classifying short texts improve if the training dataset only consists of short texts?

A2: Yes, MPU would also improve detectors that are trained on short texts only. In the TweepFake benchmark, all texts are short texts according to its token length statistics:

A performance improvement is also observed when the proposed MPU method is applied to RoBERTa according to the table below:

Q3: Does the proposed method harm the performance of extremely long texts?

A3: We analyze the performance on extremely long texts, which is defined as sentences longer than token length 256. As shown in the table below, results demonstrate that MPU actually improves extremely long texts.

Sincerely,

Authors

Dear reviewer PQsn,

Thank you very much for your review. Here are our responses:

W1: The motivation for using a PU framework for fake text detection is not entirely clear: Is there evidence that such shorts texts actually harm learning of a classifier using typical proper losses?

A1: We have conducted experiments to prove that such short texts harm the learning of the AI text detector. During finetuning, we remove sentences if they are too short. We do not change any other settings, including training hyperparameters and the dataset. As shown in the table below, compared with finetuning with all sorts of texts, finetuning without short sentences could achieve better performance on full-scale long texts and shorter sentence-level texts. The results indicate that short texts are harmful in finetuning. However, there is still much room for improvement in terms of short text performance. Hence, we try to leverage these "harmful" short texts in the PU framework rather than remove them in finetuning.

W2: Might other frameworks (e.g., noise-aware losses) besides PU be appropriate as baselines?

A2: Thanks for your suggestions. We add experiments on several popular noise-aware losses. The adopted noise-aware losses are as follows: MAE [1], RCE [2], and GCE [3]. These experiments use the identical training setting as our MPU loss experiments. The results are shown in the table below, revealing that the proposed MPU framework gains the upper hand.

W3: How does the use of the PU loss impact calibration? Uncertainty estimation is important for downstream use cases of AI text detectors.

A3: We conduct experiments on the HC3-English benchmark and report the calibration measures: Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) [4] on the full and sent datasets to reflect how calibration is impacted. It can be observed that MPU reduces error on both metrics.

W4: Section 2.3 is difficult to understand.

A4: Thank you very much for your advice. We have added more explanation about motivation in the latest revision.

W5: Too many unnecessary notations.

A5: We are sorry for the redundancy of notations. We have reduced them to make the equations concise.

Q1. Would an attention-based model also work here?

A1: We admit that attention is more powerful in language modeling compared to Recurrent Networks, as they jump out of the Markovian chain and establishes the dependency of one token with many other tokens. However, an attention-based might not work here in our scenario, because it is too complicated for our need of estimating the prior. We are estimating the general probability of a sample being positive without concrete samples. However, attention relies on concrete texts to establish the complex and precise dependency. Hence, we hold that an attention-based model might not work here.

Sincerely,

Authors

References

[1] Ghosh, Aritra, Himanshu Kumar, and P. Shanti Sastry. "Robust loss functions under label noise for deep neural networks." Proceedings of the AAAI conference on artificial intelligence. Vol. 31. No. 1. 2017.

[2] Zhang, Zhilu, and Mert Sabuncu. "Generalized cross entropy loss for training deep neural networks with noisy labels." Advances in neural information processing systems 31. 2018.

[3] Wang, Yisen, et al. "Symmetric cross entropy for robust learning with noisy labels." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[4] Naeini, Mahdi Pakdaman, Gregory Cooper, and Milos Hauskrecht. "Obtaining well calibrated probabilities using bayesian binning." In Proceedings of the AAAI conference on artificial intelligence, vol. 29, no. 1. 2015.

Dear reviewer o5iQ,

Thank you very much for your review. Here are our responses:

W1: The organization of the paper could be improved.

Thank you very much for your advice. According to your recommendations, we have re-organized the paper to reduce redundancy in the latest revision.

W2: There are some instances of misleading information in Section 2.2 regarding the estimation and expectation of PU learning.

Sorry for the misleading flaws in Section 2.2. We have corrected these mistakes in the latest revision (now in Appendix A).

Q1: What specific factors contribute to the difficulty in detecting short text? Is it feasible to accuurately differentiate between them?

The factor contributing to the difficulty in detecting short text lies in the reduced amount of information in contrast to longer texts. For longer texts, due to more information provided, they have prominent attribution to either AI or human. Hence, it is unsuitable two put both long and short text classification under the same PN classification framework.

In spite of the difficulty, we still hold that most short texts have clear attribution. As explained in "Introduction", only extremely short, information-poor texts are indistinguishable. For ordinary short sentences (like Example 1 in Table 1), the division between human and AI texts is clear from factors like punctuations, emotions, and formality. Classsifying short text is feasible.

Q2: Could you provide further insights into why \tilde \pi exhibits variation with respect to text length?

In fact, we do agree that \tilde \pi is based on the distribution of the whole dataset in the PU framework. However, for our case with multiscale texts, the difficulty of detecting long or short texts varys. Hence, long texts and short texts have different distributions, and it could be unsuitable for all kinds texts sharing different distributions to share the same prior \tilde \pi. In details, we hold that the distribution shift from long text to short text is a gradual process with respect to text lengths. In a differentiation perspective, texts of a certain length could be regarded as a small subset that features its own distribution and its own prior \pi. By setting prior estimation \tilde \pi as varying with respect to length, the gradual distribution change is considered. We have revised Section 2.3 (now Section 3.3) by adding further explanations.

Sincerely,

Authors