Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model

Weaknesses1: My main concerns with this paper are particular limitations of the evaluation of the proposed method. First of all, only one dataset is used - DetectRL - and it covers models primarily from the first half/mid-2023. Some more modern models (e.g., GPT4-o or something else) would be appreciated.

Thank you for raising this important point. We agree that evaluating detectors on outputs from more modern models is crucial, especially given their increasing prevalence in real-world use cases.

To address this, we conducted additional experiments on the DivScore [1] benchmark, which includes four professionally legal and medical datasets generated by more modern LLMs: DeepSeek-R1, DeepSeek-V3, GPT-4o, and O3-mini. For evaluation, we use the main test split of DivScore and adopt LLaMA-3.2-1B as the backbone, which demonstrated strong performance on DetectRL. In addition to AUROC and F1, we also report false positive rate (FPR) and false negative rate (FNR), as these provide a more nuanced understanding of detection quality in high-stakes settings.

Across both tables, IRM demonstrates consistently strong performance on recent reasoning models, achieving high AUROC and F1 while also maintaining low FPR and FNR. These results support the robustness and reliability of IRM for detecting high-quality responses from modern reasoning LLMs.

Weaknesses2: Secondly, evaluation metrics. In practice, type I and type II errors (false positive and false negative) have different weights. None of the metrics provided for the experiments (AUROC and $F_1$) cover this aspect; I would suggest providing FPR/FNR (or any other appropriate metric).

Thank you for this valuable suggestion. While the standard evaluation protocol of the DetectRL benchmark does not include FPR/FNR metrics, we agree that false positive and false negative rates are important in practical applications. To address this, we report below the FPR and FNR for our method and competitive baselines on the multi-LLM sub-task of the DetectRL benchmark, using Llama-3.2-1B as the backbone.

As shown, IRM consistently achieves lower FPR and FNR across models.

Weaknesses3: Also, I would like to ask if it is possible to include metrics for baselines in Table 3?

Thank you for the suggestion. We provide below an extended Table 3 with detailed results for competitive baselines.

Weaknesses4: On another note, a clearly outlined pipeline for application of the proposed method to a text (possibly, an example) in the paper (appendices) would be much appreciated.

Thank you for the suggestion. We agree that a clearly outlined pipeline would improve the clarity and usability of our method. In the final version, we will include a figure that illustrates the full application process of IRM with a step-by-step example. This figure will demonstrate how to compute the implicit reward scores from a given text and how these scores are used to make the final classification decision.

[1] DivScore: Zero-Shot Detection of LLM-Generated Text in Specialized Domains

Weaknesses 1 and Questions1: Didn't analyzed about detecting reasoning models: DetectRL doesn’t contains recent reasoning models’ responses such as openai o3 or deepseek-r1. Detecting recent reasoning models: Have you evaluated IRM and other baselines with outputs from reasoning models such as OpenAI o3 or Deepseek-R1?

Thank you for raising this important point. We agree that evaluating detectors on outputs from strong reasoning models is crucial, especially given their increasing prevalence in real-world use cases.

To address this, we conducted additional experiments on the DivScore [1] benchmark, which includes four professionally legal and medical datasets generated by recent reasoning-oriented LLMs: DeepSeek-R1, DeepSeek-V3, GPT-4o, and O3-mini. For evaluation, we use the main test split of DivScore and adopt LLaMA-3.2-1B as the backbone, which demonstrated strong performance on DetectRL. In addition to AUROC and F1, we also report false positive rate (FPR) and false negative rate (FNR), as these provide a more nuanced understanding of detection quality in high-stakes settings.

Across both tables, IRM demonstrates consistently strong performance on recent reasoning models, achieving high AUROC and F1 while also maintaining low FPR and FNR. These results support the robustness and reliability of IRM for detecting high-quality responses from modern reasoning LLMs.

Questions2: Can you provide evaluation results on detecting randomly sampled generations by adjusting temperatures?

Thank you for this insightful question. Sampling temperature indeed has a significant influence on the quality and diversity of LLM-generated outputs. However, existing benchmarks, such as DetectRL, typically fix the temperature at a scalar value (e.g., 1.0) and do not explore how detection performance varies across temperature settings.

This design choice is grounded in practical usage. In many real-world applications, such as ChatGPT or similar LLM-based systems, users cannot directly control temperature or generation strategy, and the outputs are produced under a fixed decoding setup predefined by the provider. Therefore, the kinds of generations actually presented to users are restricted to a narrow range of temperature settings.

[1] DivScore: Zero-Shot Detection of LLM-Generated Text in Specialized Domains

Weaknesses1: The motivation for specifically focusing on implicit reward models is unclear to me. Aside from DPO-style reward models, you could use any available reward model for this purpose (sequence classifier or even generative reward models). There are a bunch in RewardBench. Do you have a reason for focusing on implicit reward models specifically?

Thank you for the insightful question. While it is possible to use various existing reward models such as sequence classifiers or generative reward models, our focus on implicit reward models is motivated by both practical considerations and theoretical alignment with the nature of LLM-generated text.

Specifically, many publicly available reward models are trained for preference classification, not detection of LLM-generated content. These are fundamentally different tasks. Furthermore, reward models used during real-world RLHF pipelines are not publicly released, limiting their practical applicability in detection.

In contrast, DPO-style implicit reward models offer a principled proxy for the unknown reward functions used during RLHF. The log-likelihood difference between an instruction-tuned model and its base model can be interpreted as an implicit reward signal that reflects how much a given response is encouraged during alignment training. This aligns well with the over-optimization nature of LLM-generated text, making it a natural fit for detection tasks.

Finally, we conducted empirical evaluations of several SOTA public reward models trained on preference data (see the rebuttal for Question 3), and observed that they perform poorly on LLM-generated text detection tasks—further validating the effectiveness of our implicit reward model approach.

Questions1: On Table 5, as the paper states, "larger or newer versions of a model do not always lead to better detection performance." Do you have any interpretation of why this is the case? Intuitively, it seems that the implicit reward model from a larger backbone should be better at estimating reward, which should, in turn, improve its ability to recognize "overoptimized text."

Thank you for raising this insightful question. While larger models tend to be better at estimating reward signals within the distribution they are trained on, the zero-shot detection task inherently involves a cross-distribution generalization challenge. Specifically, IRM is designed to detect whether a given text has been overoptimized with respect to another model’s reward signal—not the preferences of the IRM model itself.

In practice, different LLMs are trained on diverse datasets and optimized via different RLHF pipelines. Even when they target similar alignment goals (e.g., helpfulness, factuality), the actual learned preferences can differ significantly due to differences in training data and optimization processes. This phenomenon, which we refer to as reward divergence, means that larger models do not necessarily provide better detection signals—sometimes, a smaller model generalizes better across such divergent reward landscapes.

Questions2: Have you tried larger models, such as 7b models or beyond? I'd be interested to see whether the scaling trends continue to hold.

Thank you for raising this insightful point. We extend our analysis in Table 5 by evaluating IRM with a range of backbone sizes. The table below reports the results on the DetectRL benchmark:

These results show that larger backbone models do not necessarily lead to better detection performance in the IRM framework. This trend is particularly evident within model families, where smaller models often outperform their larger counterparts. This supports our claim that IRM's success relies more on cross-model generalization than on reward estimation fidelity, and that larger models may overfit to their own reward landscape, reducing their effectiveness in detecting overoptimization from other models.

Questions3: Could you try the same method with other types of reward models based on the same backbones (without further fine-tuning)?

Thank you for the suggestion. We conducted additional experiments using public reward models trained on large-scale preference data, specifically the Skywork-Reward-V2 [1] models, which achieve SOTA results on the RewardBench benchmark. These reward models are trained on 40 million preference pairs and use backbones up to 8B in size, allowing us to assess whether reward model scaling improves generalization in LLM-generated text detection.

We compare IRM and Skywork-Reward-V2 under the same backbones, without additional fine-tuning, and report their performance on the DetectRL benchmark.

These results demonstrate that IRM consistently outperforms the reward-model-based methods, even when using the same or smaller backbones. This further supports our claim that standard preference-trained reward models, despite their size and strong performance on preference classification, do not generalize well to the LLM-generated text detection task. In contrast, IRM’s implicit reward modeling remains robust across settings.

[1] Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy

Weakness1: Intuitively, if text comes from the preference dataset, shouldn't they have high implicit reward, even if written by a human, since the preference dataset should align more with what humans would write? This seems to contradict the intuition provided in the paper (Figure 1, right side). I am curious how this figure would look if we compared the implicit reward distribution between preference data, non-preference data, for both human and LLM versions. While any detector signal (such as implicit reward) can in theory be used, it would be great to clarify the intuition behind the implicit reward.

Thank you for the thoughtful comment. We understand the intuition that preferred texts should receive high reward scores, possibly even human-written ones. However, we would like to clarify a key point: in RLHF pipelines, preference datasets consist of multiple LLM-generated responses, which are then ranked by human annotators. They do not include human-written completions labeled as “preferred.” This distinction is critical: the reward model is trained to distinguish between better and worse LLM outputs, not between LLM-generated and human-written texts.

We believe this confusion may stem from conflating “human-preferred” with “human-written.” While human-written texts may be of high quality, they are not optimized to align with the reward model. In contrast, LLM outputs in RLHF are directly optimized to maximize reward scores. This explains the distribution shown in Figure 1 (right): LLM outputs tend to have higher IRM scores than human-written ones, reflecting their alignment with the learned reward model.

This misunderstanding is further compounded by how ReMoDetect constructs its training data. Unlike RLHF preference datasets that rely on pairwise human annotations, ReMoDetect labels LLM outputs as “preferred” and human-written texts as “dispreferred” without any human annotation. While useful for detection tasks, they differ substantially from the RLHF reward model’s training setup and may inadvertently blur the line between "human-preferred" and "human-written."

We also appreciate the suggestion of comparing IRM score distributions across more fine-grained categories (e.g., preference vs. non-preference subsets for both LLM and human text). While insightful, such comparisons would require carefully constructed datasets containing both LLM-generated and human-written responses with explicit human preference annotations—which, to our knowledge, do not currently exist, as they would be prohibitively expensive and would undermine the scalability motivations behind RLHF.

We hope this clarification helps reconcile the intuition and resolves the concern.

Weakness2: An additional benchmark dataset would help strengthen the paper, such as RAID. While DetectRL covers many scenarios, it would help support the papers claims if at least one other standard benchmark dataset suite is included.

We sincerely appreciate the reviewer’s thoughtful suggestion to evaluate our method on an additional benchmark dataset such as RAID. We agree that demonstrating generalization across diverse benchmarks is important for validating the robustness of detection methods. In response, we have conducted comprehensive experiments on two benchmark in addition to DetectRL: RAID [1] and DivScore [2].

We firstly evaluate our method on the RAID benchmark. Due to the large size of the dataset, we randomly sample 512 annotated pairs for each source model from the training split for evaluation. We adopt LLaMA-3.2-1B as the backbone detector, as it showed strong performance on DetectRL. We report AUROC as the main evaluation metric.

As shown in the above Table, IRM achieves competitive performance across a wide range of models, demonstrating the generalizability of our approach beyond the original DetectRL setup.

To further assess our method’s robustness in domain-specific and high-stakes settings, we additionally evaluate on DivScore [2], a recently released benchmark containing human preference data in legal and medical domains. It includes outputs from four reasoning-oriented LLMs: DeepSeek-R1, DeepSeek-V3, GPT-4o, and O3-mini. We use the official test split for evaluation and adopt LLaMA-3.2-1B as the backbone.

Across both tables, IRM demonstrates consistently strong performance on recent reasoning models, achieving high AUROC and F1 while also maintaining low FPR and FNR. These results provide strong empirical evidence for the reliability and robustness of IRM when applied to real-world, safety-critical domains.

We hope these additional results address the concern and further strengthen the empirical foundation of our work.

Weakness3: It would be helpful to understand the effect of LLM size on the detector performance, similar to [3] for DetectGPT. This would help users understand which instruction-tuned/base model policy pairs and their sizes are the most "universal" across LLMs and datasets.

Thank you for raising this insightful point. We agree that understanding how the detector’s backbone size influences performance is important for identifying practical and robust detection strategies. To this end, we extend our analysis in Table 5 by evaluating IRM with larger (>7B) backbones. The table below reports the results on the DetectRL benchmark:

We observe a similar trend to that reported in [3]: smaller models within the same family often yield better detection performance. Notably, Llama-3.2-1B backbone achieves the best overall results, suggesting that smaller models may act as more universal detectors across various source models and datasets.

Weakness4: Is there a fully-supervised classifier baseline, such as RoBERTa? The fully supervised models seem trained for reward modeling, but I wonder how a baseline that directly tries to classify human/LLM text would compare.

Thank you for raising this important point. DetectRL benchmark provides 4 supervised baselines including models such as RoBERTa and XLM-RoBERTa. We report the results of these supervised baselines and IRM:

We find that IRM outperforms 3 out of the 4 supervised baselines, despite requiring no task-specific training. This highlights the strength of our framework in a zero-shot setting.

[1] RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors

[2] DivScore: Zero-Shot Detection of LLM-Generated Text in Specialized Domains

[3] Smaller Language Models are Better Black-box Machine-Generated Text Detectors