On the Empirical Power of Goodness-of-Fit Tests in Watermark Detection

Thank you for your professional and constructive review. We appreciate your recognition of the strengths of our paper and your thoughtful suggestions. Below are our responses to your specific points:

---

W1. A more straightforward application of GoF tests

Thank you for this valuable observation.

Indeed, no single GoF test consistently dominates across all scenarios—a fact we highlighted in Table 4. This diversity reflects the classical trade-offs in hypothesis testing: some tests are more sensitive to discrepancies in the tails (e.g., Anderson–Darling), while others emphasize central deviations (e.g., Watson’s test). Rather than prescribing a single solution, our goal in this paper is to empirically reveal these nuanced differences and encourage the use of GoF tests as a flexible toolkit for watermark detection.

That said, we agree that providing more actionable recommendations is valuable. We have summarized which GoF tests perform well under specific conditions (e.g., low vs. high temperature, common vs. targeted edits) in Table 4. We also view designing adaptive selection strategies—that choose tests based on input characteristics—as an important and promising direction for future work.

---

W2. Low-quality output in low-temperature

Thank you for raising this concern.

As discussed in Appendix D.2, outputs like “I; 1;1;1;1;1...” are extremely rare, even under low temperatures. These cases reflect pathological behaviors of the model and were kept for completeness. However, they account for a negligible fraction of the dataset and do not significantly influence overall detection performance in low-entropy settings.

---

W3. Statistical significance

We appreciate your suggestion.

We understand the value of including error bars to better visualize statistical variability. In our case, each detection outcome is binary (watermarked or not), so the Type II error is computed as the empirical mean over 1,000 i.i.d. Bernoulli trials. This setting allows us to estimate the standard deviation as $\sqrt{p(1-p)/N}$, where $p$ is the observed error rate and $N$ is the number of samples. Hence, all relevant statistical uncertainty is already captured by this mean, and we chose to report just the Type II error rate for clarity. We can include binomial confidence intervals in future versions if helpful.

We speculate the reviewer might be referring to variance across multiple runs (e.g., rerunning the entire experiment multiple times). However, this would be computationally expensive, and given our large sample size, the estimated error rates are already statistically stable.

---

Q1. How expensive are the GoF tests?

Thank you for raising this question. The most expensive cost for goodness-of-fit (GoF) tests is calculating the critical value.

Most GoF tests—including all those used in this work—lack non-asymptotic distributions but do have asymptotic ones. GoF tests with simpler asymptotic distributions—specifically Ney and Chi—can directly use asymptotic results to calculate critical values. As a result, their runtime is comparable to sum-based baselines.

However, several tests (e.g., Kui, Kol, And, Cra, Wat) do not use their asymptotic distributions for critical value computation (even though they have one) due to their complex analytical forms (e.g., the Anderson-Darling distribution [2, 3]). Instead, they rely on Monte Carlo simulation with a large number of samples (we use $10^5$) to estimate the critical value, which increases computational cost.

Below is the time cost for detecting 100 pivotal sequences with $n = 400$ and simulation size $10^5$. For comparison, the Ars baseline takes $1.30 \times 10^{-3}$ seconds. As shown in the table, GoF tests that rely on Monte Carlo simulation (Kui, Kol, And, Cra, Wat) take roughly $10^3$ times longer to run compared to the baseline. This is mainly due to the $10^3$-fold larger simulation size required for estimating critical values.

We will include this table in the Appendix.

---

Q2. Comparison with other statistical methods

This work addresses a different problem and is not directly comparable to ours. [1] focuses on detecting the presence and type of watermarking schemes in a black-box setting—without access to input parameters or watermark keys. This aligns more with an adversarial goal: identifying whether a model is watermarked and which scheme it uses.

In contrast, our work assumes the watermark scheme and secret keys are known, and focuses on detecting whether a given text sequence is watermarked—this reflects the use case of LLM APIs like Gemini.

That said, [1] is a very interesting and valuable contribution to the broader watermarking literature. We will mention it and discuss the difference in our next revision.

---

References

[1] Gloaguen, Thibaud, et al. "Black-box detection of language model watermarks." arXiv preprint arXiv:2405.20777 (2024).

[2] Anderson, Theodore W., and Donald A. Darling. "Asymptotic theory of certain" goodness of fit" criteria based on stochastic processes." The annals of mathematical statistics (1952): 193-212.

[3] Marsaglia, George, and John Marsaglia. "Evaluating the anderson-darling distribution." Journal of statistical software 9 (2004): 1-5.

Thank you for your thoughtful and detailed review. Below, we respond to your comments and questions point by point.

---

1. The choice of LLMs

We appreciate your concern regarding model selection.

We selected the OPT models primarily to ensure comparability with prior work [1–4], where they have been widely used. To examine the effect of model size, we included both OPT-1.3B and OPT-13B. Acknowledging that the OPT series is relatively outdated, we also included LLaMA 3.1-8B—a significantly stronger, modern model—to evaluate whether newer architectures materially affect watermark detection.

Our findings show that differences across models have minimal impact on the relative performance of detection methods. Instead, the entropy of the next-token probability (NTP) distribution—which governs how strongly watermark signals appear—plays a much more critical role. Since NTP entropy can be directly manipulated via temperature, we can effectively explore a wide range of detection regimes using temperature scaling, even on older models.

In this sense, using a newer model that naturally produces higher-entropy outputs is functionally similar to evaluating an older model at a higher temperature. As demonstrated in Appendix Figures 4–6, the performance trends across GoF tests remain consistent across models, reinforcing our claim that entropy, not model architecture, is the primary driver of detection difficulty in this setting.

---

2. Experiment clarification

Thanks for pointing this out.

In Appendix C.1, we provided the prompts used for text generation.

All three models were included in every detection task:

• Type I error on human data and Type II error evaluations (Table 2, 8; Figure 4, 5, 6)
• Type II error under various editing types (Table 3, 9)

For ablation studies:

• Results are averaged across the three models for repetition rate (left parts of Figures 2, 7, 8) and for changes in Type II error after removing repeated pivotals (Figure 3).
• In Figure 1, confidence intervals are based on pivotal sequences from all three models.

For illustrative purposes:

• In Figure 1, the three curves (light blue, dark blue, and orange) come from OPT-1.3B. This figure is meant to help readers intuitively understand the effect of repetition on the empirical CDF.
• In Figures 2, 7, and 8 (right parts), the top probability distributions are shown for OPT-1.3B. Although we ran the same experiments on OPT-13B and LLaMA 3.1, the patterns were nearly identical, so we omitted those plots for brevity.

Thanks for pointing this out—we'll include this clarification in the revision.

---

3. Shorter text sequence

This is an excellent point—thank you for highlighting it.

We did a brief analysis on shorter sequences. In Figure 1, we present the 95% confidence intervals of the empirical CDF for both long sequences ($n=400$) and short sequences ($n=100$). The figure shows that under high temperature ($T=1$), the empirical CDF diverges noticeably from the null CDF even when $n=100$. Also, Appendix Figures 4, 5, and 6 show how the Type II error decreases as the watermark sequence length increases ($n\in[1,400]$), with particularly rapid decay at $T=1$ and $T=0.7$. These results suggest that GoF tests can effectively detect watermark signals with relatively short sequences under high temperature.

However, we didn’t conduct a more detailed discussion on varying sequence lengths in the main text, as our focus is on demonstrating the broader potential of GoF tests as a general framework for improving watermark detection.

Thanks again for the thoughtful question—this is a promising direction for future work.

---

4. More details about the connection between GoF and watermark

Thanks for pointing out this concern.

We will certainly include more details in the Appendix. Below is a brief explanation to help avoid any potential misunderstandings.

• What is '$Y_{(t)}$' used in GoF tests?

$Y_t$ denotes the pivotal statistic (defined in L28) for the $t$-th token. Given a sequence $Y_1, \ldots, Y_n$ of pivotal statistics, $Y_{(1)} \leq Y_{(2)} \leq \dots \leq Y_{(n)}$ are their order statistics. For a fixed watermarking scheme, the distribution of $Y_t$ under the null hypothesis (i.e., human-written text) is known and denoted as $\mu_0$. Note that $\mu_0$ varies depending on the specific watermarking method—details are provided in Appendix A.

• Connection to GoF tests

Since we know that under the null hypothesis the pivotal statistics should always follow the known distribution $\mu_0$ independently, we can simply test whether a given sequence of $Y_1, \ldots, Y_n$ follows $\mu_0$. If not, we reject the null hypothesis and infer that the text is no way human-written and thus LLM-generated. To conduct this hypothesis testing, we use GoF tests—classical statistical tools for evaluating whether a sequence of i.i.d. samples comes from a specified distribution. More details are provided in Section 3: Rationale for Using GoF Tests.

• How are GoF test statistics calculated?

Each GoF test can be viewed as a function of the pivotal statistics $Y_1, \ldots, Y_n$, designed to measure the distance between their empirical distribution and the null distribution $\mu_0$. For example, consider the Kolmogorov–Smirnov (Kol) test applied to the Gumbel-max watermark with a significance level of $\alpha = 10^{-2}$:

1. Compute the pivotal statistics: $Y_1, \ldots, Y_n$
2. Sort them in ascending order: $Y_{(1)} \leq \dots \leq Y_{(n)}$
3. Compute the KS test statistic: $D_n = \max_{1 \leq i \leq n} (\max( Y_{(i)} - \frac{i - 1}{n}, \frac{i}{n} - Y_{(i)} ))$
4. Use Monte Carlo simulation to estimate the critical value $\gamma$ corresponding to the chosen $\alpha$
5. If $D_n > \gamma$, reject the null hypothesis; otherwise, accept it.

More details for other tests are provided in Section 3: Introduction of GoF tests.

---

5. Reiteration of known results (repeated n-gram problem)

Thanks for pointing this out.

Our work approaches the repetition issue from a different perspective.

First, we acknowledge that repetition in LLM outputs has been identified and analyzed in prior works, as we clearly state in the paper (L264). We'll also add [1] to the citations to better reflect this.

Second, we appreciate that [2, 3] addressed and effectively handled the repeated text issue in watermark detection. However, rather than focusing on how to deal with repeated text, we analyze why repetition makes sum-based methods less effective—see ‘The influence of repetition on top probabilities’. We further explore how repetition shifts the empirical CDF of pivotal statistics (not p-values) and impacts GoF detection—see ‘The influence of repetition on the empirical CDF’. This analysis brings out a key advantage of GoF: it’s particularly effective at capturing deviations between the empirical CDF of pivotal values and the null $\mu_0$.

That said, we agree that the current wording in the paper could cause confusion. We will revise it to clarify that we analyze the role of repetition in watermark detection from a new perspective—by examining its impact on the statistical behavior of pivotal values and the reliability of GoF-based detection, rather than proposing fixes or pre-processing heuristics.

---

References

[1] Li, Xiang, et al. "Robust detection of watermarks for large language models under human edits." arXiv preprint arXiv:2411.13868 (2024).

[2] Kuditipudi, Rohith, et al. "Robust distortion-free watermarks for language models." arXiv preprint arXiv:2307.15593 (2023).

[3] Li, Xiang, et al. "A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules." The Annals of Statistics 53.1 (2025): 322-351.

[4] Kirchenbauer, John, et al. "A watermark for large language models." International Conference on Machine Learning. PMLR, 2023.

Thank you for your thoughtful feedback. We are glad our clarifications improved your appraisal. The followings are our response to your comment.

1. The choice of LLMs

Thanks for continuing to discuss this with us. We respectfully offer the following clarifications:

• High watermarkablility makes detection easier so less focus on it. We agree with your point that temperature scaling cannot fully control or replicate the structure of the NTP distribution. As you noted, stronger models often produce multiple equally plausible next tokens, which makes them inherently more watermarkable. In such cases, detection is often easier than simply increasing the temperature of a weaker model.

  We acknowledge this distinction. However, in these easier cases, most detection methods—including ours—already achieve near-perfect performance. The performance on more watermarkable models is expected to be (and should be) even better. For this reason, the research community, including us, tends to focus on the more challenging scenarios, such as when the NTP distribution is peaked and exhibits relatively low entropy. This might explain why most watermarking studies—including ours—do not pursue the most recent LLMs, since the core difficulty lies in detecting watermarks under less unfavorable conditions, NOT in easy-to-detect cases. For example, [5] conducts experiments using the Gemma-Instruct models [6], which exhibit much lower-entropy NTP distributions compared to base Gemma models.

  That said, incorporating more recent models can certainly enhance the study and provide additional perspective, even if not strictly necessary. While the community often focuses on challenging detection settings, adding newer models could offer complementary insights. We will include this point in the limitations section (Section 5).

• Practical utility and flexibility of GoF tests. Our main contribution is introducing genera GoF tests to enrich the watermark detection toolkit. While we acknowledge the diversity of LLMs and text domains—and that GoF test performance may vary across different real-world cases—we emphasize that the cost of applying GoF tests is very low. As detailed in our response (Q1) to Reviewer jT6m, these tests require only lightweight statistical computations on CPUs and can be executed in parallel. Given this low overhead, users can run all suggested GoF tests and select the best-performing one in just a few minutes (depending on the time for computing critical values). To further support adoption, we will release well-organized Python code to make these tests easy to use in practice (note that we already attached them in this submission).

---

References

[5] Dathathri, Sumanth, et al. "Scalable watermarking for identifying large language model outputs." Nature 634.8035 (2024): 818-823.

[6] Team, Gemma, et al. "Gemma: Open models based on gemini research and technology." arXiv preprint arXiv:2403.08295 (2024).

2. Plan for the next revision

Thank you for the helpful suggestions. Based on your feedback, we outline our revision plan as follows:

• Clarify the scope of model generalization claims. We will revise the statement about consistency across models to explicitly state that the observed performance trends hold within the scope of the models tested. We will also acknowledge that differences in model architecture and training (e.g., between OPT and LLaMA 3.1) may influence detection outcomes. Additionally, we will clearly inform readers that, given the diversity of LLMs and text domains, greater variation in GoF test performance may exist than our current study suggests.

• Add clarifying phrases to captions and prose. We will insert concise clarifications in figure captions and relevant text to improve clarity and avoid ambiguity in the presentation of results.

• Add a pseudo-algorithm for applying GoF tests. We will include an enumerated, generalized pseudo-algorithm for implementing the GoF tests at the end of Section 3 (space permitting; otherwise, it will be added to the Appendix) to help readers better understand the implementation flow.

• Clarify novelty in how repetition affects GoF vs sum-based methods. We will strengthen our discussion of how repetition impacts both types of methods and emphasize our unique analytical perspective—specifically, by examining how repetition influences the statistical behavior of pivotal values and the reliability of GoF-based detection, rather than focusing on mitigation or preprocessing strategies. We will also clearly frame our repetition analysis as an edge-case study and point out that repetition is less prevalent in more recent LLMs.

• Further discussion on the influence of different LLMs. We will revise and expand the discussion in Appendix D.1 to include our rationale for the choice of models, acknowledge the limitations of not including newer models, and clarify the implications for generalizability. This should offer readers clearer guidance on interpreting our results in relation to model diversity.

Finally, thank you again for the thoughtful and constructive discussion, for your willingness to raise the score, and for your trust in us to thoughtfully incorporate the suggestions from all reviewers—your feedback has been truly appreciated.

Thank you for your professional review and valuable comments. Below are our responses, with clarifications and new evidence addressing your concerns.

---

Q1. Concerns about controlling Type I error

Thank you for highlighting the importance of Type I error control.

• Why we chose $\alpha = 0.01$:

  • First, we used 1,000 human texts for empirical evaluation. Under this sample size, $\alpha = 0.01$ is practical—it corresponds to 10 expected false positives, which is a reasonable estimate.
  • Second, we chose this threshold for consistency with prior influential work. Notably, Google’s SynthID watermark [2], reportedly ready for deployment in Gemini (link), also adopts $\alpha = 0.01$.

• Can GoF tests offer better Type I error control?

  Yes. To further assess this, we sampled 100k texts from the C4 dataset and removed repeated pivotals, following the approach in [1]. Even at this larger scale, GoF tests still maintain strong Type I error control—for example, achieving empirical FPRs as low as $10^{-5}$. The table below shows theoretical FPR vs. empirical FPR (We also created a plot similar to Figure 2 in [1], as suggested by the reviewer, but we cannot include it here due to rebuttal constraints; it will be added to the paper.).

• Why GoF works in practice:

  Beyond the watermark’s design, the key factor in maintaining a low Type I error is how closely the pseudo-randomness resembles true randomness—this was also highlighted in [1]. For GoF tests, even though they lack non-asymptotic distributions, they can still offer reliable Type I error control in practice. This is because the pivotal statistics for human-written tokens behave almost like samples from a true uniform distribution after repeated values are removed. The concern about non-asymptotic distributions is further addressed in W2.

---

Q2. Reiteration of known results (repeated n-gram problem)

Thanks for pointing out this issue.

We agree that [1] pointed out and addressed the repeated text issue in watermark detection. However, our work studies the same issue from a different perspective. Specifically, rather than focusing on how to deal with repeated text, we analyze its effects on watermark detection, such as why repetition makes sum-based methods less effective—see ‘The influence of repetition on top probabilities’. We also further explore how repetition shifts the empirical CDF and impacts GoF detection in ‘The influence of repetition on the empirical CDF’. More importantly, this analysis highlights a key strength of the GoF method: it’s particularly effective at detecting differences between the empirical CDF and the null $\mu_0$.

---

W2. Lack of significance/usefulness

We appreciate your concern about the absence of non-asymptotic guarantees.

We acknowledge that Aaronson's score has a non-asymptotic distribution, but a non-asymptotic distribution is NOT necessary to well-control the Type I error. The most straightforward way to ensure valid Type I error control is to use test statistics with a known non-asymptotic distribution. Most GoF tests in the statistics literature only have known limiting distributions as $n \to \infty$ (we will include the limiting distributions of the GoF test statistics we use in the Appendix). Despite this, the empirical results in the table above show that these tests already provide practically reliable Type I error guarantees.

It’s also worth noting that the limiting distributions offer valuable theoretical tools, but in practice, simulations offer users a more flexible and convenient way to compute critical values—like in Python package SciPy, due to the complex forms of some distributions (e.g., the Anderson-Darling distribution [3,4]).

Finally, simulation is a theoretically rigorous approach from both theoretical and practical perspectives. Its theoretical validity parallels that of conformal inference: for i.i.d. data points, the empirical $\alpha$-quantile of the sampled statistics serves as a valid critical value for a new test point due to the property of exchangeability. From a practical standpoint, simulation is also widely adopted in the watermarking literature—for instance, [2] uses a simulation-based method to estimate the distribution of human text (see Appendix A.4).

---

Q3. The choice of LLMs and analysis of the entropy

Thanks for raising this question.

Many previous works used OPT models in their experiments[5-8], so we chose OPT to stay consistent for comparison. We also included two versions of OPT to examine whether model size significantly impacts GoF performance. If you think that OPT is relatively outdated, we also tested with the more recent LLaMA 3.1 to see if newer models yield noticeably different results.

Instead of reporting entropy directly, we analyze top probabilities, which closely track entropy: For the NTP distribution $\mathbf{P} = (P_1, ..., P_{|\mathcal{W}|})$ where $\mathcal{W}$ denotes the LLM's vocabulary, the entropy is approximated by $\operatorname{Ent}(\boldsymbol{P})=\widetilde{\Theta}\left(1-\max _w P_w\right)$ up to logarithmic factors. Figures 2, 7, and 8 present these top probabilities for repeated tokens, helping us infer entropy concentration patterns at low temperatures.

---

Q4. Text length

Thanks for raising this question.

The length of the generated texts is fixed at 400 tokens—consistent with prior works [2, 5, 7]—across all models, temperatures, and watermarking methods, unless explicitly stated otherwise.

---

References

[1] Fernandez, Pierre, et al. "Three bricks to consolidate watermarks for large language models." 2023 IEEE international workshop on information forensics and security (WIFS). IEEE, 2023.

[2] Dathathri, Sumanth, et al. "Scalable watermarking for identifying large language model outputs." Nature 634.8035 (2024): 818-823.

[3] Anderson, Theodore W., and Donald A. Darling. "Asymptotic theory of certain" goodness of fit" criteria based on stochastic processes." The annals of mathematical statistics (1952): 193-212.

[4] Marsaglia, George, and John Marsaglia. "Evaluating the anderson-darling distribution." Journal of statistical software 9 (2004): 1-5.

[5] Li, Xiang, et al. "Robust detection of watermarks for large language models under human edits." arXiv preprint arXiv:2411.13868 (2024).

[6] Kuditipudi, Rohith, et al. "Robust distortion-free watermarks for language models." arXiv preprint arXiv:2307.15593 (2023).

[7] Li, Xiang, et al. "A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules." The Annals of Statistics 53.1 (2025): 322-351.

[8] Kirchenbauer, John, et al. "A watermark for large language models." International Conference on Machine Learning. PMLR, 2023.

Thank you for your thoughtful and constructive feedback. We're glad that our clarifications have addressed most of your concerns. As you noted, our primary point of divergence lies in the asymptotic nature of GoF tests and their role in practical detection systems.

---

1. Type II Error at Low FPR

Due to the character limit in the initial rebuttal, we were unable to include Type II results. We now report Type II errors at a stricter Type I error level of $10^{-5}$ for temperature 0.7 and sequence length $n = 400$, complementing Table 2. Here, “Baseline” denotes the best-performing method among all baseline detectors. Results are averaged over three LLMs. We will add the additional results in the Appendix (e.g., Type II errors at $10^{-4}$ and $10^{-3}$ Type I errors).

These results demonstrate that GoF tests retain strong detection power even under tighter FPR constraints, especially at higher temperatures as discussed in Section 4.2.

---

2. On Bounding Type I Error in GoF Tests

We respectfully offer the following clarifications to your points about bounding FPR:

• Confidence intervals: The accuracy of empirical critical values can be arbitrarily improved by increasing simulation size. Since the standard error of quantile estimates scales as $1/\sqrt{n}$, simulating more samples narrows the confidence interval and increases reliability. Importantly, this calibration can be done before deployment, given that the distribution of pivotal statistics is known for a given watermarking scheme.

• Match between simulation and test-time distributions: In our setup, the pivotal statistics (e.g., for Gumbel-max) follow a known distribution (e.g., uniform on \[0,1]\) both in simulation and test-time. Thus, the test statistics used in GoF methods follow the same distribution in both phases. If this assumption were violated, even exact non-asymptotic methods (e.g., Ars) would lose their guarantees, as they also rely on this distributional assumption.

Therefore, we argue that both criteria for bounding FPR can be satisfied in our framework.

---

3. Complementary Role of GoF Tests

We do not view GoF tests as a replacement for sum-based or non-asymptotic methods, but rather as complementary tools. While methods like Ars target high-value pivotal statistics, GoF tests are designed to detect distributional shifts, offering enhanced robustness especially under semantically rich or information-preserving edits (see Section 4.3). Despite relying on asymptotic or simulated critical values, GoF tests enrich the detection toolkit by capturing distribution-level anomalies.

We agree that in high-stakes applications (e.g., legal proceedings), relying solely on asymptotic methods may be insufficient. However, we believe that no single method should serve as the only defense. A robust system should draw from multiple methods, each with different strengths—e.g., Ars for sensitivity to large deviations and GoF for distributional anomalies.

To reflect this point, we will add a discussion on both the rationale for using GoF tests and the potential risks associated with their asymptotic nature, to help guide and inspire future work toward more practically robust solutions.

---

Finally, we sincerely appreciate your deep engagement with the paper and for considering an increase in your score. If further clarification is needed on any remaining points, we would be happy to provide it.

Thank you for your thoughtful review and encouraging comments. We appreciate your recognition of our contribution and your constructive suggestions for strengthening the paper. Please find our responses to your specific points below:

---

1. A more straightforward application of GoF tests

We appreciate your suggestion to provide more practical guidance.

Indeed, no single GoF test consistently dominates across all scenarios—a fact we highlighted in Table 4. This diversity reflects the classical trade-offs in hypothesis testing: some tests are more sensitive to discrepancies in the tails (e.g., Anderson–Darling), while others emphasize central deviations (e.g., Watson’s test). Rather than prescribing a single solution, our goal in this paper is to empirically reveal these nuanced differences and encourage the use of GoF tests as a flexible toolkit for watermark detection.

That said, we agree that providing more actionable recommendations is valuable. We have summarized which GoF tests perform well under specific conditions (e.g., low vs. high temperature, common vs. targeted edits) in Table 4. We also view designing adaptive selection strategies—that choose tests based on input characteristics—as an important and promising direction for future work.

---

2. Possibility of a unified evaluation scale

Thank you for this thought-provoking suggestion.

While our current focus is on evaluating individual GoF tests, we agree that aggregating multiple tests into a unified scale or ensemble could further improve detection. In classical statistics, combining test statistics (e.g., via p-value aggregation or voting rules) is often used to enhance power while maintaining Type I error control. Applying these techniques in watermark detection, especially under editing or adversarial perturbations, is an interesting direction that could yield practical benefits.

We will add a discussion in Section 5 to explicitly raise this as a potential future extension of our work.

---

3. More diverse datasets

We agree that evaluating on additional datasets is always beneficial and would further validate the generality of our findings.

Our current selection of C4 and ELI5 follows previous studies [1–5] and provides complementary text domains: C4 for factual completions and ELI5 for long-form generation. These allow us to capture a range of generation behaviors and model outputs.

---

4. How to improve the green-red watermarking scheme?

We appreciate your engagement with this limitation.

As you noted, the green-red list watermark produces binary pivotal statistics (i.e., whether each token is green), which fundamentally restricts the applicability of many GoF tests. Most GoF tests are designed for continuous or finely discretized data, where empirical CDFs offer rich signal. In contrast, binary statistics yield only counts, reducing the problem to a binomial test.

To make GoF-style methods applicable in this context, a promising direction is to design enriched decoders that produce more informative pivotal statistics beyond the binary green/red indicator. For instance, such decoders could expose additional features like token ranks, transition patterns between green and red tokens, or context-aware scores that reflect how strongly a token conforms to the watermarking policy. These enriched statistics would enable a more fine-grained analysis of distributional shifts and allow GoF tests to be meaningfully applied. Developing such decoders could bridge the gap between the widely adopted green-red watermarking scheme and the broader GoF-based detection framework explored in our work. However, this requires additional watermark design, which is beyond the scope of our current study.

---

References

[1] Li, Xiang, et al. "Robust detection of watermarks for large language models under human edits." arXiv preprint arXiv:2411.13868 (2024).

[2] Kuditipudi, Rohith, et al. "Robust distortion-free watermarks for language models." arXiv preprint arXiv:2307.15593 (2023).

[3] Li, Xiang, et al. "A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules." The Annals of Statistics 53.1 (2025): 322-351.

[4] Kirchenbauer, John, et al. "A watermark for large language models." International Conference on Machine Learning. PMLR, 2023.

[5] Dathathri, Sumanth, et al. "Scalable watermarking for identifying large language model outputs." Nature 634.8035 (2024): 818-823.