Revealing Weaknesses in Text Watermarking Through Self-Information Rewrite Attacks

We are thankful to Reviewer 7XjW for the thorough and detailed feedback, due to space limit, we address the main concerns below:

Q1: A Pareto plot, the use of \epsilon, Line 1041

A1: We greatly appreciate the reviewer’s suggestion. We will add the Pareto plot, correct the misuse of \epsilon, and revise Line 1041 to avoid confusion.

Q2: Directly computing how well the proposed metric predicts tokens will be helpful

A2: Due to space limitations, please see the reply to zCjb A1.

Q3: It would be useful to see the naive baseline results of a paraphrasing attack.

A3: We clarify that this is already included in our paper. As shown in Table 3, we compare our method to a naive baseline where the LLM paraphrases the input twice. This aligns with our approach, where contains two paraphrasing for reference text and attack text. Results show that our design outperforms naive baseline.

Q4: Possible explanations for the empirical and theoretical

A4: We clarify that prior works claim to embed watermarks in high-entropy is a heuristic description, they do not explicitly compute token-level entropy to build the greenlist. Instead, greenlist tokens are selected via a hash-based pseudorandom process conditioned on past tokens and a key, implicitly simulating high-entropy selection.

For example, in EWD [3], The high entropy concept is represent as a renormalization step:

renormed_probs = probs / (1 + z_value * probs)
sum_renormed_probs = renormed_probs.sum(dim=-1)

This renormalization acts as soft reweighting that suppresses high-probability tokens and flattens the distribution, thereby emphasizing low-probability regions. As a result, low-probability tokens are further highlighted and tend to receive higher weights regarding the watermark signals. Similarly, EXP uses exponential sampling to favor low-probability tokens, which, once selected, have higher weight during detection.

In summary,low-probability tokens are either preferred when selecting a green token or assigned stronger watermark weights once selected. Our self-information approach aligns more directly with practical implementation, enabling more accurate identification of watermark-bearing tokens and improving attack effectiveness.

Q5: The formulation in Sec 3.1 doesn't connect watermarking to token-level perturbations .

A5: We clarify that the perturbation function in Section 3.1 refers to the LLM used for paraphrasing. Definition 3 is not meant to describe token-level watermark perturbations. It corresponds to the second step of our method, where the LLM is instructed to paraphrase the input.

Q6: Why is the green token more important?

A6: The watermark signal is embedded by biasing generation toward green tokens, while red tokens act as a control group. Detection relies on whether the green token significantly exceeds the non-watermarked baseline. Thus, the green tokens carry the actual signal. Regarding distortion, robust watermarking schemes are explicitly designed to minimize it while preserving detectability; otherwise will affect text quality and stealth.

Q7: Other dataset experiment

A7: We follow the reviewer’s suggestion add the OpenGen dataset [2], which samples from WikiText-103. We use the 500 chunks as prompts, following the same evaluation protocol as in our main experiments. We show our results below, and our methods achieve the highest ASR.

Q8: Waterfall watermark experiment

A8: We followed the reviewer’s suggestion to conduct the experiment on the waterfall watermark[1]. We use 500 samples from the C4 dataset and select 0.2 as our z-threshold; others follow the same evaluation protocol as in our main experiments. The results shown below, our methods get the highest ASR.

Q9: Weaknesses regarding theoretical

A9: We thank the reviewer for the constructive suggestion. We acknowledge that our work is primarily empirical, and while we attempted to provide theoretical insights, some parts may be limited or unclear. We will revise the manuscript accordingly and de-emphasize those sections as suggested. We appreciate any further feedback to improve the theoretical part.

Anonymous Reference Link: https://docs.google.com/document/d/1-VCOEO5eJmrq-_44oGaHdRE7KOGXHumV75h9DDCTP8A/edit?usp=sharing

We are thankful to the reviewer P6b2 for the appreciation of our work and the efforts spent to review our paper. We address concerns and questions below:

Q1: Text quality slighter lower than GPT paraphrase

A1: We thank the reviewer for their suggestion and will revise the description of our experimental conclusions accordingly. We fully agree with the reviewer and would like to clarify that text quality is largely influenced by the choice of paraphrasing model. In our case, we adopted GPT-4o; the inherent ability difference of LLMs causes such a gap. For the stronger LLM like LLaMA3-70B, it outperforms GPT in two watermarks. One advantage of our method is its transferability: unlike DIPPER, our approach is training-free. As more powerful LLMs become available in the future, our method can be seamlessly adapted with zero cost to take advantage of better text quality.

Q2: Does this missing prompt affect the calculation of self-information?

A2: This is a great question. Our answer is no—the calculation of self-information does not need such a prompt. The prompts used to generate watermarked text in prior works such as DIPPER [1] and Random Walk [2] are typically referred to as “context.” However, our goal is to minimize assumptions in order to make the method a simple and broadly applicable tool. In the results we present, we do not use any such context prompts. In earlier studies, the original prompts were primarily designed to help paraphrasing models better preserve semantics through tailored instructions. We consider exploring the impact of prompt design on our method as an interesting direction for future work.

Q3: If the given watermarked text is cropped from a long output, will the missing context affect the calculation of self-information?

A3: Yes, but it depends on how long the cropped text is. The missing context theoretically affects how self-information is calculated since it depends on the exact preceding context. However, our watermarking approach remains effective in practice because it primarily relies on local context within the segment, rather than requiring the entire text. Consequently, cropping a reasonable portion of the output does not significantly degrade our detection or performance unless the given text is too short or the different segment contexts are unrelated. In such an extreme case, our method degrades to a performance level similar to that of random masking.

Q4: Using instruction instead of reference text

A4: We thank the reviewer for the suggestion. This was indeed one of the approaches we explored during the preliminary phase of our study. Adding explicit instructions to encourage the LLM to use different words proved to be ineffective; a large performance gap will exist, especially for lightweight models such as those at the 3B scale. We believe this is because such a small model cannot fully understand complex instructions. While this approach performs comparably on larger models (e.g., 70B), it still underperforms compared to using a reference text. This is likely because, as mentioned in lines 256–260, we instruct the model to do tasks similar to filling in blanks, and due to the high similarity between mask text and watermarked text, LLMs tend to take shortcuts by copying directly from the original text; instructions are not sufficient to fully prevent this behavior. We will continue to explore whether better prompt design can help to achieve this goal.

References

[1] Krishna, Kalpesh, et al. "Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense." Advances in Neural Information Processing Systems 36 (2023): 27469–27500.
[2] Zhang, Hanlin, et al. "Watermarks in the sand: Impossibility of strong watermarking for generative models." arXiv preprint arXiv:2311.04378 (2023).

We are thankful to the reviewer RVPR for the valuable time and effort spent reviewing our paper. We elaborate on the questions raised by the reviewer below:

Q1: Clarification regarding claims: SIRA-Tiny method outperforms all previous approaches

A1: We would like to clarify that this statement only refers to the fact that SIRA-Tiny outperforms previous methods in terms of ASR under the same watermark strength. We agree that a more precise statement would be better. We will revise the text to reflect this more precise phrasing to avoid overclaim and add a corresponding note in Appendix C, Table 6.

Q2: Lack of support: Claim SIRA-Large achieves better quality than DIPPER2, while Figure 3b shows that SIRA-Large has slightly worse quality than DIPPER2 on the same watermark

A2: We would like to clarify that in watermarking, text quality is typically referred to the perplexity metric, which is shown in Figure 3a. In contrast, Figure 3b, which the reviewer referenced, reports S-BERT scores that evaluate semantic preservation, a different aspect from text quality. Therefore, we respectfully argue that our claim that SIRA-Large achieves better quality than DIPPER2 remains valid. We appreciate the reviewer's feedback and will revise the related part more clearly to avoid potential misunderstandings.

Q3: Computation regarding the cost

A3: We provide a more comprehensive cost analysis here and will include it in the relevant section of the manuscript. Specifically, we estimate the cost of processing 1M tokens of watermarked text using third-party services. According to OpenAI’s pricing, using the GPT Paraphraser (GPT-4o) would cost $10 × 2 = $20 (input + output). In contrast, our method (SIRA-Small) costs $0.22 × 2 (input + output) × 2 (two iterations) = $0.88, based on the AWS Bedrock LLaMA3-8b pricing. And the cost could be further reduced if we use SIRA-Tiny (LLaMA3-3b).

Q4: Why is the self-information result different from probability?

A4: We clarify that the self-information in our method is conditional and chunk-based (e.g., calculated and percentile by segment), whereas the raw token probabilities are derived from the LLM’s output logits. This means that the statistical basis for the two measures is not aligned: the conditioning context and granularity differ. As a result, percentile ranks based on self-information may diverge from those based on raw probability. This mismatch leads to divergent percentile rankings and explains why filtering using self-information yields different token selections than filtering using raw probability. We will release our code to ensure reproducibility and transparency, and we appreciate the suggestion regarding potential numerical differences, as we will further increase the sample size to reduce numerical instability.

Q5: Why not measure the similarity with respect to the watermarked text instead?

A5: For the attack cases shown in Figure 3b and Table 2, our SBERT-based similarity is indeed computed between the attack text and the watermarked text as proposed by the reviewer. As SBERT needs one pair of texts, "No Attack" is a special case; comparing the watermarked text to itself would be meaningless. Therefore, for “No Attack” we compare the watermarked text with the non-watermarked text. This term aims to reflect how the watermark algorithm changed the semantics. We acknowledge that this was not clearly explained in the paper. We will clarify this to avoid potential misunderstanding.

We are thankful to the reviewer zCjb for the time spent reviewing our paper. Due to the reply character limit, we address the main concerns below and put the reference in anonymous link:

Q1: No theoretical analysis or direct experimental evidence on the high-entropy assumption

A1: The use of high-entropy embeddings is not an assumption but a well-established design choice in prior watermarking research [1–4, 6, 7]. Section 3 of the KGW paper [1] provides a detailed theoretical framework. Notably, we are the first to identify this design as a potential vulnerability exploitable by attackers.

To further support our results, we extend the ablation study in Table 4 with mask text and report the z-score, which reflects the green token remnant.

Q2: Clarification regarding “targeted” attack

A2: We clarify that prior paraphrasing attacks, such as DIPPER [5] and GPT-Paraphraser, rely entirely on the LLM to rewrite text without control over which parts are modified, making them untargeted. In contrast, our method treats watermark removal as a targeted problem, selectively rewriting tokens likely to carry the watermark signal. Experiments show this targeted strategy is more effective.

Q3: The motivation gap.

A3: We clarify that the use of “high entropy” in prior work is heuristic. Specifically, none of the existing watermarking methods explicitly compute entropy, making it reasonable to explore metrics under the heuristic. Entropy and self-information are mathematically related, we find that self-information provides a more direct, token-level signal that better aligns with the actual greenlist selection process. Therefore, our motivation and implementation are consistent: we empirically identify a metric that captures the core vulnerability and reflects the intended intuition.

Q4: Why can one assume the model predicts the next words with equal probability in Appendix G?

A4: We clarify that this is a deliberate simplification of high entropy and our formal proof, Appendix H does not build on such an assumption. The uniform distribution serves as a theoretical upper bound for high entropy for a given probability space and helps illustrate high-entropy scenarios where probability mass is thinly spread. In such cases, even small probability changes can significantly affect self-information. Thus, this approximation highlights an extreme, simplified case to clarify the scaling behavior under high entropy for the reader not familiar with watermarking.

Q5: It's not reasonable that the attackers use much larger models.

A5: Our work is aligned with established settings in prior related studies [5–9]. We emphasize that watermark robustness mainly depends on the algorithm design and hyperparameters, not the choice of generation LLM. The mentioned work adaptive watermark in [4] uses the same evaluation. Notably, our method significantly reduces computational and time costs, as shown in Appendix B. We would appreciate references that follow the reviewer's proposed setting regarding LLM watermark robustness evaluation.

Q6: Evaluate the accuracy of the detected watermark tokens

A6: Please see A1 response.

Q7: Adaptive watermarks experiments

A7: We appreciate the reviewer’s suggestion. We show the proposed results below; the experiment follows our main experiment setting, and the threshold is 0.75. We report the attack success rate. As explicitly mentioned in the abstract of the work[4], this watermark is embedded in high-entropy text. We would appreciate it if the reviewer could clarify what “high-entropy loss” is and could provide a specific reference that did not adopt high-entropy embedding, and we will try our best to include the corresponding experiments during the discussion window.

Q9: Why can the watermark defenders see the attack text when they design the detector at Line 192?

A9: We thank the reviewer for pointing this out. This was a typo — the goal of the detector is to distinguish between watermarked and non-watermarked text. We will revise this part accordingly.

Q10: Typo, figure and symbol unclear

A10: We sincerely thank the reviewer for pointing it out. We will correct them in our revised manuscript.

Anonymous Reference Link: https://docs.google.com/document/d/1t1HxJ5KkCydhf_AwUdqOC8R1c0bwY024Ryr_iTZo-2A/edit?usp=sharing