Optimizing Adaptive Attacks against Content Watermarks for Language Models

Dear Reviewer,

We thank you for your positive assessment of our paper and for suggesting further improvements. Please find our responses to your questions below.

1.) KGW Settings and Advantage of the Adversary

What are the exact Dist-Shift/KGW settings used by this work (as it seems rather weak) [..]

We describe the watermark settings in Appendix A.3. In the paper, we use the following setting for KGW: a green-red list ratio $\gamma$ of 0.5 and a bias parameter $\beta$ of 4 and a text-dependent sliding window (LeftHash) of size 3. Upon request of Reviewer "dv9w30", we implemented our attacks in another benchmark (MarkLLM [C]) and evaluated more settings of KGW. Specifically, we evaluate LeftHash h=1, and we observe that non-adaptive paraphrasers still achieve high evasion rates. Please refer to our response to Reviewer "dv9w30" for these results.

We hypothesize that we measure different results than Jovanović et al. [G], as we use a more carefully chosen prompt. Jovanović et al. do not state which prompt they used in the paper, but their repository uses the following paraphrasing instruction:

Paraphrase the given text carefully sentence by sentence. Make sure to keep the same length (around 1000 tokens). It is imperative not to shorten or summarize, simply paraphrase every sentence.

Our paraphrasing instruction is listed in Appendix A.4.

How to interpret the advantage an adversary gets from this as, e.g., in Figure 8 the gap of the finetuned models to GPT3.5 is already small (and might be even smaller with stronger zero-shot paraphrasers).

Strong, closed-source paraphrasers such as GPT3.5 or GPT4o have several disadvantages when using them to evade watermark detection: (1) They are expensive, (2) they could be watermarked themselves, and (3) there could be guardrails such as safety alignments which prevent them from arbitrarily paraphrasing text. In contrast, our method allows working with relatively small open-weight models that adversaries can fully control. As shown in new experiments, which we provide upon request of Reviewer "dv9w30", the gap is much larger for more robust watermarks, such as Unigram[A], where GPT3.5 and GPT4o achieve only 79.7% and 62.5% evasion rates respectively. Our adaptively tuned 3B parameter models reach a much higher evasion rate of 94.6% at similar text quality ratings. These results demonstrate the effectiveness of our approach. We will include these results in the revised paper.

2.) Intuitive Explanations

Some intuitive explanation / qualitative explanation why one would expect this methodology should work for (e.g.) Dist-Shift.

The reviewer is correct in that our attack cannot predict the red/green list of the next token in the sequence since it is pseudorandom. However, there are at least three signals an attacker can learn: (1) As mentioned by the reviewer, repeated token sequences should be avoided as they likely contain the watermark. (2) The attacker can learn which words could be replaced at a low cost in text quality that maximizes evasion rate (e.g., uncommon punctuations, words or made-up names), and (3) our attacker can calibrate the minimum Token Edit distance that on average (over the randomness of the key generation procedure, see Eq. 2) suffices to evade watermark detection.

Since our attacks are optimizable and adaptive, we believe that including them when developing new defenses has the strong potential to enhance robustness. We agree with the reviewer that such adaptive defenses are worth investigating. We emphasize that the attacker is at a disadvantage because they need to optimize with uncertainty (see Section 4.1) since they do not have some information, such as the randomness used by the provider to generate their watermarking key. Our approach enables further research into the robustness of watermarks for LLMs. We will revise the paper to include this intuitive explanation.

I thank the authors for their updated results, which re-affirm my current choice. Overall, I do favor acceptance of the paper as it makes a relevant contribution. I have hence increased the certainty in my score.

Reasons why not lower

• The overall effect of the method is there. Training paraphrasers to specifically remove watermarks is a valuable baseline for stronger evaluations and works.
• The reviewer believes this is a sensible and useful addition to the current state of LM watermarking, showing the possibility of cost-efficient watermark removal via tuned paraphrasers.

Reasons why not higher

• While certainly improved, the current presentation (and the qualitative analysis) still falls short of what the reviewer considers to be a clear acceptance at the conference. In particular, this might make it harder for later work to build on it.
• The provided experiments with respect to "why it works" are a certain improvement over the previous state (esp. Figure 10a), but they could be improved (e.g., relating n-gram overlap more closely with WM strength for specific watermarks), the token-edit distance for GPT-3.5 vs. GPT-4o in Figure 10b (is it sensible metric when GPT-4o achieves the same evasion with much less token edits?). Also, here, it is not a priori clear how high the token edit distance can/should be to evade a watermark or maintain text quality. This reduces the insights that can be drawn from the work for future work/defenses.

Dear Reviewer,

We really appreciate your positive assessment of our work, and thank you for your feedback. We were surprised by the low confidence that you assigned to your review. We think you understood our work very well. Please find our response to your concerns below.

1.) Rigorous Examination

[..] without knowing which properties of watermarking schemes make them vulnerable to transfer attacks, it becomes difficult to design more robust systems. The lack of formal bounds or guarantees on attack effectiveness also leaves us wondering about the fundamental limits of watermark robustness.

Thank you for raising this interesting question. It is true that our paper does not introduce impossibility results or formal bounds on the robustness of LLM watermarking against paraphrasing attacks. Proving such bounds has been done using metrics such as the Token-Edit distance [A], which can be easily defined and are interpretable. The issue with such bounds is that they often lack relevance, as in practice, an attacker may not be limited by the Token Edit distance to launch their evasion attack. For instance, as shown in our paper, attackers can paraphrase text using open-weight LLMs to preserve "semantic similarity" instead of Token Edit distance. Metrics that measure semantic similarity typically rely on language models, for which it is extremely challenging to establish provable guarantees. We believe this question is outside the scope of our paper. Solving it would have wider implications also for the adversarial machine learning literature, which suffers from similar limitations.

Despite this, we believe our attacks mark a substantial improvement towards developing more robust watermarks by introducing a test to assess robustness. Our attacks empower model providers to develop more robust watermarks by including our attacks in their design process. For instance, one could have an adversarially trained watermark using our attacks. This line of optimizable defences has not been sufficiently explored, but it potentially offers a way to develop watermarks with enhanced robustness. We will include this discussion in our revised paper.

2.) Attack Transferability

Why do these attacks transfer so well to unseen watermarks? The paper shows impressive empirical results but doesn't help us understand the underlying reasons. Could there be some common vulnerability across current watermarking approaches that we're missing?

The transferability of our attacks stems from similarities in how the surveyed watermarking methods operate. All tested watermarks are similar in that (i) they operate on the token level and (ii) minimally modify the target model’s output distribution for the next-token prediction. Adaptive attacks can find optimal paraphrasing solutions to evade detection. The effectiveness of non-adaptive attacks can be explained because the optimal paraphrasing solutions are similar across the surveyed watermarking methods. Adaptivity allows finding one effective attack against one watermark, which has a high probability of also being effective against other watermarks due to their similarity.

Interestingly, we find a counterexample with limited transferability. We refer to our response to Reviewer "dv9w30", where we run additional experiments against other watermarking methods. There, we find that an adaptively optimized Llama-3.2-3B against Unigram [A] has only limited transferability against the SIR [B] watermark. This points to the fact that both watermarking methods use different strategies to hide the watermark, which we believe is an interesting property as it enhances robustness in the non-adaptive setting. We hope future watermarks use this insight and leverage this property to enhance robustness in the non-adaptive setting.

We will revise the paper to include this discussion.

3.) Defense Aspects

The authors' threat model focuses primarily on demonstrating attack capabilities but pays insufficient attention to the defensive aspects of the security game. [..]

You raise an excellent point about defensive measures, which we hope we answered in our response to question 1 above.

4.) Attack Generalization across Languages and Domains

How well do these attacks generalize across different languages and domains?

While this is an interesting question, we believe it is outside the scope of our work. However, our attacks enable further research to study this question in detail. We agree that there is too little research on the generalization of watermarking across languages and domains.

Summary

We hope we answered all questions and want to encourage the reviewer to increase their (confidence) scores. Thank you!

---

[A] Zhao, Xuandong, et al. "Provable robust watermarking for ai-generated text.", ICLR 2024.

[B] Liu, Aiwei et al. "A Semantic Invariant Robust Watermark for Large Language Models.", ICLR 2024

Dear Reviewer,

Thank you for taking the time to read and carefully review our paper. We appreciate the questions raised and ran additional experiments to address all of the reviewer’s requests. We implemented our paper in the framework of MarkLLM [C] and included experiments against the requested watermarks. Please find detailed answers below.

1.) Baseline Testing

The paper lacks comprehensive baseline testing, particularly against robust watermarking algorithms such as UnIgram (Zhao 2024 et al.) and SIR (Liu 2024 et al.).

Thank you for pointing us to these other watermarking methods. We have evaluated SynthID [D], SIR [B] and Unigram [A] and report results below. We find that our attacks remain effective against all watermarking methods. The notation is as follows: Qwen2.5-3B-Instruct is the unmodified, open-weight LLM. Brackets behind an LLM's name denote that the LLM was adaptively tuned against this watermark (e.g., Qwen2.5-3B-Instruct[Unigram] was adaptively tuned against the “Unigram” watermark). Below are the evasion rates (higher is better for the attacker). The best are in bold for each column.

Evasion Rates (Higher is Better):

We list the residual text quality ratings with LLM-as-a-Judge below (higher is better). As opposed to the paper, we use GPT-4o-mini as the judge. We only use this for evaluation, not for training the paraphrases. Again, the best value per column is highlighted bold.

Residual Text Quality (Higher is Better):

Main Observations: These results show that optimizable, adaptive attacks are incredibly powerful at evading watermark detection while preserving text quality. All paraphrases have a relatively high residual text quality. However, GPT3.5, GPT4o, and an unoptimized Qwen2.5-3B have relatively low evasion rates against Unigram of 79.7%, 62.5%, and 59.5%, respectively, whereas our adaptively tuned models achieve 94.6%. Similarly, for SIR, only our non-adaptive attacks achieve a 93.2% evasion rate with Qwen2.5-3B-Instruct[Unigram].

2.) Quality Testing

The quality test results appear questionable, particularly where Llama2-7B outperforms GPT-3.5, suggesting potential issues with data selection or evaluation methodology.

Interestingly, we observe that even the non-optimized baseline model Qwen2.5-3B-Instruct consistently outperforms GPT3.5 (but not GPT4o) in the residual text quality metric. This is likely because Qwen2.5-3B is a more recent model than GPT3.5, which ranks high on the LMSYS arena benchmark. However, we emphasize that all paraphrasers produce text with a high quality.

After tuning, our Qwen2.5-3B paraphraser sometimes exceeds the text quality of GPT4o. We believe this is likely because we use the LLM-as-a-judge metric during training to select preferable samples, making the model good at producing them. We are aware of this potential source of bias in reporting quality metrics. For this reason, in our paper, we also report other quality metrics such as perplexity, LLM-CoT, LLM-Compare and Mauve in Table 4 in the Appendix. Our model performs favourably across these metrics, which also confirms our manual inspection of the samples. Since there is no perfect text quality metric, we will revise the paper to emphasize and discuss this concern and also include quality metrics in the main part of the paper that measure the paraphrase quality of GPT3.5 and GPT4o.

Thank you for your prompt and thoughtful feedback. I've addressed each of your points below:

Although the author explained why Qwen2.5-3B-Instruct performs better than GPT3.5 and even GPT4o in many cases (in text quality evaluation), this remains counterintuitive. These anomalous data points still make me question the overall text quality evaluation metrics.

• Qwen2.5-3B-Instruct is a much more recent model compared to GPT3.5 and includes significant improvements in instruction following, generating long texts, understanding structured data, and handling diverse prompts. These features make it particularly effective for tasks like paraphrasing, which helps explain its stronger performance compared to GPT3.5.
• It's also important to note that Qwen2.5-3B-Instruct only outperforms GPT4o after being optimized within our framework, which specifically emphasizes producing high-quality paraphrases. We use the LLM-as-a-judge metric to select the most preferable outputs during training, leading to occasional performance beyond even GPT4o.
• We acknowledge that text quality metrics are imperfect, and most paraphrases (of all models) are of similar quality, with subtle differences. That's why we included other metrics (perplexity, LLM-CoT, LLM-Compare, and Mauve) in Table 4 of the Appendix, and these consistently support our findings and manual inspection of samples. We will revise the paper to better highlight and discuss these metrics in the main text to provide a more comprehensive view of the quality evaluation, including GPT3.5 and GPT4o metrics.

In terms of watermark removal effectiveness, if it's not specifically customized for a particular watermarking algorithm (like unigram), the performance might not surpass GPT3.5 or GPT4o for certain algorithms. If special customization is needed for each algorithm, it could reduce the model's generality.

• The watermark removal task is inherently challenging, which is why we propose an adaptive attack strategy that can target any known watermarking algorithm. With a GPU cost of under $10, our approach can train a paraphraser to effectively remove a specific watermark.
• During evaluation, we hypothesized that, since most watermarks in the literature share a common structure, an attack tailored to one watermark might generalize to others. Indeed, that was the case. While it is not as effective as one directly optimized for a specific watermark, this further supports our claim that watermarking methods should consider specific adaptive attackers rather than general attacks when they claim robustness of their algorithms.
• We will clarify this in the revised manuscript to emphasize the adaptability of our approach and the importance of considering adaptive attackers in watermarking evaluations.

My biggest concern is still about the underlying principles: why is this paraphraser better? It seems too much like a black box. While your explanations make sense, it would be beneficial to have quantitative evaluations. In the Dipper rewriting model, two key parameters are lexical diversity and order diversity. How does our model perform in these dimensions?

• The paraphraser is better because it: 1) uses knowledge of the underlying watermarking algorithm to generate targeted adversarial examples rather than relying on general parameters (like lexical or order diversity from Dipper); 2) is directly optimized to counteract the robustness of the watermarking algorithm, aiming to both maximize paraphrase quality and minimize watermark detectability, as directed by the preference dataset.
• We agree with the reviewer that the paraphraser operates as a black box, but this is inherent to its design. It is a neural network trained on pairs of sentences: one being a high-quality evaded paraphrase and the other not. The model learns by maximizing the likelihood of generating the evaded high-quality paraphrase given an input. Unlike explicitly programmed approaches targeting lexical or order diversity, our paraphraser implicitly learns to produce paraphrases similar to the high-quality examples that successfully evade detection in the training data.

Dear Reviewer,

We appreciate the opportunity to address all concerns and clarify misunderstandings. Please find detailed responses below.

1.) Unclear Optimization Formula.

“Why do you formulate robustness for attack optimization?”

The attacker's goal is to undermine robustness. We show that by formulating robustness as an objective, we can leverage optimization to find highly effective attacks. Our paper shows that these attacks are substantially more effective than any existing attack regarding evasion rates and residual text quality after paraphrasing. We highlight that all of our attacks (both adaptive and non-adaptive) are Pareto optimal. Hence, future watermarking methods must consider our attacks to test robustness.

We will further clarify this in the introduction.

“The robustness of what (the surrogate model or the target model)?”

Eq. 2 formalises the robustness of any watermarking method, not any specific model. However, optimizing Eq. 2 against the provider's watermark requires access to the provider's target model. We assume our attacker does not have any access to the target model (i.e., not even black-box access to query for watermarked text). To overcome this uncertainty, the attacker optimizes against an open-weight surrogate model (such as Llama2-7b) that they have access to and hopes that the adaptively fine-tuned paraphrasers remain effective when transferring to the provider’s target model (e.g., Llama3-70b). Our experiments show that it is indeed the case that our attacks transfer across all tested configurations. We refer the reviewer to Section 4 for more details about our optimization.

We will further clarify this in Section 4.

“Are the paraphraser and surrogate the same model?”

Yes. Section 5.2 - “Attacker’s Model” states that we only consider matching paraphraser and surrogate models. In principle, attackers can use non-matching pairs of paraphrase and surrogate models. For example, if the goal is to limit computational costs during inference, the attacker can choose a smaller paraphrasing model but train with a large surrogate model. We do not consider non-matching settings as our focus lies on demonstrating the effectiveness of adaptive, optimizable attacks and establishing a benchmark that must be passed for future LLM watermarking methods to claim robustness.

2.) The algorithm is confusing.

[..] However, in Algorithm 1, generating this dataset requires the paraphraser. How do you obtain the paraphraser?

Thank you for raising this question. Sections 4.2 and 5.1 describe the problem of “bootstrapping optimization” in detail. In short, our attacker bootstraps by using best-of-N rejection sampling with a non-optimized, open-weight paraphrasing model. We start with a non-optimized open-weight paraphraser. Since the attacker knows the watermarking method (adaptiveness assumption), they can generate watermarked text using the surrogate model, paraphrase it with the open-weight paraphraser and then verify if the paraphraser successfully removed the watermark (by random chance). The core idea is that instead of a single paraphrase, our attacker generates N=16 paraphrases and chooses the best-of-N paraphrase (according to text quality and evasion rate) to collect the preference dataset, which is then used to optimize the paraphraser adaptively. The trained paraphraser is then used in our evaluation. The open-weight model can be any off-the-shelf model, such as Llama2-7b.

3.) The method lacks important technical details.

The method only covers how the dataset is created, not how the paraphraser is optimized using DPO

As described in Section 5, once the dataset has been curated, we use the standard, unmodified DPO algorithm from the TRL library with default hyperparameters. We will revise the paper to include which hyperparameters we use. We will open-source our implementation to enhance reproducibility.

4.) Lack of important analysis of experiment results.

“In Figure 3 (Left), why does the adaptive setting have better performance than the non-adaptive setting”

(We assume there is a typo, and the reviewer asks why the non-adaptive setting sometimes has better performance than the adaptive setting.)

Figure 3 shows the evasion rate (left) and the residual text quality (right), where higher values are more desirable for the attacker. While non-adaptive attacks are indeed highly effective in evading detection, one must consider both (i) evasion rates and (ii) text quality. When doing so, we observe that adaptive attacks are Pareto optimal. The only exception is the “Inverse” Watermark, which is a special case since all attacks are already perfect at evading watermark detection for this p-value threshold.

We will revise the paper to include this analysis in the discussion.

Thank you for raising your score and your engagement during the rebuttal period. We really appreciate that. However, we think that our work's contribution goes beyond "creating a dataset and applying a standard algorithm for optimization".

Many new methods for LLM watermarking have been proposed and continue to appear, like Deepmind's SynthID, which is featured on Nature's cover. These methods can be surprisingly robust, even against attacks where the text is translated back and forth to a different language. However, as our paper shows, they offer no robustness against adaptive attacks that can be optimized with less than 10$ USD of computational resources. Furthermore, they also cannot withstand our non-adaptive attacks, which points to a flaw in how current watermarking methods are designed and tested. These vulnerabilities have not been shown before, and we offer a holistic framework to instantiate such attacks and test robustness. Currently, robustness is tested using non-optimal attacks, leading to a false sense of security. Robustness is a min-max objective; thus, it should always be tested against the most effective attacks, and other attacks become obsolete.

Notably, we highlight that our attacks assume an equally or less capable attacker than what was assumed in the literature. That is because they are tested in (i) a non-adaptive setting and (ii) without any access to the watermarked generator. Achieving robustness against our attacks should be possible if the watermarking method is well-designed and sufficiently randomized. Our paper is a litmus test for robustness, and we think it will support the development of more robust watermarking methods, as we describe in our revised paper.

We hope the reviewer reconsiders the significance of our contributions. Thank you for your time.