Optimizing Adaptive Attacks against Watermarks for Language Models

Thank you for your time, valuable suggestions, and excitement about our paper. Please find our responses below.

[..] explanations are largely empirical in nature, leaving some uncertainty about where to go from here for future, more robust, watermarks.

Thank you for raising this point, which was also raised by other reviewers. We discuss future work in Section 6, where we propose 'adversarial training', which would first require designing optimizable defenses against our optimizable attacks. This is not a trivial task, hence we believe this is future work. We kindly refer to our response to reviewer 'z2QJ' for more information on 'where to go from here'.

The phrasing "existing work only exists against non-adaptive attackers" (e.g., in the abstract) does not seem to do justice to existing work that (while not in a no-box setting) does make some assumptions about the watermark under attack and adapts/learns based on this [1].

In the abstract, we claim that "[..] robustness is tested only against non-adaptive attackers" which we believe is valid for all the surveyed watermarking methods. We do not claim that we are the first adaptive attack. While Jovanović et al. [1] do not explicitly use the term 'adaptive' in their paper, we agree with the reviewer that their spoofing attack works only if they know the watermarking method (i.e., they adaptively handcrafted their attack). We will revise the paper to clarify this.

Zero-shot baselines already seem very strong [..] However, it raises the question of how far one could push a zero-shot LLM, especially with some additional filters/modifications.

Thank you for raising this point. We also think this would be an interesting question. One can use our attack method to adaptively optimize for such a prompt (i.e., treat the prompt as the optimizable parameters), which we did not do in our paper. We already used handcrafted prompts that we manually optimized, which likely explains the strong zero-shot baselines.

[..] Related to this, a full overview of the score distributions in Table 3 could help people contextualize the score distributions.

This is a good idea, and we thank the reviewer for bringing it up. We will revise the paper to include a histogram of GPT-Judge scores in the Appendix.

To the reviewer's understanding, Pareto optimal is really a theoretical statement (which one would have to make overall existing adversaries) - what can be surely stated is that their method is on the current Pareto-frontier (but one should probably refrain from calling it optimal).

We are unsure if we correctly understood what the reviewer refers to when they say that 'Pareto optimal is really a theoretical statement' and would kindly ask for further clarification in case our answer missed the point. Given all other surveyed attacks, our attacks are Pareto-optimal. However, as the reviewer points out, there could be even better attacks that we did not evaluate.

The notion of including the watermark directly in the model parameters (L55R) is somewhat of a strange choice as there exists an entire field of Open-Source watermarking that specializes on this [2] (containing the watermark directly in the model). As this is not the focus of this work, it may help the presentation if it is slightly revised.

We agree with the reviewer that this might be confusing, as the watermarks we consider do not necessarily modify the model's parameters. We will clarify that $\operatorname{Embed}$ can change the entire generation procedure (such as warping the logits of the LLM during generation).

The provided examples seem to show that paraphrases by the method are shorter. Can the authors provide additional statistics on this and potential reasons (e.g., from the preference dataset)?

Yes, this is likely an effect of the optimization, which does not explicitly penalize shorter responses. This property of the output could be controlled by modifying the objective function. We evaluated our data and observe that a non-optimized baseline Qwen2.5-3b has a mean output token length after paraphrasing of $\mu=230$ tokens ($\sigma=25.25$), whereas our optimized version has an output length of $\hat{\mu}=214$ tokens ($\hat{\sigma}=33.16$). We will include these results in the revised paper.

On which dataset was the evaluation conducted (number and source of samples, distribution, etc.)?

We follow Piet et al. [A] 's evaluation and kindly refer the reviewer to their paper for more details. The number of samples is 296 across different tasks (storylines, reports, fake news prompts, etc.). We will also add a short section describing the dataset in the Appendix to ensure our paper remains self-contained.

---

[A] Piet, Julien, et al. "Mark my words: Analyzing and evaluating language model watermarks." arXiv preprint arXiv:2312.00273 (2023).

Thank you for the detailed response and generally positive outlook on our paper. We appreciate your time and effort. Please find our responses below.

[..] key message of the paper is that training against any watermark can improve removal success against the target watermark.

We agree with the reviewer that all watermarks are vulnerable to our optimizable attacks in the non-adaptive setting. Our non-adaptive attacks differ from existing ones in that we adaptively optimize them against other (similar) watermarks and then show in our experiments their effectiveness transfers to unseen watermarks (likely due to their similarity; see Section 6). Hence, even though we evaluate these attacks in a non-adaptive setting, they were still adaptively tuned against other known watermarks.

We believe these are core findings of our paper and will follow the reviewer's suggestion to highlight that 'training against any watermark can improve removal success against the target watermark'. However, we would also like to emphasize that in the future, more robust watermarks may be developed that resist our attacks in the non-adaptive setting but not in the adaptive setting.

[..] in which scenario does an attacker aim to remove a watermark from an LLM deployment but does not have prior blackbox access to that deployment?

We agree with the reviewer that all deployed watermarking methods allow users to generate (some) watermarked samples. Realistic attackers typically have at least black-box access to the LLM and would not be confined to the no-box setting.

Our motivation is to evaluate the robustness of watermarking against constrained attackers that (i) have limited resources and (ii) lack any information about the watermarking key and samples. If successful attacks exist in this pessimistic no-box setting, the provider cannot hope to have a robust watermark against more capable attackers (e.g., with black-box access). We show that (i) such attacks exist, (ii) they are cheap, and (iii) they do not require access to watermarked samples. We believe the development of defenses should focus on the no-box setting first. We will discuss this in the revised paper.

[..] If these [the datasets] are the same/very similar this could imply leakage and perhaps the paraphrasers would not be as effective outside of this domain?

We used different query datasets to train and test the paraphraser and will clarify this in the paper. However, we highlight that in a practical setting, it is not unreasonable to assume that the attacker uses the same dataset to train and test the paraphraser. Since our attacks are cheap ($\leq10$$ USD), the attacker can train such a specialized paraphraser.

The (ϵ,δ) robustness definition is interesting, but I have not seen it used later, why is htis needed?

We re-use $\delta$ in Alg. 2 and Section 4.2, and $\epsilon$ corresponds to the evasion rate, which we show in Figures 2, 3, 4. From these figures, one can deduce the $(\epsilon, \delta)$-robustness of the surveyed watermarks, subject to the metrics we study. We will clarify this in the paper.

What are the "messages" m here given that all methods are zero-bit to the best of my understanding?

The reviewer is correct that all watermarks we focus on in the papers are zero-bit. Our definition in Section 2 is more general and allows for multi-bit watermarks as defined by Zhao et al. [A].

Even weak off-the-shelf models generally get high scores (>80%) which does not match my intuition from prior work [..]

To enhance evasion rates, we have manually optimized the prompt used for the baseline models (page 18 in the Appendix).

When mining negative samples how many of those fail the quality criterion and how many simply don't remove the watermark (and how many fail both criteria)? [..] do you think balancing it on purpose could improve the results further?

That is an interesting idea. To answer your question, we investigated the samples collected for training the paraphrasers on Unigram for Llama-3.2-3B. High quality means LLM-Judge had a score of $\geq 0.8$.

We did not investigate methods to 'balance' negative samples since our current approach is already highly effective. This would be an interesting idea for further optimization.

We will follow the reviewer's suggestions for the remaining points: (i) The citations in the introduction, (ii) add GPT4o to Figure 4 (it performs similarly to GPT3.5 in terms of evasion rate), and (iii) move the related work section to an earlier part of the paper. We believe the colors for Figure 5 are correct.

---

[A] Zhao, Xuandong, et al. "SoK: Watermarking for AI-Generated Content." arXiv preprint arXiv:2411.18479 (2024).

Thank you for your time, consideration, and many valuable suggestions. Please find our responses below.

Yes, the methods and evaluation are reasonable. However, the main body primarily relies on LLM-Judge for quality evaluation, while the perplexity performance, as shown in Table 3, is not particularly strong.

Perplexity is indeed worse than the baseline model. However, as noted in the paper, perplexity is not always a good quality indicator since it penalizes diverse outputs. For this reason, we include many quality metrics (LLM-Judge, LLM-CoT, LLM-Compare, Mauve, and Perplexity) as described in Appendix A.1. These metrics show that our paraphrased text has a high quality.

[..] reference [1] proves that achieving strong watermarking is fundamentally impossible [..] I disagree with the authors' claim that investigating strong attacks is necessary, as defending them is nearly impossible. Instead, to provide meaningful guarantees, it is more practical to formulate the problem by restricting the attacker's capabilities (e.g., a college student trying to submit an AI-generated essay) rather than assuming an adversary familiar with LLM watermarking.

Thank you for bringing up the impossibility results for 'strong' watermarking. We agree with the reviewer, and our primary setting is studying a resource-constrained attacker - but we are interested in the strongest possible such attacker. Staying with their analogy of viewing watermark evasion as graphs, the attacks in [1] use random walks, whereas our attacks are optimized to find the shortest paths. This makes our attacks more computationally efficient and allows the use of smaller paraphrasers than [1].

We already assume a particularly constrained attacker who (i) has no access to the provider's watermarked LLM and (ii) has limited computational resources. Strong watermarking is fundamentally impossible (if quality and perturbation oracles exist), which makes it interesting to study whether watermarking exists that is robust in practice against constrained attackers (i.e., evasion would incur prohibitively high costs). Our work studies this setting and proposes effective attacks that incur little computational costs to the attacker of less than $10$$ USD.

Conceptually, can the proposed attack be viewed as a specific instance of the one studied in [1]? How should I position this work given their impossibility result?

As described above, the attack in [1] uses a random walk and proves that under certain assumptions, this theoretically guarantees removal, irrespective of the computational costs incurred for the attacker. Since we optimize our paraphraser, we have a higher probability of evading detection with fewer steps (1 step in our work). We could additionally control other properties of the output (e.g., semantic similarity to the original watermarked text). We will add this discussion to the revised paper.

Reference [3] introduces a more robust semantic-based watermarking method compared to SIR. Is the proposed attack still effective against this approach? A new plot similar to Figure 7 should be included to illustrate the results.

Thank you for bringing [3] to our attention. We evaluated it against our pre-trained models and obtained results similar to other watermarks we surveyed. We use the provider model Llama-2-13b. Note that $\epsilon$ refers to the evasion rate, as described in our paper. Below, we use watermarking strengths $(\delta_0=0.33, \delta_1=0.33)$ (not to confuse with our notation of $\delta$ for text quality)

Below, we show results for $(\delta_0=0.67, \delta_1=0.67)$.

The results show that our attacks evade detection of this watermark. We will include these results in the revised paper.

Essential References Not Discussed

Thank you for bringing up these references. We will discuss them in the revised paper.

Thank you for your time and valuable suggestions. Please find our answers below.

The paper could benefit from a more in-depth discussion of potential defenses against adaptive attacks, such as adversarial training.

We appreciate this suggestion. While we focused on the robustness of current watermarking methods to adaptive attacks, we agree that discussing potential defenses would strengthen the paper. In Section 6, we briefly mention the idea of adversarial training to set up a two-player game where both attacker and defenders optimize their strategies, and the goal is to find an Equilibrium. We do not present results on this in our paper. The first challenge would be to develop a watermarking method that can be optimized, following the optimizable attacks we present in this work, which we believe requires substantial effort. This is an interesting direction for future research.

The reliance on LLM-as-a-judge for text quality assessment is a known limitation. Although this approach is convenient in practice, LLM-as-a-judge may be biased and not necessarily consistent with human judgments.

We thank the reviewer for mentioning this limitation, which we also highlight in our paper. Section 6 acknowledges that LLM-as-a-Judge is an imperfect and noisy metric that may not perfectly align with human judgment. To address this concern, we used multiple evaluation metrics (LLM-Judge, LLM-CoT, LLM-Compare, Mauve, and Perplexity) as described in Appendix A.1, and we have included results with different judge models (both Llama3-8B-Instruct and GPT4o-mini in Appendix A.3). We agree that more work is needed to study the metric's alignment with human judgment as stated in our limitations. However, we believe this work may be outside of the scope of our paper. We will strengthen the discussion by suggesting how future work could incorporate human evaluations to validate our findings.

It would be interesting to explore the transferability of adaptive attacks across different types of LLMs (e.g., encoder-decoder models).

We agree that it would be interesting to examine different types of LLMs. Our current work focuses on autoregressive decoder-only LLMs. This is the primary setting studied by related work and the most commonly deployed for text generation tasks (e.g., SynthID [A]). Our revised paper will add this as a promising direction for future work. Specifically, we will discuss how the principles of our adaptive attacks might transfer to encoder-decoder architectures like T5 or BART and whether the different generation process might affect the effectiveness of watermarking and our evasion techniques.

The authors could consider investigating the impact of different training objectives for the paraphraser model.

We appreciate this suggestion. In our current work, we focused on Direct Preference Optimization (DPO) as our training method for the paraphraser. We would kindly ask the reviewer for more information about which specific training objectives they are considering, as this would help us better address their suggestion. In principle, any objective that can optimize for the dual goals of watermark evasion and text quality preservation could be used.

Could you elaborate on the ethical considerations related to the release of your source code and adaptively tuned paraphrasers, given the potential for misuse in evading existing watermarks?

We carefully considered the ethical implications of releasing our code and models, which we address in our Impact Statement in Section 9. We believe the responsible disclosure of vulnerabilities is essential for improving security systems. Current watermarking deployments are still experimental, and our work highlights vulnerabilities that should be addressed before widespread adoption. By releasing our methods, we enable researchers to test against stronger attacks and build more robust watermarks.

Have you explored the possibility of combining your adaptive attacks with other evasion techniques, such as those targeting safety alignment or content filtering, to assess their combined effectiveness? This would give a more comprehensive view of real-world attack scenarios.

This is a good point that we also mention as a limitation in our discussion section (Section 6). We focus on robustness of watermarks and do not analyze the interplay of different defenses at the same time. We agree that examining combined attacks would provide a more comprehensive view of real-world scenarios where multiple defenses might be deployed simultaneously. This is a promising direction for future work and we will discuss this in the revised paper.

---

[A] Dathathri, Sumanth, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani et al. "Scalable watermarking for identifying large language model outputs." Nature 634, no. 8035 (2024): 818-823.

Thank you for your time and effort in providing constructive feedback that will improve our paper. Please find our responses below.

[..] they use standard RL techniques such as DPO to optimize their objective function from a preference dataset.

Yes, we use standard RL techniques for optimization. However, we view our contributions as follows: (i) Formulating watermark robustness as an objective function, (ii) proposing a framework to optimize this objective adaptively, and (iii) showing the efficacy and efficiency of adaptively optimized paraphrasers. A core finding is that even relatively constrained attackers can evade detection using limited resources.

How about having a regularization parameter to trade off the quality of text quality and scrubbing?

This is an excellent suggestion. We will introduce a regularization parameter $\beta$ in our objective function (Eq. 2) as follows:

$\underset{\theta_P}{\max} \mathbb{E} \left[\operatorname{Verify}(P_{\theta_P}(x), \tau', m') + \beta Q(P_{\theta_P}(x), x)\right]$

Since we use LoRA adapters, another option that does not require re-training is to scale the weight $\alpha$ of the adapter. When doing so, we measure the following results against EXP using Qwen2.5-3b adaptively tuned against EXP:

We will include these results in our revised paper.

Can this be utilized for spoofing attacks and if so how could that be implemented?

This is an interesting question and an active area of research. We can share our thoughts on a simple way of how this could be done. First, we would like to point out the difference in settings and objectives: (1) We operate under the no-box setting, but spoofing requires at least some watermarked samples (e.g., black-box access). (2) Spoofing attacks try to maximize the spoofing success rate using a limited number of watermarked samples.

One spoofing attack was proposed by Jovanović et al [1]. In a nutshell, (i) they collect $N$ watermarked samples, (ii) apply a scrubber to create corresponding non-watermarked samples, and then (iii) train a mapper from non-watermarked to watermarked samples. Since our scrubber has substantially higher evasion rates than all existing ones, it could be used as a scrubber in Jovanović's attack to make training the mapper more sample efficient, thus improving the attack. As mentioned before, there are other ways in which our attack could be used, which is an interesting question for follow-up research.

Also, given that we have this offline attack, is there a way to set this up as game to mitigate for these attacks? It seems relatively easy to scrub. Are there any fundamental results in this space for your setting such as this paper (Watermarks in the Sand).

Again, thank you for this interesting question related to follow-up research. In Section 6, we briefly mention the idea of adversarial training to set up a two-player game where both attackers and defenders optimize their strategies, and the goal is to find Equilibrium. We do not present results on this in our paper. The first challenge would be to develop a watermarking method that can be optimized, following the optimizable attacks we present in this work.

As for fundamental results, there are impossibility results for robustness [2] and provable robustness for watermarks under certain metrics (e.g., token-edit) [3], but we are not aware of fundamental results when treating watermarking as a two-player game.

We hope our responses addressed all your questions and are happy to answer any further questions the reviewer may have.

---

[1] Jovanović, Nikola, Robin Staab, and Martin Vechev. "Watermark stealing in large language models." ICML (2024).

[2] Zhang, Hanlin, Benjamin L. Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. "Watermarks in the sand: Impossibility of strong watermarking for generative models." arXiv preprint arXiv:2311.04378 (2023).

[3] Zhao, Xuandong, et al. "Provable robust watermarking for ai-generated text." arXiv preprint arXiv:2306.17439 (2023).