No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices

We thank the reviewer for their constructive comments. In the following, we respond to each question.

---

Q1: One of the biggest weaknesses of this paper is that the proposed attacks mainly explore the drawbacks of existing literature in [11,14,33], and some of the tradeoffs described in the paper are tied to these algorithms or specific formulations, which are not fundamental to the watermarking problem itself.

A1: Please see global response C2.

---

Q2: The robustness issue discussed in Section 4 is mainly due to the specific definition of robustness Definition 3. [...] I am curious to see if the proposed attacks still work for semantic-based watermarking.

A2: Thanks for this suggestion; please see global response C2.1.

---

Q3: As shown in "Adaptive Text Watermark for Large Language Models," [...] The authors are encouraged to provide more discussion regarding the applicability of their empirical findings to different types of watermarks.

A3: Thanks for the comments. Our findings in the tradeoffs of using multiple watermark keys do not apply to the semantics-based watermarks, as they rely on a separate semantics embedding model instead of determining the watermark logit bias using a watermark key. The general spoofing attack considered in [2] works for the PRF-based robust watermarks, and we may need substantial modifications to the general spoofing attacks to gain a better attack performance in semantics-based watermarks. For instance, instead of estimating the watermarked tokens’ distributions by providing uniformly distributed prompts, the attacker will need to carefully construct the prompts to guarantee that the outputs will be semantically close, such that they can gain some information on which portion of the tokens is more likely to appear in a specific semantic context by making a large number of queries. To defend against such potential attacks, one option could be for the model provider to introduce randomness into the semantics embedding model. For example, multiple random semantics embedding models could be used during inference, similar to the setting of using multiple watermark keys, to make it more resistant to watermark stealing. In this case, our findings will still apply, but this would need a more thorough and rigorous investigation.

As we mentioned in the global response C2, attacking semantics-based watermarks is not the focus of our paper, but we will clarify this and provide corresponding discussions in the revision.

[2] Liu et al. Adaptive Text Watermark for Large Language Models. ICML 24

---

Q4: I can imagine that similar issues discussed in the paper will also occur for watermarking images generated by the diffusion model. [...] This paper would greatly benefit from a thorough discussion of the differences and connections between these two problems by having a more thorough literature review.

A4: We agree that the attacks utilizing the detection API can be generalized to the image watermarks, as the attackers can adopt a similar oracle attack pipeline. The attackers will need to integrate domain-specific constraints to guarantee that the generated sentences or images are meaningful and high-quality. We will discuss potential opportunities and challenges in extending our attacks to image watermarks in the limitations/future work section of our revision (e.g., using the 1 extra camera-ready page if necessary). Thanks for this suggestion.

We appreciate the reviewer's constructive comments. Please refer to C1 in the global response for our clarifications on the positioning and contributions of our work. Our work studies the feasibility and ramifications of potential attacks, with the goal of better informing the public and the LLM watermarking community about key design tradeoffs. Below, we respond to each question.

---

Q1: Robust watermarks are not suitable for proof of content authenticity.

A1: We agree robust watermarks are more suited for AI content detection, while fragile (signature-based) watermarks are used for authentication.

Our contribution in this section is a rigorous study of the tradeoff between robustness and spoofing resistance, therefore showing that no single scheme is sufficient to protect against both types of attacks. Our experiments show the extent to which robust watermarks are vulnerable. We’ve recommended using fragile watermarks for spoofing defense in Sec 4.2 (L 215), but it is also known that fragile watermarks such as signatures are not robust to editing.

We emphasize that increasing a watermark’s robustness to editing diminishes its suitability for content authenticity. This tradeoff is valuable information for the LLM watermark community. Indeed, we note that we are not the first to explore spoofing and robust watermarks, as recent works [1,2] have proposed more complex spoofing attacks on a specific set of robust watermarks. In contrast to these works, we explore a more simple and general piggyback spoofing attack that allows us to explore the inherent trade-off between spoofing and robustness. The potential for spoofing has also been cited as a major barrier to industrial LLM watermarking deployment [3], making it important to study this tradeoff rigorously.

We agree that the current guideline can be revised to better convey our conclusions. We will follow your suggestion to revise our guideline to: Guideline #1: Robust watermarks are vulnerable to spoofing attacks and are not suitable as proof of content authenticity alone. To mitigate spoofing while preserving robustness, it may be necessary to combine additional measures such as signature-based fragile watermarks.

---

Q2: The findings of attacks exploiting public detection API are not surprising.

A2: In Sec 6 (L 288), we stated that “Although this (public detection API) makes it easier to detect watermarked text, it is commonly acknowledged that it will make the system vulnerable to attacks. Here, we study this statement more precisely by examining the specific risk trade-offs that exist, as well as introducing a novel defense that may make the public detection API more feasible in practice.”

We mentioned the use of public detection APIs is an ‘open question’ given recent activity in the community—for example, recent keynotes mention such settings [4], and commercial AI content detection services that return confidence scores to users [5]. However, we can remove this sentence to avoid misunderstandings. Despite the risks, public detection APIs have a lot of benefits, such as improving transparency in AI usage and supporting regulatory compliance. We believe providing a detection API isn't a simple yes or no question, and that there are different knobs one can tune when providing such an API. Our work both provides and explores such knobs, so that the risks/benefits of public detection APIs may be considered for practical deployment.

Oracle attacks have existed for decades, but rigorously exploring them in the context of LLM watermarks is necessary. We show that making detection scores differentially private can effectively mitigate the spoofing attack without compromising detection accuracy (see A7). Our findings can help to enable public detection APIs deployment, inspiring future LLM watermark designs.

---

Q3: I recommend using p-values as a detection metric.

A3: Thanks for this suggestion; please see global response C6.

---

Q4: Whether your attack still holds with a larger h?

A4: See global response C3.

---

Q5: Key management.

A5: See global response C4.

---

Q6: The assumption of watermarked LLM providing the top-5 tokens.

A6: See global response C5.

---

Q7: DP noise scale.

A7: For KGW and Unigram, we add noise to the z-scores. Sensitivity varies with sentence length (e.g., $\Delta=\frac{h+1}{\sqrt{\gamma(1-\gamma)l}}$ for replacement editing, where $l$ is the sentence length, $h,\gamma$ are watermark parameters). The actual noise scale is proportional to $\sigma\Delta$. For a 200-token sentence, $h=1,\gamma = 0.5,\sigma = 4$, the noise scale is 0.8. We’ve tested FPR with DP defense on OpenGen dataset, and FPRs are close to 0 (below 1e-3) using our recommended noise scale. We’ll explain more clearly in the text to avoid confusion.

---

Q8: Are other more exotic schemes (like semantic-based) also vulnerable?

A8: See global response C2.

---

Q9: One may easily think of easy counter-attacks: forbidding querying the LLM with the same prompt repeatedly (or the same prompt plus one extra token), forbidding querying the detector with too similar text, etc.

A9: In Guidelines #2 and #3, we recommended “defense-in-depth” techniques like anomaly detection, query rate limiting, and user verification. However, with just rate limiting, our attacks remain possible, as service providers can’t always ensure trusted users. Thus, it’s important to consider the tradeoffs when deploying LLM watermarking systems.

---

[1] Jovanović et al. Watermark Stealing in Large Language Models. ICML 24

[2] Sadasivan et al. Can AI-generated text be reliably detected? arXiv 23

[3] Somesh Jha. Keynote at SaTML 2024-Watermarking (The State of the Union). 2024

[4] Scott Aaronson. Watermarking of large language models. 2023

[5] AI Purity; GPTZero; Winston AI

We thank the reviewer for their constructive comments. In the following, we respond to each question.

---

Q1: In Sec 3.1, what are the differences between "piggyback" and "general" spoofing attacks? Specifically, what does "piggyback" refer to?

A1: Piggybacking (Sec 4, L 142) is a classic attack in computer networks, where the attacker tags along with another person who is authorized to gain entry into a restricted area. The general spoofing attacks for LLM watermarks [1, 2] usually require the attacker to first estimate the watermark pattern by making a large number of queries (observe millions of watermarked tokens) to the watermarked LLM, and then they can create malicious content with a target watermark embedded but in fact it is not generated by the watermarked LLM to ruin the reputation of LLMs.

Our piggyback spoofing attack does not require estimating the watermark pattern, instead, we launch the attack based on the content generated by the watermarked LLM, which is similar to piggybacking in computer networks where the attacker relies on well-established authorization. The attacker’s goal of piggyback spoofing is the same as the general spoofing attack as they both aim to create malicious/inaccurate content with a target watermark embedded. However, a benefit of our attack is that it has much weaker assumptions on the attacker’s ability. The attacker can simply exploit the robustness property of the watermarks and maliciously edit the watermarked content without altering the watermark detection result to generate malicious but watermarked content to ruin the LLM’s reputation.

[1] Jovanović et al. Watermark Stealing in Large Language Models. ICML 24

[2] Sadasivan et al. Can AI-generated text be reliably detected? arXiv 23

---

Q2: In Sec 3.1, regarding attacks on detection APIs, the reviewer is confused by the statement "the attacker can auto-regressively synthesize (toxic) sentences." What does this mean?

A2: In the attacks exploiting the detection APIs, the attacker will generate sentences auto-regressively, similar to how LLMs generate sentences. That is, the attacker will select each token based on the prior tokens and the detection results. Please also refer to Alg.1 and Alg.2 in the Appendix J of our paper. We will clearly explain this in the revision.

---

Q3: Regarding attacks discussed in Sec 5, the attackers can discover watermarking rules by observing a large amount of watermarked text, thus enabling attacks. The original KGW paper in Sec 5 mentions using a large context width h to defend against such attacks. However, this paper lacks explanation and discussion regarding the parameter h.

A3: Please see global response C4.

---

Q4: In the detection API attack, the description of the adversary's capabilities is unclear. In Sec 6.1, the authors assume the adversary can access the target watermarked LLM's API and query watermark detection results. This implies the adversary can generate watermarked text and obtain detection results for any given text. But why can the adversary generate a list of possible replacements for x_i? Does this mean the adversary can access the perturbed probability distribution and logits of tokens? If so, this seems to exceed the stated capabilities of the adversary.

A4: Please see global response C5.

---

Q5: There are some minor issues, such as the undefined notation $V^{\ast}$ in the definition in Sec 3.

A5: The $V^{\ast}$ refers to a sequence of tokens, where each token belongs to the vocabulary set $V$. We will clearly explain this in the revision.

We thank the reviewer for their constructive comments. In the following, we respond to each question.

---

Q1: The citation in Line 248 and Figure 2 is not correct. The authors are supposed to cite [1].

A1: Thanks for pointing this out. We will fix this typo in the revision to avoid confusions.

---

Q2: Lack of discussions in the Publicly-Detectable Watermarking [2], which compromises robustness to defend against spoofing attacks.

A2: There exist some recent works [1,2] that study mitigating the spoofing attack vulnerabilities in robust watermarks. The high-level idea is to embed a cryptographic signature into the subsequent tokens, and the signatures are computed using the first $m$ high-entropy tokens and the secret key. They further incorporate error correction code to make the design robust.

As also mentioned by the reviewer, such designs are not as robust as the watermarks we study as they prioritize the resistance against spoofing instead of strong robustness. For instance, by simply modifying the first $m$ tokens, the signature check no longer passes. The designs of these works are consistent with our findings in the piggyback spoofing attacks: to defend against spoofing attacks, the design needs to incorporate less robust (or even non-robust) signature-based watermarks. We will include these related works in the revision to provide a more comprehensive study.

[1] Zhou et al. Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signature. arXiv 24

[2] Fairoze, et al. Publicly detectable watermarking for language models. arXiv 23

---

Q3: How generalizable are the findings? Would the vulnerabilities and trade-offs identified apply to all types of LLMs, or are they specific to certain architectures or applications?

A3: Please see global response C2.

---

Q4: What are the limitations of the attack methods presented in this study? Are there scenarios where these attacks might not be effective?

A4: As we have discussed in C2, the semantics-based watermarks rely on a high-quality semantics embedding model instead of using secret keys to embed the watermark. As such designs fundamentally differ from the watermarks we study, our findings of using multiple watermark keys are not applicable here. We will include a limitation and future work section in the revision to discuss this issue and present a more comprehensive study.

---

Q5: Are there any practical considerations or potential drawbacks to implementing DP defense mechanisms in real-world systems?

A5: DP adds noise to the watermark detection results. The service provider needs to determine the optimal noise scale, as larger noise will make the detection inaccurate, and less noise will be ineffective to defend against attackers. According to our empirical findings, we can find a sweet point to achieve both high detection accuracy (low FPR) and low attack success rate. Overall, we believe that our DP defense can potentially make the detection API publicly available while protecting the secret watermark pattern information without sacrificing detection accuracy.