On the Learnability of Watermarks for Language Models

We thank you for the useful and insightful feedback and for taking the time to review our paper. We address questions and concerns below.

Contribution

Please see our general comment clarifying our contribution. https://openreview.net/forum?id=9k0krNzvlV&noteId=UyaqBwxSNh Our main contribution is our empirical investigation and findings on watermark learnability. It is true that knowledge distillation is a well-established concept, but our method is not our primary contribution. Our findings and evaluations are novel and new, as the learnability of decoding-based text watermarking was previously unexplored, and it was unknown whether existing decoding-based watermarks are learnable or not.

Relationship to model watermarking for protection against model extraction attacks

We have updated the related work section to include a discussion of the relationship of these provided works with ours.

These works concern model watermarking for protection against model extraction attacks. Model watermarking for model protection is crucially different from text watermarking, which we study. Model watermarking is meant to protect language model APIs from model stealing/extraction attacks, where an adversary trains a model on outputs from the victim model API. The model watermark enables detecting if a suspect model was extracted from the victim model. Detection is done by querying the suspect model multiple times on specially chosen inputs. The model watermark is specifically designed to be learnable, so that the adversary model learns the watermark during the model extraction attack. On the other hand, text watermarking enables detecting if a given text was model-generated. Detection does not need access to any language model, only the tokenizer, and detection is meant to be performed on any generated text, regardless of the input prompt. Existing decoding-based text watermarking methods are specifically designed to be subtle and imperceptible in order to have minimal impact on text quality, and it was previously unknown whether they are learnable or not.

In short, the object being detected is different (model itself versus generated text), and model watermarking for model protection is specifically designed to be learnable, whereas the learnability of existing text watermarks was previously unexplored and unknown. Because of these crucial differences, these prior works into model watermarking for protection against model extraction attacks do not diminish the value and significance of our findings.

AUROC and FPR

As requested, we have included the AUROC metric for classifying watermarked and non-watermarked text for a more comprehensive understanding of watermark detection, as shown below for the main logits-based watermark distillation (WD-L) and sampling-based watermark distillation (WD-S) experiments. Dec indicates decoding-based watermarking. Additional full tables can be found in the paper.

Because watermark detection returns p-values, the false positive rate is equal to the significance level that is chosen as the threshold for classifying a text as model-generated or not. So we do not report FPR because it is set by the detection user depending on the FPR they are willing to tolerate.

Dear Reviewer GhKC,

We appreciate all of the valuable time and effort you have spent reviewing our paper. As today is the last day of the discussion period, we gently request that you review our reply and consider updating your evaluation accordingly. We believe that we have addressed all questions and concerns raised, but please feel free to ask any clarifying questions you might have before the end of the discussion period.

Best,

Authors

We thank you for the useful and insightful feedback and for taking the time to review our paper. We are glad that you found our findings valuable and significant, and our experiments thorough and rigorous. We address questions and concerns below.

Llama-2 sampling-based watermark distillation evaluation

As requested, we have included an evaluation of sampling-based watermark distillation using Llama 2 7B as both the teacher and student model. We use the same evaluation procedure as before. Results are shown below, and also have been added to the paper. The trends are broadly similar to the original logits-based and sampling-based distillation experiments.

However, we note that logits-based and sampling-based distillation have key differences, which means that they should not be compared solely based on efficacy to determine which method is better. Logits-based distillation requires white-box access to the teacher model and that the teacher and student models share the same tokenizer and vocabulary. It is also convenient if the teacher and student models share the same model architecture. On the other hand, sampling-based distillation only requires black-box access to samples from the teacher model, and the teacher and student models can have different tokenizers and vocabularies. Also, sampling-based distillation requires autoregressive sampling from the teacher model, which is slow, whereas logits-based distillation does not require sampling and is faster overall. So, logits-based and sampling-based distillation are each suitable and applicable for different settings, so neither is strictly better than the other across all scenarios.

Increasing dataset diversity

We have included additional evaluations of logits-based and sampling-based watermark distillation (including both Llama 2 7B and Pythia 1.4B as student model) on two additional datasets, Wikipedia articles and arXiv papers. We observe similar trends to the original main evaluation on the C4 RealNewsLike dataset. The full results are in Appendix F.

Seq-rep-3 on sampling-based distillation

Apologies for the confusion, we originally omitted seq-rep-3 on sampling-based distillation in order to respect margin constraints, but we have now re-included it in Appendix B, as shown below.

Expanding explanations around distillation objectives

We have expanded the explanations around the watermark distillation objectives to improve readability.

Related work relationship to model watermarking

We have expanded the related work section to include a discussion of model watermarking methods for protection against model extraction attacks [1, 2, 3, 4]. Note that text watermarking, which we study, is crucially different from model watermarking/protection in both setting and method. Model watermarking is meant to protect language model APIs from model stealing/extraction attacks, where an adversary trains a model on outputs from the victim model API. The model watermark enables detecting if a suspect model was extracted from the victim model. Detection is done by querying the suspect model multiple times on specially chosen inputs. The model watermark is specifically designed to be learnable, so that the adversary model learns the watermark during the model extraction attack. On the other hand, text watermarking enables detecting if a given text was model-generated. Detection does not need access to any language model, only the tokenizer, and detection is meant to be performed on any generated text, regardless of the input prompt. Existing decoding-based text watermarking methods are specifically designed to be subtle and imperceptible in order to have minimal impact on text quality, and it was previously unknown whether they are learnable or not.

1. He et al., "Protecting Intellectual Property of Language Generation APIs with Lexical Watermark," AAAI 2022.
2. Zhao et al., "Distillation-Resistant Watermarking for Model Protection in NLP," EMNLP 2022.
3. He et al., "CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks," NeurIPS 2022.
4. Zhao et al., "Protecting Language Generation Models via Invisible Watermarking," ICML 2023.

Robustness to fine-tuning

We agree that improving robustness of weights-based watermarking to fine-tuning is an important challenge towards developing watermarking for open models. However, we believe that methods to improve robustness to fine-tuning are not as directly related to our main focus of investigating watermark learnability and would best be explored in future work.

More in-depth spoofing experiments

We agree that more in-depth studies and evaluations of watermark spoofing attacks across different settings would be valuable for more comprehensively evaluating the practical implications and risks of spoofing attacks. However, we believe that our findings are still valuable and lay a foundation for future work in this direction. We demonstrate that sampling-based distillation is able to learn to generate watermarked text from black-box watermarked samples, which is the setting for spoofing attacks. We also include experiments on sample efficiency (Appendix D) and different generation datasets (Appendix F). These findings suggest the possibility of spoofing attacks, and we show a simple proof-of-concept spoofing attack to substantiate their possibility. We hope that our work will inform future research and responsible usage of watermark detection.

Distinguishing between distilled and original watermarked distributions

Your assumption that the spoofed watermark tends to learn more common watermarked sequences first seems reasonable. If a defender tries to evade the spoofing attack by ignoring the most common watermarked sequences or n-grams, they could reduce the proportion of texts generated by the adversary which are detected as watermarked, but this would also reduce the proportion of texts genuinely generated by original model which are detected as watermarked. This is because the most common watermarked sequences/n-grams contribute more towards the watermark detection score. Rare watermarked sequences are less likely to appear, so it is more likely that a genuinely generated text contains none or few rare sequences, thus resulting in a false negative. So this attempt to evade the spoofing attack will reduce the watermark’s efficacy to detect model-generated texts, which is its primary purpose.

Spoofing attacks vs defenses

More generally, we believe that defenses against spoofing attacks vs. spoofing attacks to overcome these defenses is a security cat and mouse game, with no conclusive winner or answer at this time. Some new defense could later be defeated by a new attack, which is then defeated by a new defense, and so on. Ultimately, potential vulnerability to spoofing attacks is one reason among others suggesting that text watermarking should not be used to conclusively attribute provenance or blame to a specific model. Text watermarking is best suited for its original use case of detecting model-generated text, e.g., for preventing students from cheating on essays or preventing LLM-generated misinformation campaigns.

We thank you for the useful feedback and for taking the time to review our paper. We are glad that you liked the paper and think that the problem is timely and important. We address questions and concerns below.

Contribution to open model watermarking

Please see our general comment clarifying our contribution. https://openreview.net/forum?id=9k0krNzvlV&noteId=UyaqBwxSNh One important technical challenge towards open model watermarking is robustness to decoding procedure, since users of open models can run inference themselves and choose the decoding procedure, so it is easy to not use a decoding-based watermark algorithm. Our findings on watermark learnability and weights-based watermarking address this key challenge, taking a necessary first step towards developing open model watermarking. However, as you correctly point out, robustness to fine-tuning is another important technical challenge which we leave for future work. We believe that our contribution is still valuable because it investigates a necessary first step towards open model watermarking that was previously unexplored.

Regarding point (ii), we do not design watermarks to be spoofable. We investigate the learnability of existing decoding-based watermarks, rather than design new watermark algorithms, and high learnability suggests the possibility of being vulnerable to spoofing attacks. Spoofability suggests that watermarks should not be used to assign provenance or blame to a specific model, but it does not hinder the primary case of detecting model-generated text, e.g., preventing students from cheating on essays or preventing LLM-generated misinformation campaigns. Watermarking is still useful and valuable even if it is spoofable.

Sample efficiency

To investigate sample efficiency, we have included additional experiments measuring watermark detection p-values as we vary the number of training samples for sampling-based watermark distillation. As in the main experiments, each sample is 256 tokens long, and we use Llama 2 7B as the teacher model and Pythia 1.4B as the student model. Note that the training batch size is held the same, so fewer training samples means fewer gradient steps. P-value results are shown below, and we have also included a plot of p-values against the number of training samples in the paper (Appendix D).

	               Number of training samples

As expected, the results show that as the number of training samples increases, the watermark is learned more strongly. However, even at small numbers of training samples, the watermark is still learned (the p-values are much lower than chance), just to a lesser degree. Overall, we note that watermark learnability and sample efficiency is a continuous spectrum. More samples are helpful but not necessarily always crucial, and sample efficiency varies across watermarking strategies and hyperparameters.

Using multiple different keys/messages

We run additional experiments in Appendix E to investigate the behavior of sampling-based watermark distillation if the samples are watermarked using multiple different keys/messages. We also vary the number of training samples as before. The table below shows watermark detection p-values.

	                     Number of training samples

The results show that using two different keys hinders watermark learning, as indicated by the larger p-values. However, learning is not completely prevented, as the p-values are still much lower than 0.5 and decrease as the number of training samples increases.

This suggests that using multiple watermark keys for generation may be a potential defense against spoofing attacks. However, if multiple watermark keys are used for generation, then all of those keys will need to be tested at detection, so multiple testing correction will need to be performed to obtain accurate, weaker p-values. So, this defense will slightly reduce the statistical power and increase the false negative rate of watermark detection.

We thank you for the useful and insightful feedback and for taking the time to review our paper. We address questions and concerns below.

Clarification of contribution

Please see our general comment clarifying our contribution. https://openreview.net/forum?id=9k0krNzvlV&noteId=UyaqBwxSNh

Replacing decoding-based watermarking with normal decoder

For API-only LLMs, decoding-based watermarking is suitable, since the LLM provider controls generation, and users cannot replace the decoding-based watermarking algorithm with a normal decoder. However, for open LLMs, e.g., Llama 2, users run generation themselves, so they can easily choose not to use a decoding-based watermark algorithm, and many inference libraries do not watermark outputs by default or at all. So, decoding-based watermarking is unsuitable for robust watermarking for open LLMs. Our findings into weights-based watermarking address this key technical challenge towards watermarking for open LLMs, though additional challenges remain, e.g., robustness to further fine-tuning.

Attacks besides fine-tuning

We demonstrate that simple fine-tuning removes weights-based watermarking capabilities, showing that developing weights-based watermarking that is robust to modifying model weights is an important unsolved challenge. Further attacks would not change our conclusion.

Paper length

Per the ICLR author guide https://iclr.cc/Conferences/2024/AuthorGuide, the optional ethics statement and reproducibility statement do not count towards the page limit. The main body of our paper does not exceed 9 pages.

Assigning different watermarking keys

We run additional experiments in Appendix E to investigate the behavior of sampling-based watermark distillation if the samples are watermarked using multiple different keys. We also run different trials varying the number of training samples. The table below shows watermark detection p-values.

                      Number of training samples

The results show that using two different keys hinders watermark learning, as indicated by the larger p-values. However, learning is not completely prevented, as the p-values are still much lower than 0.5 and decrease as the number of training samples increases.

Note that if multiple watermarking keys are used, then the watermark detection will lose statistical power. Any suspected text will need to be tested using each of the keys in use, so multiple testing correction must be performed, leading to weaker p-values and a higher false negative rate.

Additional computational cost

Weights-based watermarking uses standard generation procedures, so there is no additional computational cost at inference time compared to normal LLMs. Compared to decoding-based watermarking, weights-based watermarking has less computational cost at generation time, since there is no need to run an additional decoding-based watermarking algorithm on top of the LLM. Weights-based watermarking does require model training, the computational cost of which varies depending on model size and dataset size, but this cost is one-time and not incurred at inference time.

Relationship to backdoor-based model watermarking

Our findings concern the learnability of decoding-based text watermarking methods. Backdoor-based model watermarking is crucially different in both setting and method. Backdoor-based model watermarking is meant to enable detecting if a suspect model was stolen from a victim model, e.g., through leaking model weights. Detection is done by querying the suspect model multiple times on specially chosen inputs, i.e., the backdoored inputs. If the suspect model produces the chosen backdoored outputs on these backdoor inputs (which are chosen to be different from how a normal, non-backdoored model would behave), then the suspect model is detected as stolen. Note that the backdoored inputs are meant to be stealthy and hard to find, and the backdoored model still behaves normally on normal inputs. On the other hand, decoding-based text watermarking enables detecting if a given text was generated by an LLM using any input prompt. Detection is done without using any LLM, only the tokenizer is needed. Decoding-based text watermarking is designed to affect all model outputs regardless of input, so that ideally all generated texts can be detected as watermarked. Because of these crucial differences in both setting and method, our findings are not directly relevant to backdoor-based model watermarking, and vice versa.

Dear Reviewer 2p7U,

We appreciate all of the valuable time and effort you have spent reviewing our paper. As today is the last day of the discussion period, we gently request that you review our reply and consider updating your evaluation accordingly. We believe that we have addressed all questions and concerns raised, but please feel free to ask any clarifying questions you might have before the end of the discussion period.

Best,

Authors