Learning to Watermark LLM-generated Text via Reinforcement Learning

We thank the reviewer for the valuable feedback. We provide our responses to the questions below.

Convergence analysis (Please provided additional explanations on any convergence analysis of the proposed iterative training algorithm.)

Response: We iteratively train the LLM and the detector to minimize the detection loss as shown in Equation 2. Note that both the LLM and the detector training goals are to decrease the detection loss. Since the two goals are aligned (rather than adverse), the process can be viewed as optimizing one goal over two parameter sets, and it is expected to converge to the minimal possible detection loss.

Related works ([1] and [2] mentioned by the reviewer)

Response: We thank the reviewer for pointing to these new categories of related works. We will clarify the difference here and also discuss them in the revision of the paper. First, our method is different with the backdoor watermarking methods ([1] mentioned by the reviewer) in that their methods focus on model watermarking, while our method focuses on output watermarking. The model watermarking methods will usually have a "trigger set", so that only prompts from this set will lead to different outputs that can be detected. On the other hand, our method requires that all the output from the model will be detected by the detector. Second, our method is different with the text-level watermarks ([2] mentioned by the reviewer) in that they focus on the watermarks injected into a piece of text, but not the LLM model. Although we can apply such methods to watermark LLM outputs in a post-hoc way, it requires extra generation overhead (in the case of [2], the time to run a 13B model for paraphrasing). We will include the discussion above in the revision of the paper.

Fair comparison (For the main results, as the proposed method involves finetuning with the training data, a fair comparison would involve the various methods applied to the fine-tuned model on the same training data.)

Response: We agree with the author that the finetuning on the PKU data could help with model performance on the safety score. We would like to argue that the finetuning on the C4 data is purely for watermark injection and does not have impact on model performance (i.e. perplexity). We will only regularize the watermarked model to have low KL divergence with the original model on the dataset, so the extra data will not improve the model performance.

Training setting (Please elaborate on specifically what training data was used and what test dataset was used to evaluate the model performance, especially since the proposed method will have access to the training data while the baselines do not. Would the prompts used during testing need to be similar to the training dataset? The appendix on OOD tasks seem to imply that.)

Response: In the main experiments, the training and testing datasets are from the same data distribution (C4 or PKU), but split in the standard way so that there are no identical data cases in training and testing sets. The experiments on OOD tasks evaluate the case where training and testing datasets are from different distributions.

We thank the reviewer for the valuable feedback. We provide our responses below.

Adversarial attack (The proposed method uses a learnable DNN as the detector, which can increase the watermark detection accuracy since both the LLM and the detector are optimized during watermarking. What if the adversary also provides a detector? That is to say, the adversary can keep the watermarked LLM fixed and optimize another LLM as his/her own detector. Note that the adversary does not need to modify the watermarked LLM, simply train another detector that can extract his/her watermark from the generated text, which would cast ambiguity over the verification process. How to deal with this situation?)

Response: We thank the reviewer for pointing out a potential attack against our pipeline, where an adversary may train a surrogate watermark detector given the output of our watermarked LLM. We have two interpretations of the proposed attack based on the surrogate detector and will discuss them separately.

In the first interpretation, the adversary may claim that the detector trained by him/her is the true detector and therefore casting ambiguity on which detector to trust during the verification. We think that this would not happen in common use cases. Usually, an LLM provider will inject the watermark and keep the detector as a secret. To detect the watermark, the LLM provider will apply the secret detector, so it is difficult for the adversary to add another detector in the pipeline.

In the second interpretation, the adversary may launch an adversarial attack based on the surrogate model, so that a watermarked text can be repharsed to not be detected by the detector. We argue that this attack requires a costly adversarial attack on transformer-based language models, which is still ongoing research and we do not know, to the best of our knowledge, any widely recognized perfect attacks. Merely attacking the surrogate model is already difficult, as the AdvGLUE benchmark on language model robustness [a] shows that the average attack success rate on binary classification language models is only around 60%. The transferred attack success rate would be lower.

[a] Wang, Boxin, et al. "Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.

Code Opensourcing (Will you release the source codes?)

Response: Yes, we will open-source the code of our work when we publish our paper.

We thank the reviewer for the valuable feedback. We provide our responses to the questions below.

Requirement of Original Prompt (Does the Detector D require prompt-completion text pairs as input? Does this mean the original prompt is needed during use? Doesn't this severely limit the watermark's practical applications?)

Response: We thank the reviewer for pointing out the requirement of original prompt in the paper. We would like to make two clarifications. First, we have the experiments of training and detecting the watermarks without the knowledge of original prompt in Appendix B.4 and Table 9. We show that without the knowledge of the prompt, our method can still achieve very high detection performance on C4 (AUC=0.9984) and PKU (AUC=0.9991). Second, we choose the setting of knowing the original prompt because it is reasonable in the prompt completion setting. Suppose the LLM is used to complete some prompt in the C4 news dataset, the resulting output to show to the public will more likely be the overall text (prompt + completion), rather than the completion-only text. We will make these two points clear in the main text of the paper.

Perplexity (Why doesn't the no-fine-tuning method achieve the lowest perplexity?)

Response: We thank the reviewer for referring to the perplexity numbers in the table. The perplexity metric is a commonly used metric in LLM watermark literatures (e.g. KGW / ITS papers) to evaluate the text quality of the watermark. In the KGW paper, the PPL values have a wide range (3 to 13) and the text quality changes notably. In our case, perplexity is generally similar, and we view the small difference of perplexity as a result of randomness in text generation. The numbers show that our method will not affect text quality in general.

Baseline Implementations (Would it be more fair to compare robustness with KGW family's unigram methods (“Provable robust watermarking for ai-generated text”); What are the parameter settings for baseline methods in the experiments? For example, what are the size of green list and delta values in KGW?)

Response: We thank the reviewer for referring to the unigram methods. We viewed the provable robust watermark work as a variant of KGW, and therefore chose to discuss it in the related work but did not compare it in the experiments. According to Table 2 in their paper, the watermark still suffers from a notable performance drop under paraphrasing. For the baseline parameter setting, we use the default setting in their original implementation. Specifically, we use $\gamma=0.5$ and $\delta=2.0$ for KGW; we use $n=256$ for ITS and EXP. We will add the information in the revision of the paper.

Robustness evaluation (In the robustness experiments, did they only consider modifications to the generated text, or did they also examine how prompt modifications might affect watermark detection results?)

Response: We only made modifications to the generated texts but not the prompts. It is true that prompt modifications may also affect watermark detection. On the other hand, if we use the setting where the detector only sees the generated text (discussed in the text above, Appendix B.4 and Table 9), it will not affect the detection.

We thank the reviewer for the valuable feedback. We provide our responses to the questions below.

Requirement of Original Prompt (The detection needs the original prompt, which is usually unavailable during the detection process.)

Response: We thank the reviewer for pointing out the requirement of original prompt in the paper. We would like to make two clarifications. First, we have the experiments of training and detecting the watermarks without the knowledge of original prompt in Appendix B.4 and Table 9. We show that without the knowledge of the prompt, our method can still achieve very high detection performance on C4 (AUC=0.9984) and PKU (AUC=0.9991). Second, we choose the setting of knowing the original prompt because it is reasonable in the prompt completion setting. Suppose the LLM is used to complete some prompt in the C4 news dataset, the resulting output to show to the public will more likely be the overall text (prompt + completion), rather than the completion-only text. We will make these two points clear in the main text of the paper.

Distinguishing between other LLMs (This paper uses the D^{nw} (human-written prompt and answer) to fine-tune the LLM and detector. What I am worried about is that the detector learned the difference between human-written text and LLM-generated text instead of un-watermarked text (text generated by unwatermarked LLMs) and watermarked text.)

Response: We appreciate the reviewer's concern on the possibility of detector's bias towards human vs. non-human texts. In Appendix A and Table 6, we evaluated the watermark detector on texts generated by other LLMs. We show that the detector can correctly classify the text by other LLMs as non-watermarked with high accuracy. In addition, we argue that this problem can be addressed with better training data selection. When we include the text generated by other LLMs in the training process, we can further improve the detector performance (denoted as "H+L" in Table 6).

PPL computation (Could authors specify the model used to measure the PPL?)

Response: We followed the KGW paper and used the OPT-2.7B model to measure the PPL.

Other model performance (This watermarking method changed the parameters of the original LLM. I think it would be good to measure if this fine-tuning affects the performance of the original LLM using methods like FActScore, AlpacaFarm, etc.)

Response: Currently, we evaluate the performance of LLM on the original task with perplexity (for C4) and safety score (for PKU). These are standard methods to evaluate the LLM performance after watermarking. We agree with the reviewer that the extra metrics are good ways to evaluate the effect of watermarks on original performance, and we plan to include them in future versions of the paper.