An End-to-End Model For Logits Based Large Language Models Watermarking

W1: Distortion-Free Watermarking

• We have indeed considered distortion-free watermarking in our manuscript and described them as sampling-based methods (EXP). However, we argue that even distortion-free watermarking can degrade the quality of output text. This is supported not only by our empirical results in Fig. 6 but also by the LLM watermark benchmarking studies [A], [B]. Consequently, we emphasize that the robustness/quality trade-off remains a persistent challenge for existing LLM watermarking models and is still an open problem to be solved in the field.

W2: Contributions

• Given the persistence of the robustness/quality trade-off, our primary contribution is achieving a better trade-off through encoder/decoder end-to-end training.
• Additionally, we address the key challenge of incorporating non-differentiable modules into the end-to-end training pipeline by proposing an innovative online prompting approach.
• As highlighted in Fig. 9, our model achieves a better performance across key metrics against SOTA competitors.

W3: Robustness and quality Trade-offs

• Please refer to the "Response to All Reviewers". A detailed comparison of our method with SOTA competitors is presented in Table 5, accompanied by a radar graph in Fig. 9. These results demonstrate the clear advantages of our approach, including significant improvements in robustness, perplexity, and downstream task performance. This highlights our model's ability to achieve a well-balanced performance across key metrics.

[A] Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, et al. Markllm: An open-source toolkit for llm watermarking. arXiv preprint arXiv:2405.10051, 2024

[B] Piet, Julien, et al. "Mark my words: Analyzing and evaluating language model watermarks." arXiv preprint arXiv:2312.00273, 2023.

Dear Reviewer JkAA,

We apologize for reaching out again, but with the rebuttal discussion period concluding in less than 36 hours, Nov 26, 11:59 PM AoE, we sincerely appreciate the continued time and effort you dedicate to discussing our submission.

We justify our motivation with recent LLM watermark benchmarking studies, which confirm the existence of a robustness/quality trade-off in distortion-free watermarking methods. Our contributions are as follows: 1) addressing the robustness/quality trade-off through end-to-end training of the watermark encoder/decoder; 2) tackling the challenge of non-differentiable modules in the training pipeline with an innovative online prompting; and 3) demonstrating superior performance across key metrics compared to state-of-the-art competitors, as highlighted in Figure 9.

We are truly grateful if you could let us know if there are any remaining questions or concerns about our submission. If our responses satisfactorily address your concerns, we kindly hope you might consider reflecting this in your review score.

Thank you once again for your thoughtful feedback and for your consideration.

Best regards,
Submission9685 Authors

Thanks for the rebuttal. My main concerns about the motivation and the contribution are not addressed. I decide to keep my rating.

Distortion-free watermarks have a theoretical guarantee to preserve the text quality, the statement "even distortion-free watermarking can degrade the quality of output text" is not true. The degrade of the text quality observed in the related works could be caused by the different experimental settings. In Hu et al [1], the distortion-free watermark has the same generation quality of the output text as the original LM. Thus, the trade-off between the quality and detectability does not exist.

Hu et al. Unbiased watermark for large language models, ICLR 2024

We would like to clarify that our primary claim focuses on the trade-off between quality and robustness, rather than quality and detectability. In our work, we define detectability as the detection accuracy on unaltered, watermarked text; while robustness refers to the detection accuracy on modified watermarked text. The modifications may include edits made by the user after receiving the watermarked text, such as synonym substitution, paraphrasing, or other changes, which are common in practice.

An ideal watermarking scheme would produce a watermark that is distortion-free (as defined in EXP-Edit[A] and Unbiased[B]) and also demonstrates resilience to potential text modifications, similar to the robustness of Unigram[C]. Unfortunately, to the best of our knowledge, no existing watermarking scheme meets both of these criteria simultaneously.

Therefore, the motivation behind our work is clear: we aim to achieve a better trade-off between quality and robustness by explicitly optimizing the quality and robustness objectives in an end-to-end manner.

In addition to the quality/robustness trade-off, we would like to highlight several key factors of EXP-Edit and Unbiased that compromise, in order to achieve distortion-free watermark embedding, especially when compared to our method:

1. Detection Time Complexity
   We compare the average time required to detect a single watermarked sample using KGW, EXP-Edit, Unbiased, and our method. The experiment is conducted with Llama2-7B on a single NVIDIA A6000 GPU, with the following results:

   Method	Required Time (seconds)
   KGW	0.3
   EXP-Edit	80
   Unbiased	3.4
   Ours	0.005

   Our method is highly efficient, requiring no access to the LLM and benefiting from GPU parallel acceleration. In contrast, Unbiased requires additional access to the LLM and prompts, while EXP-Edit has significantly longer detection times. Our method is 16,000 times faster than EXP-Edit and 680 times faster than Unbiased, making it feasible for scalable watermarking systems.

2. Accessibility
   Unbiased requires access to the token logits of the LLM API and the prompt, which could reduce its accessibility. In contrast, our method, similar to KGW, only requires the text to be detected, making it simpler to deploy.

3. Choice of LLM Decoding Strategy
   Both EXP-Edit and Unbiased work by manipulating the sampling process, and thus do not function with beam search due to the deterministic nature. As shown in Appendix I, our method performs even better with beam search than with multinomial sampling. This is because beam search tends to select higher-probability tokens, which implicitly favors more of the green-list tokens in the generated text.

4. Low-Entropy Scenarios
   EXP-Edit and Unbiased are not effective in generative processes with low entropy, such as code generation, as empirically shown in the MarkLLM benchmark [E]. In contrast, our method demonstrates superior performance in code generation, as illustrated in Fig. 6 (c). Our method is logits-based and can be further enhanced with techniques like SWEET[D] to improve performance in low-entropy scenarios.

We hope this clarifies the motivation behind our work and emphasizes the practical advantages of our approach. Thank you for your valuable feedback.

[A] Kuditipudi et al. Robust Distortion-free Watermarks for Language Models, TMLR 2024

[B] Hu et al. Unbiased watermark for large language models, ICLR 2024

[C] Zhao et al. Provable Robust Watermarking for AI-Generated Text, ICLR 2024

[D] Lee et al. Who Wrote this Code? Watermarking for Code Generation, ACL 2024

[E] Pan et al. MarkLLM: An Open-Source Toolkit for LLM Watermarking, EMNLP 2024

Weaknesses

• We have revised the related work section [A] and added a discussion in Appendix H, highlighting the advantages of logits-based watermarking compared to generation-based methods.
• Our network design prioritizes efficient watermark embedding and detection.
• While our experiments utilize SOTA open-source LLMs, we acknowledge the importance of evaluating our method on the latest architectures, which will be addressed in future work.
• Following the recent benchmark work [B], we present attack evaluation results in Fig. 4 and Fig. 9, with detailed analysis provided.
• The caption for Fig. 2 has been revised to enhance clarity.

Q1: Comparative Analysis

• We evaluate our model against training-based methods (UPV and SIR) in terms of effectiveness, robustness, and text quality. The results demonstrate how our model achieves a favorable trade-off between robustness and quality.

Q2: Efficiency and Generalization

• The inference overhead arises from tokenizing $k$ sequences per step. This can be mitigated through parallel tokenization, reducing time complexity by up to $1/k$.
• As shown in Fig. 4, the similarity in tokenization results across tokenizers enables our converter to perform well if the target sentences align with the training-phase tokenizer.
• Potential failure modes include handling sentences with unrecognized symbols or characters by the training-phase tokenizer.

Q3: Evaluation Scope

• While we have compared our method with several SOTA open-source LLMs, we recognize the need for broader testing across diverse architectures and scales. Due to time constraints, this will be explored in future research. We agree that this is a valuable area for further study.

Q4: Related Work and Claims

• Additional related works have been incorporated in Appendix H.
• The "first end-to-end framework" refers specifically to logits-based LLM watermarking, as stated in the abstract.

Q5: Security Analysis

• The watermark encoder should be protected as a private key (as in KGW), while the watermark decoder can be publicly accessible due to the neural network's black-box nature.
• Appendix E includes experiments on undetectability when attackers have access to paired watermarked/unwatermarked samples. Results show increased detectability compared to unpaired samples in Table 2.

Q6: Architecture Choices

• The LSTM structure mimics the KGW-based method, where green/red lists are derived from preceding tokens. The LSTM network captures temporal dependencies from preceding sequences.
• The lightweight architecture ensures efficient watermark embedding and detection. Without resource limitations, advanced architectures like Transformers could be employed in the watermark encoder/decoder.

[A] Abdelnabi, Sahar, and Mario Fritz. "Adversarial watermarking transformer: Towards tracing text provenance with data hiding." 2021 IEEE Symposium on Security and Privacy (SP), 2021

[B] Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, et al. Markllm: An open-source toolkit for llm watermarking. arXiv preprint arXiv:2405.10051, 2024

W1: Robustness/Quality Trade-off

• For the explanation of the log diversity, please refer to the "Response to All Reviewers."

W2: OOD

• We have fully addressed the OOD problem in our evaluation. The prompts used in the training and testing phases come from different datasets. For training the watermark encoder/decoder, we use the WikiText-103 dataset, which contains articles from Wikipedia. For evaluation, we use the RealNewsLike subset of the C4 dataset, which contains news articles that differ significantly in content from the training set. The results in Fig. 5 demonstrate that our method is robust to OOD scenarios.

W3: Detection and Interpretability

• We use a deep classifier for detection and provide flexibility in controlling the false positive rate by adjusting the threshold for the decoder’s output logits. The ROC-AUC curve in Appendix B illustrates our model’s ability to effectively manage false positive rates through the threshold.
• Additionally, for any given sentence, we can retrieve the logits perturbation (output of the watermark encoder) for each token as shown in Fig. 11, allowing us to determine whether tokens fall into green/red lists. This capability enables the interpretability of the watermark signal, similar to the method used in KGW.

Q1: Watermark Embedding and Model Training

• The watermark embedding is applied at each generation step during the training phase. Our end-to-end model was trained on a single NVIDIA RTX A6000 48GB GPU for 35k steps with approximately 5 days and the GPU memory usage is 21.96 GB. By adjusting the maximum number of generated tokens (set to 100 in our training), we can control the complexity of the computational graph. Further details of our model training are provided in Appendix F.

Q2: Addressing OOD

• Please refer to the response for 'W2: OOD'.

Q3: Cross-LLM Evaluation

• Our model is trained with OPT-1.3B and evaluated on Llama-2 and Llama-2-Chat for cross-LLM inference.

Dear Reviewer QAhy,

We apologize for reaching out again, but with the rebuttal discussion period concluding in less than 36 hours, Nov 26, 11:59 PM AoE, we sincerely appreciate the continued time and effort you dedicate to discussing our submission.

To demonstrate the flexibility of our model in adjusting quality metrics trade-offs, Table 5 shows that setting $k$ to 40 enhances log diversity with a slight increase in perplexity. The ROC-AUC curves in Appendix B show that our model effectively manages false positive rates through threshold adjustments. Fig. 11 illustrates the watermark encoder's output values for each token in a sentence, identifying whether tokens fall into green or red lists, thereby enhancing the interpretability of our method. Detailed training information is provided in Appendix F, and all concerns about OOD issues in prompts and LLMs have been addressed.

We are truly grateful if you could let us know if there are any remaining questions or concerns about our submission. If our responses satisfactorily address your concerns, we kindly hope you might consider reflecting this in your review score.

Thank you once again for your thoughtful feedback and for your consideration.

Best regards,
Submission9685 Authors

W1: Transparency and Robustness/Quality Trade-off

• We understand your concerns regarding transparency. To address this, we have introduced robustness/quality scatter plots based on watermark strength in Fig. 8. These plots demonstrate that our model achieves the best robustness/quality trade-off overall.
• We strictly follow the attack settings established by prior works [A] and [B]. For added clarity, we have included a table in Appendix G that provides descriptions, specific prompts, and hyperparameter settings for each attack presented in Fig. 5.

W2: Related Work of Generation-based Methods

• We have included a citation to [C] and added a discussion on the advantages of logits-based watermarking compared to generation-based methods in Appendix H.

W3: Token distribution and undetectability

• For the explanation of token distribution, please refer to the "Response to All Reviewers". There is indeed an inherent trade-off between robustness and token bias. As shown in Fig. 12, reducing the watermark strength helps lower token bias, though this adjustment could compromise robustness.
• We present detection accuracy against training size for classifying paired (using the same prompt) watermarked/non-watermarked samples in Appendix E. The results demonstrate that existing methods (including our method) can be detected by an external classifier with paired samples when the training size is sufficient. However, we argue that such paired samples are unlikely to be accessible to adversaries, as the adversaries can only request watermarked text from the watermarked LLM API and search for unwatermarked text elsewhere. However, if watermarked/non-watermarked samples are obtained using identical prompts from different LLMs, a detector could exploit the domain gap between the two LLMs to achieve high accuracy with trivial solutions. Thus, we still conduct the paired samples experiment for reference.

W4: Watermarked Samples and Training Pipeline

• We have added examples of watermarked and non-watermarked text samples in Appendix C to validate the quality of our watermarked sentences.
• The parameters of the online LLM are frozen during training so that the online LLM is not updated in training. Therefore, the use of Gumbel-Softmax does not degrade the quality of the LLM output.
• Since we involve the entire LLM in the training pipeline for backpropagating gradients, our method requires relatively large computational resources and time compared to existing training-based methods. We have included detailed information about the training process in Appendix F. Despite this, once the model is trained, we develop an efficient converter for cross-LLM inference, ensuring that the computational cost during inference remains low.

Q1: Detectability Experiment

• We follow the undetectability experiment setup from [A], where the prompts for watermarked and non-watermarked texts differ.
• Refer to ‘W3: Token distribution and undetectability’

Q2: Embedding Domain and Semantic Similarity

• The $L_{\text{sem}}$ term is not a novel contribution but a widely used method to compute semantic similarity between two sentences [D].
• At the beginning of the generation, the strong alignment between $X_{\text{wm}}$ and $X_{\text{nwm}}$ provides solid supervision. As the sequences diverge, gradients flowing back from $L_{\text{sem}}$ at every step still help the watermark encoder learn to minimize semantic differences.
• We argue that computing the $L_{\text{sem}}$ term across a batch of samples remains effective, as shown in the ablation study in Table 4. Removing $L_{\text{sem}}$ leads to a significant increase in perplexity.

Q3: Gumbel-Softmax and Online Text Editor

• The Gumbel-Softmax is employed in the online text editor $N$ to introduce randomness. $N$ is activated with a probability of 0.5, ensuring that the decoder also receives non-edited watermarked text $X_{\text{wm}}$.

[A] Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, et al. Markllm: An open-source toolkit for llm watermarking. arXiv preprint arXiv:2405.10051, 2024

[B] Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. A semantic invariant robust watermark for large language models. In Proc. Int. Conf. Learn. Representat., 2024.

[C] Abdelnabi, Sahar, and Mario Fritz. "Adversarial watermarking transformer: Towards tracing text provenance with data hiding." 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021.

[D] Sachin Chanchani and Ruihong Huang. Composition-contrastive learning for sentence embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.

4. Computational Resources

While our model requires additional resources during the training phase (details provided in Appendix F), we have developed a converter to enable efficient and cross-LLM inference. As shown in Table 3, the time and memory usage for watermark detection remains negligible. Additionally, the time overhead for watermark embedding can be mitigated through parallel tokenization, reducing the time complexity by up to $1/k$. Although this optimization is beyond the scope of our current work, it can be explored further in future research.

Dear Reviewer sSkQ,

We apologize for reaching out again, but with the rebuttal discussion period concluding in less than 36 hours, Nov 26, 11:59 PM AoE, we sincerely appreciate the continued time and effort you dedicate to discussing our submission.

To ensure clarity regarding the robustness-quality trade-offs, we have provided additional explanations, including detailed comparisons in Appendix A that demonstrate the superiority of our model. Furthermore, we have included comprehensive training details in Appendix F and have addressed all the weaknesses and questions raised earlier.

We are truly grateful if you could let us know if there are any remaining questions or concerns about our submission. If our responses satisfactorily address your concerns, we kindly hope you might consider reflecting this in your review score.

Thank you once again for your thoughtful feedback and for your consideration.

Best regards,
Submission9685 Authors