SAT-LDM: Provably Generalizable Image Watermarking for Latent Diffusion Models with Self-Augmented Training

Q4: Does it hold in practice?

A4: By incorporating regularization techniques such as spectral normalization [10] and weight decay, we can empirically maintain the Lipschitz constant $K$ within a manageable range, making this assumption both practical and reasonable in real-world scenarios.

References:
[1] Anil, C., Lucas, J., & Grosse, R. (2019). Sorting out Lipschitz function approximation. arXiv preprint arXiv:1903.03252.
[2] Fazlyab, M., Morari, M., & Pappas, G. J. (2019). Efficient and accurate estimation of Lipschitz constants for deep neural networks. NeurIPS 2019.
[3] Latorre, F., Lodi, A., & Martello, S. (2020). Lipschitz constant estimation for Neural Networks via sparse polynomial optimization. ICLR 2019.
[4] Bousquet, O., & Elisseeff, A. (2002). Stability and generalization. Journal of Machine Learning Research, 2, 499–526.
[5] Zhang, C., et al. (2021). Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115.
[6] Bartlett, P. L., Foster, D. J., & Telgarsky, M. J. (2017). Spectrally-normalized margin bounds for neural networks. NeurIPS 2017.
[7] Ledoux, M. (2001). The concentration of measure phenomenon. Mathematical Surveys and Monographs, Vol. 89.
[8] Bengio, Y., et al. (2017). Deep learning. MIT Press.
[9] Srivastava, N., et al. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56), 1929–1958.
[10] Takeru, M., et al. (2018). Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957.

Thank you for your time and thoughtful dedication to our paper! We have addressed your concerns below and revised the paper to incorporate the reviewers' suggestions. Please let us know if you have further questions.

Q1: The assumption on the Lipschitz continuity of the loss function is somewhat strong:

1. How can one check if it is true?
2. How can one estimate the Lipschitz constant $K$, which may be large and lead to an unsatisfactory upper bound?

A1:

1. Verifying Lipschitz Continuity:
   Many common loss functions, including Mean Squared Error (MSE) and Cross-Entropy Loss, are Lipschitz continuous when predictions $\hat{y}$ are constrained to bounded domains. For example, the MSE loss $L(y, \hat{y}) = (\hat{y} - y)^2$ is Lipschitz continuous when $\hat{y}$ is restricted to a closed interval $[a, b]$, as the gradient $\nabla L = 2(\hat{y} - y)$ is bounded within this range. Similarly, the Cross-Entropy Loss is Lipschitz continuous when prediction probabilities $\hat{y}$ are confined to $(0, 1)$, ensuring bounded gradients. Additionally, activation functions commonly employed in neural networks, such as ReLU and Sigmoid, are proven to be Lipschitz continuous in prior works [1].

2. Estimating the Lipschitz Constant $K$:
   Estimating the Lipschitz constant of neural networks is an active area of research. For instance, Fazlyab et al. proposed a convex optimization-based approach to efficiently estimate the upper bound of a neural network's Lipschitz constant [2]. Additionally, Latorre et al. employed polynomial optimization techniques to compute tight upper bounds for $K$ [3]. These methods provide both theoretical support and practical tools for estimating $K$.

Q2: Provide a detailed explanation on the assumption of Lipschitz continuity of the loss function.

A2: For most commonly used loss functions (e.g., MSE loss, Cross-Entropy Loss) [1,4,5], their variations in the parameter space are generally smooth and continuous. Within the bounded data range (e.g., image data or binary vectors in our case), these loss functions are Lipschitz continuous. Furthermore, the fundamental linear mappings and activation functions (e.g., Sigmoid and ReLU) [6,7] used in neural networks are also Lipschitz continuous. Additionally, weights in neural networks are often regularized to prevent overfitting [8,9], which help ensure that the constant $K$ does not grow excessively large. Thus, this assumption is not overly restrictive and can be satisfied by carefully selecting the loss functions, model architectures, and training strategies.

Q3: How to verify an assumption?

A3: Verifying the Lipschitz continuity assumption can be approached both theoretically and empirically:

1. Theoretical Verification:

   • Function Composition: Analyze the composition of functions within the model. If each individual function is Lipschitz continuous with known constants, the overall Lipschitz constant can be derived based on the composition rules.
   • Bounded Inputs and Parameters: Ensure that the inputs to the loss function and the model parameters are bounded. Boundedness, combined with Lipschitz continuous activation functions, simplifies the verification of the overall Lipschitz continuity.

2. Empirical Verification:

   • Gradient Norms: Compute the gradient norms of the loss function with respect to inputs over a validation dataset. Consistently bounded gradient norms provide empirical evidence supporting Lipschitz continuity.
   • Spectral Norm Analysis: Utilize techniques like spectral normalization to empirically bound the Lipschitz constant $K$. By normalizing the spectral norms of weight matrices, we can effectively control and estimate $K$.

Thank you for your time and dedication to our paper! We have addressed your concerns below and revised the paper to incorporate the reviewers' suggestions. Please let us know if you have further questions.

Q1: The authors claim that under 1,000 AI-generated prompts, the proposed method surpasses the baseline. However, this number of prompts is far too small for general case measurements, especially considering potential bias in the language models used.

A1: Due to the high computational cost, we chose 1,000 prompts for comparison, consistent with prior works such as FSW[1], Tree-Ring[2], and Gaussian Shading[3]. Additionally, we have conducted experiments on two new tasks—watermark detection and user identification (Section E.1)—with generated image sizes of 1,000 and 5,000, respectively. These experiments further demonstrate the effectiveness of our method.

Q2: Additionally, this comparison seems unfair. The authors should provide a comparison of results using selected prompts from LAION-400M (but not used during training). The reported advantage may solely come from the prompt distribution shift between GPT-generated prompts and those from LAION-400M. Since the proposed method forgoes all prompt information during training (“using empty prompt”), it avoids this shift and may appear better (due to bias from GPT-generated prompts).

A2: Thank you for pointing this out. We would like to emphasize that Table 1 already includes comparisons across diverse datasets: COCO, LAION-400M, Diffusion Prompts, and AI-Generated Prompts. Importantly, the prompts used during testing are independent of those in training, ensuring both robustness and fairness in the evaluation.

Q3: Fig. 3 can be interpreted as $d(z_{\text{laion-prompt}}, z_{\text{gpt-prompt}}) > d(z_{\text{no-prompt}}, z_{\text{gpt-prompt}})$, which does not provide any evidence that $d(z_{\text{real-prompt}}, z_{\text{no-prompt}})$ is small in most cases.

A3: To clarify, the interpretation should be $d(z_{\text{laion-image}}, z_{\text{gpt-prompt}}) > d(z_{\text{no-prompt}}, z_{\text{gpt-prompt}})$, as the training distributions consist of images from LAION and free distributions. Regarding the specific concern that $d(z_{\text{real-prompt}}, z_{\text{no-prompt}})$ is small in most cases, we provide additional analysis in Section E.3 (see A13) where we replace GPT-generated prompts with real-world LAION prompts, yielding similar trends and strengthening the robustness of our claims. This approach, while not exhaustive, offers a practical approximation to the diversity of real-world settings and further corroborates the effectiveness of our proposed method.

Q4: In fact, it is widely believed that there is a large distance between the conditional and unconditional distributions in DMs, which is why we use (either classifier or classifier-free) guidance.

A4: We would like to clarify that our theoretical assumption focuses on the similarity between the composite distribution of all prompts and the free/unconditional distribution, rather than on the specific conditional distribution to the unconditional distribution.

Thank you for your time and thoughtful engagement with our paper! We have addressed your concerns below and revised the paper to incorporate the reviewers' suggestions. Please let us know if you have further questions.

Q1: The authors claim that current diffusion-native watermarking methods degrade image quality by comparing watermarked images to unwatermarked counterparts generated from the same prompt. This comparison is irrelevant because users of diffusion-native watermarking methods would not see unwatermarked images beforehand. Such comparisons would only apply to post-watermarking methods, where preserving the original image content is necessary.

A1: We respectfully disagree with the critique of the comparison methodology. The comparison between watermarked and non-watermarked images is designed as a diagnostic tool to quantitatively assess the impact of watermark embedding on image quality. It is not intended to imply that end-users would directly compare these two types of images. Instead, this approach establishes a fair baseline for evaluating the effects of watermarking methods on image quality. By measuring changes introduced by watermarking against an unaltered baseline, our methodology ensures a rigorous assessment. This practice is consistent with prior works, such as Stable Signature[1], FSW[2], and WaDiff[3], which also evaluate methods against non-watermarked counterparts. While we acknowledge that end-users may not encounter non-watermarked images, such comparisons are essential for benchmarking and advancing methods within the research community.

Q2: Additionally, the claim that watermarking performance is compromised across image styles is questionable, as style diversity is largely dictated by the diffusion model's training data rather than the watermarking process itself.

A2: This concern appears to arise from a misunderstanding of the relationship between watermarking and image style diversity. While image style diversity is inherently derived from the diffusion model’s training data, the ability of the watermarking module to generalize across image styles is determined by its training strategy and the distribution of its training data. It is important to note that watermarking methods trained on external datasets may exhibit biases when encountering rare or previously unseen styles.

Q3: The paper’s Figure 2 depicts a VAE decoder as frozen while taking message embedding as input. If the authors follow the FSW structure, as stated in Section 4.2, the figure is incorrect because FSW fuses message embeddings into the UNet-decoder, not the VAE decoder.

A3: Regrettably, this statement is factually incorrect.
Section 4.3 of the FSW [2] paper explicitly states:

“In order to achieve the goal of changing embedding message flexibly without training or fine-tuning again, we fuse the message-matrix $m_w$ in fine-tuned LDM-decoder $D_w$.”

Furthermore, Section 3.2 specifies:

“Note that, the LDM-decoder used in this work is the variational autoencoder (VAE), which is widely used in stable diffusion models.”

As described in the FSW paper, watermark embedding is performed at the VAE decoder stage, which aligns with our implementation. Figure 2 in our paper accurately represents this architecture, where the VAE decoder parameters are kept frozen, and the message embedding is incorporated through a plug-in message processor.

Dear Reviewer iTqJ,

Thank you for taking the time to review our submission and provide your thoughtful feedback. We hope our rebuttal has adequately addressed your concerns. Specifically, we have clarified the misunderstandings, highlighted our main contributions, and provided additional ablation experiments. As the discussion period approaches its end, we kindly request that you review these points and consider updating your evaluation accordingly. Moreover, if you find our response satisfactory, we kindly ask you to consider the possibility of improving your rating.

Thank you very much for your valuable contribution, and we look forward to your response.

Best regards,
The authors

Thank you for your detailed feedback. However, we believe there might still be some misunderstandings, and we would like to provide further clarification.

Q1: actually 'free' (what you proposed) is not really a watermarking method, but a training strategy to improve the generalization ability of the watermarking methods, which is orthogonal to proposing a new watermarking method, and that's why your implementation is highly depended on FSW in this paper.

A1: Thank you for your thoughtful analysis of our method. Regarding the positioning of the "free" strategy, our primary goal was to rethink how watermarking modules in LDM are trained and propose a training approach that enhances training-based watermarking methods. As you pointed out, this strategy extends beyond watermarking tasks; the same design principle can be applied to other tasks with similar architectures that demand improved generalization capability. We greatly appreciate your insights into the broader potential of our method.

Our decision to adopt the FSW-like structure was driven by the need to control experimental variables, ensuring fairness and clarity in demonstrating the core advantages of the "free" strategy. This choice also aligns with our commitment to scientific rigor and resource constraints. For example, training our method from scratch on FSW-like structure using a single RTX 4090 takes less than half a day on average, whereas Stable Signature requires 8 GPUs for around a full day [1], not including its further fine-tuning for different users. According to the Stable Signature paper, their experiments demanded approximately 2000 days or ≈ 50,000 GPU-hours. Additionally, Stable Signature requires separate fine-tuning for different users, whereas the more flexible watermarking design like FSW represents a forward-looking trend in this field.

Q2: Besides, your motivation now comes from 'better generalizing to different image styles when applied to real-world scenarios'. It is kind of tricky to prove such generalization capability. Yes, you proved that the watermarked images have good quality compared to the original image in terms of PSNR and SSIM, those pixel-to-pixel metrics. But there is still a gap between "a good generalization capability" and "a good image quality".

A2: First, we would like to clarify that we have never claimed, nor attempted to demonstrate that, "good generalization ability" and "good image quality" are equivalent. Rather, our definition of "a good generalization capability" explicitly includes image watermark robustness. A robust watermarking method should embody both attributes; therefore, we believe it is essential to address both aspects, rather than overlooking one.

Second, it appears your concern pertains to the gap between theory and practice in our work. However, we argue that this gap does not signify a disconnect, but rather reflects the inherent challenges of aligning local experimental metrics with the complexities of real-world data. In our theoretical analysis, we employed the Wasserstein distance to demonstrate that the "free" generation distribution mitigates the discrepancy between training and testing distributions, providing theoretically supports for improved generalization.

In our experiments, we used watermarked image quality metrics and watermark robustness to assess generalization across different prompts and image styles as proxy indicators of generalization ability. While these metrics may not fully capture all facets of generalization, they are interpretable, practical and relevant for real-world applications. Furthermore, we designed experiments across diverse data distributions (e.g., COCO, LAION-400M, Diffusion Prompts, and AI-generated prompts) to reflect real-world generalization performance. This combination of theoretical analysis and empirical evaluation strengthens our argument of improved generalization.

Thank you once again for acknowledging the contributions of our work! As the rebuttal process is coming to a close, we would appreciate if you could let us know if you have any further concerns and/or consider raising the score.

References:
[1] Fernandez, P., et al. "The stable signature: Rooting watermarks in latent diffusion models.", in CVPR 2023.

Thank you for your time and dedication to our paper! We have addressed your concerns below and revised the paper to incorporate the reviewers' suggestions. Please let us know if you have further questions.

Q1: More visualization results, especially of different image styles, can help demonstrate the generalization of the proposed SAT-LDM.

A1: Thank you for recognizing the significance of our work. In response to your feedback, we have added additional visualization examples in Section F of the revised manuscript. These new results encompass various image styles, as discussed in Section C. By comparing SAT-LDM with different watermarking methods, we further highlight its generalization capability across diverse image styles. We hope these enhancements address your request, and we sincerely appreciate your efforts in helping us improve the quality of our paper.

The authors addressed my concerns. Yet I agree with reviewer iTqJ that the novelty of the proposed method might be relatively limited. Therefore, I will maintain my score.

Thank you for your feedback. We appreciate your acknowledgment that our rebuttal effectively addressed your concerns. Notably, reviewer iTqJ has recognized the theoretical novelty of our proposed method following the rebuttal. In light of this clarification, we kindly request you to reevaluate your score or opinion.