WMAdapter: Adding WaterMark Control to Latent Diffusion Models

We sincerely thank the reviewer for their critical insights and thoughtful suggestions. However, we would like to politely point out that some statements in the review appear to be factually incorrect. We aim to clarify these points and are open to further discussion.

 

Argument 1: “I think both changing the VAE decoder's parameters and changing the internal embedding will hugely affect the quality of the generated image.”

This argument contradicts the facts. Our WMAdapter demonstrates that changing the internal embedding does not significantly affect the quality of the generated image. This is supported by the numerical image quality metrics provided in Table 2, as well as the visualizations in Figures 10, 11, and 12.

 

Argument 2: “I do not think changing the VAE decoder's parameters and changing the internal embedding of the image are mathematically different.”

1. They are indeed mathematically different. Intuitively, pretrained knowledge (including how to generate high-quality images) is stored in the VAE decoder’s parameters. Fine-tuning the VAE decoder disrupts these parameters and significantly damages the pretrained knowledge. In contrast, changing the internal embedding leaves all network parameters intact, thus preserving the pretrained knowledge and avoiding catastrophic forgetting. This approach effectively utilizes the pretrained VAE as a regularizer, maintaining better image quality while optimizing the internal embeddings.

2. In simplified mathematical terms:

   • Let $f_{\theta}()$ represent a layer of the VAE with parameters $\theta$, and $x$ represent intermediate features.
   • Tuning parameters: $\min_{\theta} f_{\theta}(x)$, disrupts the pretrained knowledge stored in $\theta$.
   • Changing embeddings: $\min_{x} f_{\theta}(x)$, keeps $f_{\theta}$ as a regularizer, preserving the pretrained knowledge in $\theta$.

3. Examples from other domains further support this distinction:

   • Vison-language fine-tuning techniques: Adding and fine-tuning newly introduced modules (changing the embedding) while freezing pretrained components is a widely adopted strategy in multi-modal fine-tuning. Typical examples include flamingo [6] and blip [7].
   • Video diffusion models: Many video diffusion methods, such as AnimateDiff [4] and Tune-A-Video [5], are based on well-pretrained image diffusion models. These methods only fine-tune newly introduced temporal layers while keeping all other parameters intact, thereby preserving the pretrained knowledge of image diffusion models to ensure high-quality spatial appearance generation.

 

Q3: Is WMAdapter just simply a trade-off between image quality and robustness?

1. Comprehensive Evaluation of the Trade-off: All watermarking methods inherently involve a trade-off between image quality and robustness. A more comprehensive and fair evaluation considers these two attributes as independent dimensions. In Figure 1, we present such a two-dimensional comparison, where points closer to the top-left corner represent a better quality-robustness trade-off. The results show that our method achieves a superior trade-off compared to other methods. Notably, we push up the trade-off boundary instead of merely moving along it.

2. Value and Insights: Our method not only provides better image quality in numerical metrics but also offers critical insights for diffusion-native methods to address the challenge of visually perceptible artifacts introduced by watermarks. This is an important and long-standing problem that prior methods have not adequately addressed (Figure 13). Our solution is both unique and novel in this regard.

3. Different Methods Have Different Strengths: It is important to recognize that different watermarking methods excel under different types of attacks. For example, while Stable Signature may perform better under certain image transformations, our method demonstrates greater robustness against regeneration attacks. detailed in the next question.

We sincerely appreciate your thoughtful feedback and are pleased to provide the following responses to address the concerns raised.

 

Q7: Does this imply that the output image quality remains unchanged regardless of modifications to the internal embedding, provided that the parameters of the VAE remain intact? This behavior seems counterintuitive—could you provide evidence or references to support this claim?

No, this does not imply that the output image quality remains entirely unchanged when the internal embedding is modified. Modifying the internal embedding and fine-tuning the parameters of the VAE both impact the output image quality. This just means that optimizing the internal embedding is a significantly better approach than fine-tuning the VAE parameters, as it preserves the pretrained knowledge and results in much higher image quality.

For evidence, please refer to the image quality metrics comparison in Table 5, Table 2 and the visual results in Figure 6. Both Adapter-F and Adapter-I (modifying the embedding) achieve substantially better PSNR, SSIM, and FID scores than Adapter-V and StableSignature (fine-tuning the parameters). Furthermore, Adapter-I suppresses visual artifacts more effectively.

 

Q8: Tune the hyperparameters to ensure comparable image quality for robustness comparisons or comparable robustness for image quality evaluations.

Thank you for the suggestion. Evaluating all methods at comparable visual quality or robustness levels is an effective approach. However, in practice, most watermarking methods lack explicit hyperparameters to directly control the final trained model to achieve a predetermined visual quality or robustness level. Additionally, the trade-offs between image quality and robustness vary significantly across different methods. Forcing these methods to achieve the same level of quality or robustness would likely result in suboptimal solutions for most. Consequently, controlling variables for a direct comparison is challenging, which is why prior works typically compare the final model results as they are.

In Figure 1, we present an alternative, more fair and practical approach: directly comparing the two-dimensional robustness-quality trade-off of the final models. Our method achieves a better trade-off than other methods. Importantly, we push up the trade-off boundary rather than merely moving along it, clearly demonstrating the improvements enabled by our approach.

Dear Reviewer 7bru,

As the deadline for the final decision is approaching, we want to kindly check if you have any further concerns or questions regarding our rebuttal. Please let us know if there is any additional clarification or information we can provide to address your concerns. We appreciate your efforts and understanding, especially during this busy period.

Best regards,

The Authors

Q1: WADIFF is contextual watermarks.

1. You are correct that WADIFF is contextual watermarks. We apologize for the incorrect citation in line 413, which may have caused confusion. This has been corrected, and the citation has been removed in the revised manuscript. In the introduction (line 90), we briefly compared WADIFF and WMAdapter. Here, we provide a more detailed comparison.

2. Comparison between WADIFF and WMAdapter:

   • Adapter Structure: WADIFF uses ControlNet as its adapter structure, which is heavyweight, with approximately 700M parameters. In contrast, WMAdapter is specifically designed to be lightweight, with only 1.3M parameters, greatly reducing memory cost.
   • Computational Cost: WADIFF integrates the adapter into the diffusion UNet backbone, requiring a 1000-step denoising sampling process during training, which incurs significant computational overhead. WMAdapter, on the other hand, introduces the adapter at the VAE decoder level, avoiding the denoising sampling process during training and greatly reducing computational cost.
   • Visual Effects: WADIFF alters the layout of the generated images, while WMAdapter maintains the original layout and is an imperceptible watermark.
   • Numerical Results: WMAdapter achieves better tracing accuracy than WADIFF (~1.000 vs 0.934 when tracing $10^6$ keys), as demonstrated in Table 3.

 

Q2: Compared to Stable Signature, WMAdapter sacrificies robustness to achieve better visual quality.

1. All watermarking methods inherently involve a tradeoff between image quality and robustness. A more comprehensive and fair evaluation considers these two attributes as separate dimensions, directly comparing the quality-robustness tradeoff in a two-dimensional figure. In Figure 1, we present such a comparison, where points closer to the top-left corner indicate a better balance. The results show that our method achieves a superior quality-robustness tradeoff compared to Stable Signature. Specifically, WMAdapter-I achieves a 22% FID improvement with only a 3% accuracy gap. Unlike merely moving along the existing quality-robustness tradeoff boundary, our method pushes up the boundary, achieving a better overall balance.

2. It is also worth emphasizing the unique advantages of WMAdapter that Stable Signature does not offer. Unlike Stable Signature, WMAdapter provides scalability by encoding $2^{48}$ watermark patterns directly into the adapter, eliminating the need to fine-tune the VAE decoder for each different watermark. Furthermore, as shown in the new experiments in Figure 5, WMAdapter demonstrates better robustness against regeneration attacks compared to Stable Signature. Regeneration attacks typically cost 4-6 db PSNR to remove our watermark, whereas removing Stable Signature costs 2 db.

3. In summary, our method achieves a better quality-robustness tradeoff than Stable Signature when considering both image quality and robustness. Additionally, WMAdapter offers unique advantages, including scalability and enhanced robustness to regeneration attacks, further demonstrating its distinctive strengths.

 

Q3: Reconsider setup for the FID metric.

Thank you for your careful inspection. We apologize for the misleading description in the original manuscript. To clarify, we did calculate the FID between the watermarked images and the real images from the COCO validation set (referred to as "the images before watermarking" in the previous version). We have updated the description in the revised manuscript to avoid any confusion.

 

Q4: Evaluate WMAdapter-I against adaptive attacks.

We have updated Figure 5 and corresponding texts to include WMAdapter-I and another baseline method, Stable Signature, for a more comprehensive comparison under different adaptive attacks (regeneration attacks, black/white-box adversarial attacks, black-box query-based attacks). All three methods are robust to black-box adversarial attacks and vulnerable to white-box adversarial attacks. For regeneration attacks, the results show that attackers need to incur a 4–6 dB PSNR reduction to remove our watermark (both WMAdapter-I and WMAdapter-F), whereas only a 2 dB reduction is sufficient to remove the watermark of Stable Signature. This demonstrates that our method exhibits significantly better robustness to regeneration attacks. Although the query-based attack can successfully attack all three methods, it will cause the image quality to drop significantly to about 8db.

Q1: Lack of Baseline Comparisons

1. Comparison with StegaStamp: We have included an additional comparison with StegaStamp. Our results show that WMAdapters achieve better image quality and generally stronger robustness compared to StegaStamp. Note that we don't include ECC for a fair comparison.

   Method	PSNR	FID	TPR	None	JPEG	Crop	Brightness	Combine
   StegaStamp	29.3	9.9	1.0	0.96	0.96	0.49	0.94	0.49
   WMAdapter-F	33.1	2.7	1.0	0.99	0.92	0.99	0.99	0.92
   WMAdapter-I	34.8	2.5	1.0	0.98	0.90	0.97	0.97	0.90

2. Comparison with Stable Messenger: Stable Messenger is concurrent work, and since its code is not open-sourced and it was trained on different datasets, performing a direct numerical comparison is challenging. Instead, we provide a comparison from the perspective of method design.

   • Motivation: Stable Messenger primarily focuses on improving message (word) accuracy and introduces a novel Loss-Sum-Exponential (LSE) Loss and a latent-aware message encoder to achieve this goal. In contrast, WMAdapter focuses on improving image quality and suppressing generated artifacts.
   • Solution and Model Design: Both works adopt different solutions tailored to their objectives. While the latent-aware message encoder in Stable Messenger is somewhat similar to our contextual adapter, our adapter is specifically designed to take multi-level features from the VAE decoder as contexts, whereas the latent-aware message encoder relies solely on single-level latent features.
   • Robustness: Stable Messenger, as noted in its limitations section, is vulnerable to geometric transformations. In contrast, WMAdapter demonstrates robustness to such transformations, including crop and resize operations.

   These differences in motivation, solution, model design, and robustness characteristics distinguish WMAdapter from Stable Messenger.

3. Updates:

   • We have included a comparison with Stable Messenger in the updated Related Work section.
   • The results for StegaStamp have been added to Table 2 in the updated manuscript.

 

Q2: Confusing Contribution – No extra fine-tuning and hybrid fine-tuning demonstrate a trade-off between accuracy and image quality, a common issue in this field

1. Methodological Perspective: While it is true that "no extra fine-tuning" and different fine-tuning strategies result in varying numerical trade-offs, we want to emphasize that hybrid fine-tuning is a novel and unique strategy. It is the only approach that effectively suppresses visual artifacts beyond the numerical trade-offs. As shown in Figure 6, both "no extra fine-tuning" and other fine-tuning strategies introduce perceptible artifacts. Hybrid fine-tuning is neither a trivial solution nor merely a trade-off mechanism that sacrifices accuracy for improved numerical quality metrics.

2. Value and Insights: Hybrid fine-tuning provides crucial insights into how diffusion-native methods can eliminate visually perceptible artifacts introduced by watermarks. This addresses an important and long-standing problem that prior methods (see the artifacts generated by other methods in Figure 13, although some of them achieve good numerical indicators) or other fine-tuning strategies have not resolved.

 

Q3: Unclear Robustness Evaluation – Evaluating other watermarking methods under regeneration attacks

1. Comparison with Stable Signature: We have included a side-by-side comparison with Stable Signature and WMAdapter-I in Figure 5. For regeneration attacks (Cheng’20, Balle’18, Zhao’23), we observe that removing the watermark imprinted by WMAdapter requires a PSNR drop of 4–6 dB. In contrast, the watermark of Stable Signature can be removed with only a 2 dB reduction in image quality. This demonstrates that WMAdapter exhibits significantly better robustness against regeneration attacks compared to Stable Signature.

2. In terms of AquaLoRA: AquaLoRA greatly alters the layout of the original image, resulting in a very low PSNR of 8.6 dB, which deviates substantially from imperceptible watermarks like Stable Signature and WMAdapter, both of which maintain PSNR around 30 dB. This significant difference makes it challenging to include AquaLoRA in Figure 5 for a fair comparison on PSNR changes under different levels of attacks.

Q1: Dependency on Pre-trained Components

Our method can work with any public or privately trained watermark decoders. In principle, it also supports joint fine-tuning of the watermark decoder during training. Using a pretrained decoder aligns with the conventions of other watermarking methods, such as Stable Signature, SSL, and WADIFF. Additionally, it facilitates direct and fair comparisons with the baseline method, Stable Signature.

 

Q2: Compare with Stable Signature and ControlNet to highlight unique contributions

1. Comparison with Stable Signature: Stable Signature fine-tunes the VAE decoder to embed a fixed watermark, requiring a new fine-tuning process each time a different watermark is embedded. This fine-tuning, especially on small datasets, often degrades image quality and introduces artifacts. In contrast, WMAdapter trains an adapter capable of embedding any 48-bit message, offering scalability and eliminating the need for repeated fine-tuning. Compared to other external watermark adapters [1][2], WMAdapter introduces a contextual structure and employs a hybrid fine-tuning strategy, avoiding modifications to pretrained VAE parameters and maintaining high image quality while embedding watermarks.

2. Comparison with ControlNet: ControlNet is a heavyweight adapter attached to the diffusion UNet, with a parameter size of approximately 700M. In contrast, WMAdapter is a lightweight adapter attached to the VAE decoder, with only 1.3M parameters. Watermarking methods using ControlNet, such as WADIFF [3], alter the layout of generated images and require an additional computationally expensive training process with 1000 steps of diffusion sampling. WMAdapter, however, does not alter the image layout, and its training process is simple and low-cost, avoiding the lengthy diffusion sampling steps. In addition, WMAdapter achieves better tracing accuracy than WADIFF, as shown in Table 3.

3. Summary: WMAdapter fundamentally differs from both Stable Signature and ControlNet in methodology, providing a scalable, lightweight, and efficient solution while maintaining high-quality image generation.

 

Q3: Include additional adaptive attacks

1. Per the reviewer's request, we have added experiments for both query-based black-box attacks and white-box attacks. We also compared the performance of WMAdapter-I and StableSignature under these new attack scenarios. The results and relevant descriptions have been updated in Figure 5 and the corresponding experimental section. Please refer to the manuscript for detailed explanations.

2. Below is a summary table of the results for WMAdapter-F under the newly introduced adaptive attacks.

   • For the query-based black-box attack, we adopted the WEvade-B-Q [4] method and set the detection threshold $\tau = 40$ bits to control $FPR = 10^{-6}$. For the white-box attack, we adopt the targeted attack with random labels.
   • The results show that both the black-box query-based attack and the white-box attack were successful. However, the black-box query-based attack significantly degraded image quality, with an average PSNR of approximately 8 dB.
   • Another key observation is that defending against white-box attacks remains challenging for watermarks, as this issue applies generally to any other classifiers. Incorporating adversarial training with PGD augmentation could potentially improve watermark robustness against white-box attacks. However, resistance to white-box attacks is not the main focus of current watermarking methods.

3. Comparison Table

   Attack Method	PSNR	Bit Accuracy
   Black-Box Regeneration Attack	29.2	0.56
   Black-Box Adversarial Attack	32.5	0.97
   White-Box Adversarial Attack	32.2	0.51
   Black-Box Query-based Attack	8.2	0.77

 

Q4: More severe JPEG compression levels

Figure 8 in the Appendix provides a more detailed performance evaluation across various attack intensities and types. Notably, under a more severe JPEG compression rate (JPEG 30), WMAdapter still achieves a bit accuracy of approximately 0.8. Additionally, WMAdapter demonstrates strong robustness under other types of image transformations as well.

Q1: The mechanism behind hybrid fine-tuning is not fully explained

1. The core idea of hybrid fine-tuning is to preserve as much knowledge as possible that the pretrained VAE decoder has learned from large-scale datasets, such as reconstructing high-quality images, as the quality of the VAE decoder significantly impacts the final image generation quality.

2. We argue that fine-tuning the VAE decoder can easily disrupt its finely tuned pretrained state due to the following factors: (1) Fine-tuning involves hyperparameters such as learning rate, batch size, and data distribution, all of which differ from those in pre-training and may cause large parameter state shift during finetuning. (2) Fine-tuning on a small dataset risks overfitting to the specific data domain. These challenges often lead to a significant drop in generation quality, introducing additional artifacts. Despite using various image consistency regularizations, it remains difficult to match the reconstruction quality achieved during pretraining. Such phenomena have been observed in watermarking methods like StableSignature and FSW [1] that fine-tune VAE decoders. In our experiments, Adapter-V, which uses a fine-tuned VAE decoder, shows a similar drop in image quality to StableSignature, with comparable PSNR scores (29.9 vs. 29.7).

3. By substituting the fine-tuned VAE decoder with the original pretrained VAE decoder (as done in Adapter-I), we effectively mitigate the quality degradation caused by fine-tuning. Since the Adapter adapts well to the original VAE decoder during the first stage of training and introduces only a minimal 48-bit watermark, they do not significantly compromise the generation quality in our design. This explains why Adapter-I achieves better quality than Adapter-V in our experiments, even though this outcome might seem counterintuitive at first.

 

Q2: The robustness is weaker than that of StableSignature and SSL, evaluating all methods at comparable visual quality levels to assess robustness fairly.

1. Thanks for the suggestion. We agree that all watermarking methods inherently involve a tradeoff between image quality and robustness. Evaluating all methods at comparable visual quality levels is indeed a fairer approach. However, in practice, most watermarking methods lack explicit hyperparameters to directly control the final trained model to achieve a predetermined, unified visual quality level (e.g., in terms of PSNR, FID, and visual artifacts). As a result, controlling variables for a direct comparison is challenging, which is why prior works typically compare the final model results as they are.

2. Another approach for a fair comparison is to treat quality and robustness as two dimensions and directly compare the quality-robustness tradeoff in a two-dimensional figure. We present the relevant results in Figure 1, where points closer to the top-left corner represent a better robustness-quality tradeoff. Our method achieves a superior tradeoff compared to both SSL and StableSignature:

   • Compared to SSL, WMAdapter-I achieves an 83% improvement in FID and a 2% improvement in accuracy under combined attacks.
   • Compared to StableSignature, WMAdapter-I achieves a 22% improvement in FID with only a 3% accuracy gap.
   • Rather than simply moving along the quality-robustness tradeoff boundary, our method pushes up this boundary to achieve a better balance

3. Additionally, it is worth highlighting the unique advantages of WMAdapter that neither StableSignature nor SSL can provide.

   • Compared to StableSignature, WMAdapter offers scalability, eliminating the need to fine-tune the VAE decoder for each new watermark. As shown in the new experiments in Figure 5, WMAdapter also demonstrates superior robustness against regeneration attacks.
   • Compared to SSL, WMAdapter introduces no visual artifacts beyond numerical quality metrics (Figure 12) and avoids the need for per-image optimization.

 

Q3: The bit length used by methods like RoSteALS is not specified.

1. All methods embed a consistent 48-bit message. We train RoSteALS and WOUAF from scratch to make it a fair comparison.

[1] Xiong, Cheng, et al. "Flexible and secure watermarking for latent diffusion model." Proceedings of the 31st ACM International Conference on Multimedia. 2023.