WMAdapter: Adding WaterMark Control to Latent Diffusion Models

Q: Discrepancy between qualitative and quantitative evaluation in Figure 1.

Different types of watermarks introduce different types of artifacts. Although FID is one of the most widely used quantitative metrics for image quality, it often struggles to accurately reflect the visual impact of diverse artifacts. Bridging the gap between quantitative metrics and human perception remains an open research problem and an active area of investigation.

 

Q: Why weren't all distortions in Figure 8 quantitatively presented in Table 2

Due to space limitations, we had to make a trade-off in presentation. In Table 2, our goal was to provide a balanced overview of both image quality and robustness metrics. Including all attack types and quality metrics in a single table would have been impractical. This design choice is similar to that of Table 1 in Stable Signature. To complement Table 2, we included Figure 8, which provides a more comprehensive evaluation by illustrating robustness under a wider range of attack types and intensities.

 

Q: How does WMAdapter perform against rotation attacks?

Please refer to this anonymous link for our evaluation of rotation robustness. It is important to note that WMAdapter is built upon the public checkpoint provided by Stable Signature, which was not pretrained with rotation augmentation. Despite this, WMAdapter demonstrates robustness to moderate rotation (up to 15 degrees) and performs comparably to Stable Signature. To further enhance robustness under stronger rotation, WMAdapter would need to be built upon a watermark decoder that is pretrained with rotation-augmented data.

 

Q: “The paper does not evaluate against state-of-the-art adaptive removal methods using optimization or generative models”

We would like to clarify that our paper does evaluate against SOTA opensource adaptive removal methods, including both optimization-based and generative approaches. Specifically, for optimization-based methods, we include evaluations against the adversarial attack proposed in [An’24] under both black-box and white-box settings. For generative models, we assess performance against VAE-based methods ([Cheng’20], [Balle’18]) as well as a diffusion-based method ([Zhao’23]). Please refer to Sec. 4.3 and Fig. 5 for detailed results.

 

Q: Lack evaluation against "Leveraging Optimization for Adaptive Attacks on Image Watermarks”

As the code for the referenced paper is not publicly available, we have made our best effort to reproduce the adversarial noising method described in the paper. Specifically, we implemented their approach using the reported hyperparameters. We found that the suggested $\epsilon$-ball of 2/255 produced negligible attack effects. We increased the $\epsilon$-ball to 8/255, reducing PSNR from 34.8 to 30.3 (similar drop to other attacks in our Fig.5), while the bit accuracy dropped moderately from 0.98 to 0.93. This suggests that our method demonstrates resilience to such attacks. We'll include the result in updated version.

 

Q: Discuss the design choice and the explicit trade-offs made between visual quality and robustness”

Our goal is to develop a high-quality, artifact-free watermarking technique. To this end, we propose a non-intrusive watermarking framework that avoids modifying the pretrained diffusion pipeline. So we design Contextual Adapter as a plugin avoiding direct modification of diffusion models. Also we develop Hybrid Finetuning to conduct non-intrusive training.

The relative importance of robustness versus visual quality is indeed a nuanced and context-dependent question. From the perspective of watermarking as a standalone technique, robustness is often considered the primary metric. However, when we consider broader deployment scenarios—such as integrating watermarking into GenAI products, which is a key motivation of our work—the priorities shift. In these contexts, visual quality becomes paramount, as it directly impacts user experience and product adoption. There is a strong practical demand for artifact-free watermarking that preserves the high visual fidelity expected from GenAI products. Therefore, our design values visual quality without compromising essential robustness, striking a balance suitable for real-world applications.

 

Q: “How does the approach scale to larger or smaller watermark sizes beyond the 48-bit watermark?”

Our approach is built on a pretrained watermark decoder. To scale to different watermark sizes, one can simply replace the decoder with a pretrained version of the desired size and retrain only the adapter—the rest of the system remains unchanged. Alternatively, users may train a watermark decoder of arbitrary size themselves.

 

Q: Robustness to significant editing

Thanks for your suggestion. We randomly edit 80% of the image by inpainting, the bit accuracy remains 0.91.

Q: Is it possible to select a better $\lambda$ or or find better checkpoints to enhance image quality?

Thank you for your insightful question. In practice, selecting a better $\lambda$ or checkpoint to enhance image quality proves very challenging. During our joint training experiments (Adapter-V), we extensively explored different $\lambda$ values and checkpoints. However, none could achieve a better trade-off between image quality and robustness compared to Hybrid Finetuning. Specifically, increasing the weight of the image quality loss significantly deteriorated bit accuracy, sometimes even leading to training collapse. Moreover, merely raising the quality loss weight does not effectively suppress the lens flare artifacts shown in Figure 6; it only slightly improves the quantitative metrics.

We believe this issue arises not only from the choice of loss weights or checkpoints but also from modifications to the VAE decoder itself. Joint finetuning significantly differs from the large-scale pretraining originally conducted for the VAE. It adjusts full VAE decoder parameters based on much smaller datasets with distinct distributions, batch sizes, and training objectives (bit acc + quality). Consequently, this easily disrupts the inherent knowledge encoded in the pretrained VAE decoder, leading to reduced image quality or the emergence of artifacts.

In other words, adjusting the loss weight or selecting different checkpoints during joint finetuning cannot effectively resolve the substantial deviations from the pretrained parameter landscape. The most reliable solution remains using the original pretrained VAE decoder directly, as employed in our proposed Hybrid Finetuning method.

Regarding the loss, lower values of $\mathcal{L}(\mathcal{A}(w), \mathcal{V}^{new})$ achieved during joint fine-tuning do not necessarily indicate genuine improvements, as the VAE decoder $\mathcal{V}^{new}$ tends to overfit on the limited training data.

 

Q: Relation To Broader Scientific Literature

Thank you for highlighting this point. We will clarify and discuss the relation of our work to the broader scientific literature in the revised manuscript.

Q: Essential References Not Discussed: Gaussian Shading and FSW

Thank you for the suggestion. In fact, we have already referenced and discussed both works in the Introduction section. Specifically, they are cited as [Yang et al., 2024] and [Xiong et al., 2023], corresponding to Gaussian Shading and FSW, respectively.

 

Q: “It is recommended to include a comparison of watermark effects and potential artifacts under both hybrid training and standalone training modes.”

Thank you for the suggestion. We kindly refer the reviewer to Figure 6, which provides a visual comparison of watermark effects and potential artifacts under different training modes. We will include more examples in the revised manuscript to improve clarity.

 

Q: “The motivation lacks clarity — how do the observations from existing methods logically lead to the need for fine-tuning the VAE decoder and the inclusion of associated modules?”

We would like to further clarify our motivation. Existing methods typically watermark diffusion models by modifying the backbone or VAE decoder parameters, which often results in visible artifacts and degraded visual quality. To address this, we propose a non-intrusive framework that preserves all original diffusion parameters. Specifically, we design a Contextual Adapter that is attached to the VAE decoder, avoiding direct modification of its internal structure. In addition, we introduce a Hybrid Finetuning Strategy to train the Contextual Adapter in a similarly non-intrusive fashion.

If there are any remaining questions regarding our motivation, we would be happy to provide further clarification.