Attack-Resilient Image Watermarking Using Stable Diffusion

Thank you for your constructive comments and critical feedback. Below we provide one-to-one responses and detailed discussions.

Q1: Are all watermarking methods compared fairly?

A1: Please refer to the global response to all reviewers. Thank you!

Q2: Has the definition of robustness clearly explained in this paper?
Q3: Can you justify the selection of the attack methods used in testing the robustness?

A2&3: We address these two questions together as they are closely related.

We define watermark robustness as the ability to detect the encoded watermark in images after they have been subjected to malicious attacks, as mentioned in lines 94-96. Consider a real scenario involving the use of image watermarking techniques. An image owner will watermark their work before publishing it online. Malicious actors may then attempt to remove the watermark and republish the image. The goal of these attacks is to maintain the image quality within an acceptable range of the published watermarked images while removing the watermark.

Consequently, when evaluating the robustness of watermarking methods, we selected attack methods that do not cause significant image changes, as measured by PSNR and SSIM metrics relative to the watermarked images before the attack. Specifically, we adopted the attack methods used in [1] as mentioned in line 264. These attacks degrade the image slightly but still maintain acceptable PSNR (above 25dB) and SSIM (above 0.7) levels (refer to Table 1 in [1]).

For your reference, we provide the detailed attack settings below and will include these details in the paper:

• Adjustments in brightness or contrast with a factor of 0.5.
• JPEG compression with a quality setting of 50.
• Image rotation by 90 degrees.
• Addition of Gaussian noise with a std of 0.05.
• Gaussian blur with a kernel size of 5 and std of 1.
• BM3D denoising algorithm with a std of 0.1.
• Two VAE-based image compression models, Bmshj18 and Cheng20 with compression factors of 3.
• A stable diffusion-based image regeneration model, Zhao23 with 60 denoising steps.

In Table 1, we also present the image quality of the attacked watermarked images on the MS-COCO dataset in our experiments. Our results indicate that the image quality of the attacked watermarked images is not significantly degraded compared to the watermarked versions. An exception is the image rotation by 90 degrees, which results in lower PSNR and SSIM due to the rotated content; however, this case is still worth discussing as it presents unique challenges, as detailed in line 300-305 and Appendix Section C.1. In short, we expect a robust watermarking method to detect the watermark from these slightly degraded images.

Table 1. The image quality of the attacked watermarked images compared to those without being attacked.

[1] Zhao, Xuandong, et al. "Invisible image watermarks are provably removable using generative ai." arXiv 2023.

I appreciate the authors' effort in the rebuttal. However, the response does not sufficiently address my concerns. Here are the reasons:

1. How is table 1 (image quality of the attacked watermarked image) related to your claim "image quality within an acceptable range"? Or, what is an acceptable range?

If I take it as "introducing changes to the images that are undetectable by human eyes", I am pretty sure a number of perturbations tested (e.g., brightness, G-Noise, maybe more) can be visibly detected. If I take it as PSNR $\ge 20$, then the question is why PSNR $= 20$ is a reasonable cutoff. Can you further clarify your argument and make it rigorous?

2. I do not see how the provided response support a "fair comparison".

First, comparison at the same watermark detection threshold is hardly ever a fair strategy. Let me give you an example with two watermarking systems, where both rely on thresholding the bitwise accuracy (with some abuse of notation, let's use $I$ as un-watermarked image, $I_w$ watermarked images and $D_i$ as the decoder):

1. System (1): $D_1(I) = 0, ~ \forall I$ and $D_1(I_w) = 1, ~ \forall I_w$
2. System (2): $D_2(I) = 0.4, ~ \forall I$ and $D_2(I_w) = 0.6, ~ \forall I_w$

System (1) and (2) essentially provide the same performance (if measured by TPR-FPR curve for example). Will setting a threshold $p=0.01$ and compare reach the same conclusion?

The above example only uses bitwise acc., not to mention that the proposed method "ZoDiac" is thresholding on a template matching distance.

Second, comparison at equivalent FPR levels is not correct. The only rigorous way in this direction is to tune the threshold $p$ of each method and achieve the exact same FPR level (control of variables), then TPR (not only for $I$, but also $I_w$ under different attacks) can be meaningfully compared. If we want to cross-compare the performance among different attacks, the level of attack (e.g., image quality loss) should also be explicitly controlled to the same level. Otherwise, they are not cross-comparable as my comment in the previous iteration.

Third, why the numbers in Table: False Positive Rate on original images and Watermark Detection Rate (WDR) can be meaningfully sum and average to come up with metric Avg. WDR?

For example, I can make a counter-argument that (1) rotation, contrast, G-noise and Zhao23 introduce visible corruptions by human eyes and should not be considered; (2) Bmshj18 and Cheng20 are essentially similar methods and should be combined as one; etc. Of course, my argument above is debatable and not convincing, but so as yours at current level. Can you provide further logical justification why this is a valid metric? Otherwise, it falls into my comment above (in "Second").

Fourth, if we take one step back and acknowledge that no single watermark currently outperforms the others under every attack, would you agree that showing and discussing the difference in behavior (e.g., XX is better under G-Noise but worse under Cheng20) is more sensible than trying to get a score and claim the overall superiority?

Lastly, I acknowledge the paper [1] the authors pointed to as their major support of "fair" comparison setups. Unfortunately, it is not a published work in any peer-reviewed conferences yet and it also suffers the non-rigorous evaluation setups as well. It does not sufficiently justify the evaluation approach you have taken in this paper. I highly recommend the authors to [2] and consider how they evaluated the watermarking systems and why.

[1] Zhao, Xuandong, et al. "Invisible image watermarks are provably removable using generative AI." arXiv 2023.

[2] An, Bang, et al. "WAVES: Benchmarking the Robustness of Image Watermarks." Forty-first International Conference on Machine Learning.

Thank you for the prompt response. However, it seems to me that most of my concerns still remain. Please allow me to be brief:

1. About acceptable attack quality of image.

The authors have pointed to reference [1]. However, I do not see how [1] can support the authors claim that "PSNR < 20 is deemed unacceptable" in the watermark application. Can the authors give any further logical argument how they arrive from [1] to this claim, for example, in terms of content protection in watermarking problems which has been mentioned as the motivation of this work in the introduction?

2. About WDR

I do not have question how WDR is calculated. My concern is why WDR is a valid measure logically. That is to say, what is the logical argument that the watermark detection results from different attacks can be cross-comparable and averaged?

For example, if "PSNR < 20 is deemed unacceptable" is the standard, why don't we tune the hyper-parameters of all attack methods to achieve the attack images with PSNR=20 first, then test the watermark detection performance using these attacked images and calculate the WDR? To clarify, I do not mean that WDR is a sensible measure under this condition, but it seems to me a fairer setup than the authors have practiced.

3. About the detailed discussion/comparison of performance between ZoDiac and other existing methods provided by the authors in the reply (excluding content related to WDR).

This detailed discussion is indeed sensible, e.g., identifying ZoDiac performs better under Zhao23 but worse on other more traditional image editing (such as jpeg, G-Noise), and is in fact more constructive and informative than trying to use WDR as a unified score and claim "the best". I would suggest the authors consider using these discussions in their main paper.

4. About suggesting following the evaluation proposed by [2]

This suggestion is largely due to the unsettled logical augment of 1. and 2. Under this condition, faithfully showing the influence of robustness caused by image quality, FPR and TPR is necessary.

Again, using the table at the top of the rebuttal (False Positive Rate on original images and Watermark Detection Rate) as an example.

(1) Problems related to TPR/FPR point: without reporting different FPR point for other methods (e.g., StegaStamp of CIN), it is unclear if when FPR=0.062, these methods will have much more robust performance under attack.

(2) Problems related to image quality (generated by attack methods): if G-Noise attack level is doubled, it is unclear if ZoDiac (p=0.9) can maintain 0.996 detection rate. It would be unclear if the performance of ZoDiac will reduce to 0.5; what if StegaStamp can still maintain 1 at the same time? Similar concerns can be raised for all other attacks.

Therefore, extra experiments are necessary to confirm the above cases are not happening and verify the robustness of the proposed method.

I appreciate the authors acknowledgment and promises of adding the experiments that I suggested. Unfortunately, the rating can be only given to what is available in the submitted paper and rebuttal. The change of rating should be subject to timely addressing these concerns.

[1] Sara U, Akter M, Uddin M S. Image quality assessment through FSIM, SSIM, MSE and PSNR—a comparative study. Journal of Computer and Communications, 2019, 7(3): 8-18.

[2] An, Bang, et al. "WAVES: Benchmarking the Robustness of Image Watermarks." Forty-first International Conference on Machine Learning.

We really appreciate your decision and enjoy the discussion with you.

In response to your suggestions, we conducted additional experiments under various attack levels, following the settings in WAVES, and will include these findings in the revised paper. The experiments span multiple attack scenarios:

• Tables 6 and 7: Adjustments in brightness or contrast with a factor of 0.2 to 1.0 (original image).
• Table 8: JPEG compression with a quality setting from 10 to 90
• Table 9: Gaussian blur with a kernel size of 5 to 19.
• Tables 10 and 11: Two VAE-based image compression models, Bmshj18 and Cheng20 with quality levels of 2 to 6.
• Table 12: A stable diffusion-based image regeneration model, Zhao23 with 40 to 200 denoising steps.

We compared ZoDiac and the strongest baseline, StegaStamp, in terms of WDR under different FPR levels, leading to several insights:

• Brightness and Contrast Attacks: As the adjustment factors for brightness and contrast decrease from 0.9 to 0.2, the PSNR of the attacked watermarked images drops significantly—from 37.79 to 11.41 for brightness and from 38.57 to 14.54 for contrast. Despite this degradation, ZoDiac consistently maintains a WDR above 0.99 across all scenarios. Even at extreme settings, where PSNRs fall to 11.41 for brightness and 14.54 for contrast, ZoDiac significantly outperforms StegaStamp, achieving higher WDR and lower FPR. Specifically, ZoDiac records a WDR of 0.994 at an FPR of 0.032 compared to StegaStamp's 0.852 at an FPR of 0.056 for brightness, and a WDR of 0.998 at an FPR of 0.032 compared to StegaStamp's 0.730 at an FPR of 0.056 for contrast. This clearly demonstrates ZoDiac’s robustness even under severe quality degradation.
• JPEG, G-Blur, Bmshj18, and Cheng20 Attacks: With varying attack parameters, including the quality settings for JPEG, Bmshj18, Cheng20, and the kernel size for G-Blur, the PSNR of the attacked watermarked images remains within the range of 25 to 40. In these scenarios, both ZoDiac and StegaStamp consistently sustain a high WDR of around 0.98. The only exception is JPEG with a quality setting of 10, where the PSNR drops to 28.05. Under this condition, the WDR of both ZoDiac and StegaStamp slightly decreases from their usual range of 0.956-0.986 to 0.746-0.834 for ZoDiac, and from 0.996-1.0 to 0.758-0.806 for StegaStamp.
• Zhao23: As the number of image regeneration steps in Zhao23 increases, the PSNR of the attacked watermarked images decreases from 27.15 to 22.44. Despite this degradation, ZoDiac remains highly resistant across most scenarios, with its WDR slightly decreasing from 0.998 to 0.898. In contrast, StegaStamp significantly underperforms, completely failing when the regeneration steps exceed 120.

In summary, the additional ablation study demonstrates that ZoDiac achieves either competitive or superior robustness performance compared with StegaStamp across a wide range of attack scenarios, highlighting its robustness and reliability even under severe conditions.

Table 10. The PSNR of attacked images and the WDR of ZoDiac and StegaStamp under different strengths of the Bmshij18 attack.

Table 11. The PSNR of attacked images and the WDR of ZoDiac and StegaStamp under different strengths of the Cheng20 attack.

Table 12. The PSNR of attacked images and the WDR of ZoDiac and StegaStamp under different strengths of the Zhao23 attack.

Table 6. The PSNR of attacked images and the WDR of ZoDiac and StegaStamp under different strengths of the Brightness attack.

Table 7. The PSNR of attacked images and the WDR of ZoDiac and StegaStamp under different strengths of the Contrast attack.

Table 8. The PSNR of attacked images and the WDR of ZoDiac and StegaStamp under different strengths of the JPEG attack.

Table 9. The PSNR of attacked images and the WDR of ZoDiac and StegaStamp under different strengths of the G-Blur attack.

Thank you for your positive comments and rating! Below we provide one-to-one responses to the three questions.

Q1: Since previous works use different types of watermark, such as StegaStamp uses hyperlink bitstring, there is a concern regarding comparison fairness.

A1: Please refer to the global response to all reviewers. Thank you!

Q2: It will be interesting to see if ZoDiac can work with watermark of different patterns.

A2: The choice of a concentric ring-like pattern for our watermark is influenced by two key factors:
(1) After frequency shifting, the center of the frequency map contains the low-frequency components, which can be slightly modified without significantly impacting the image content. This allows us to embed the watermark while maintaining the visual integrity of the image.
(2) The frequency domain exhibits Hermitian symmetry, where the real part is even symmetric and the imaginary part is odd symmetric. This symmetry makes circularly symmetric watermarks the most straightforward option to align with this natural property. These considerations together led us to select the concentric ring-like watermark pattern. And we are excited about the potential to extend our approach to other patterns. Different patterns may open new possibilities for encoding longer and more meaningful information.

Q3: It will be exciting to see if ZoDiac can work with Transformer-based ones like Pixart-alpha or DiT.

A3: As requested, we have conducted additional experiments using the latest Transformer-based diffusion model, Huanyuan-DiT [1], and present the performance results in Tables 1 and 2 using MS-COCO dataset.
Due to the time limit, the denoising step inside each optimization iteration for ZoDiac with DiT is set to 1 for the fastest watermarking on 50 images, and the SSIM threshold is set to 0.92 for fair comparison. Results from ZoDiac with other backbones are from Figure 9 in the paper and uses 50 iterations. In summary, our method can seamlessly integrate with any stable diffusion backbone and achieve good performance. The experimental results demonstrate the robustness of our approach, regardless of the specific diffusion model being employed.

Table 1. Watermarked image quality of ZoDiac with different backbones.

Table 2. Watermark Detection Rate (WDR) (↑) before and after attacking.

[1] Li, Zhimin, et al. "Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding." arXiv preprint arXiv:2405.08748 (2024).

Thank you for your positive comments and rating! Below we provide one-to-one responses to the two mentioned weaknesses.

W1: While ZoDiac demonstrates excellent robustness under various attacks, as shown in Table 1, it sacrifices image quality, achieving only slightly better visual quality than StegaStamp.

A1: We appreciate your observation regarding the trade-off between image quality and watermark robustness, which is a recognized challenge in watermarking techniques. Indeed, ZoDiac prioritizes watermark robustness, as shown in Table 1, which may result in a slight reduction in image quality. However, this reduction does not cause significant visible degradation, as evidenced by the corresponding watermarked images in Figure 11 in Appendix E. Furthermore, the trade-off curves in Figure 3 and Figures 6 to 8 in the Appendix also demonstrate that ZoDiac can even obtain comparable visual quality while achieving watermark robustness that is still on par with or surpasses existing state-of-the-art methods.

W2: As the authors mentioned, the main weakness of the current method is that it is limited to zero-bit watermarking. How to increase the length of the embedded information is an issue that needs to be addressed.

A2: We agree with you that extending ZoDiac to encode longer information is an exciting and valuable direction for future research. One potential approach is to develop an encoding-decoding mechanism that converts meaningful information into a Gaussian distribution, which can adhere to the distribution constraint in the latent vector of the diffusion model. We are committed to exploring this avenue and making progress in this area.