Can Simple Averaging Defeat Modern Watermarks?

Why Tree-Ring conforms to the additive Simple Linear Assumption model?

1. Addition in spatial domain equates to addition in frequency domain: $x(t)+y(t) \xleftrightarrow{\mathcal{F}} X(j\omega)+Y(j\omega)$.
2. Tree-Ring's pattern added to the initial noise propagates through the generation pipeline to the final image (Figure 6).
3. Thus, Tree-Ring conforms to our model, despite appearing to be a Ring in the frequency domain.

The definition of steganalysis needs to be clearly stated, as it is an important concept in this paper.

We appreciate the suggestion. Steganalysis is the detection of hidden data within cover media. In our context, it's applied as a technique for watermark removal. We'll clarify this definition in the revised paper.

In section 3.1, the paper mentions that ''Instead of applying strong distortions (e.g., noise), the adversary could take a steganalysis approach, first approximating w and then crafting minimal distortions to fool D''. However, in the following method, the watermark is never approximated. Instead, the additive perturbation is directly predicted to fool D.

Your understanding is correct. It was $\delta_w$ we approximated (denoted as $\hat{\delta}_w$), not $w$. We appreciate this observation and we will revise Section 3.1 to accurately reflect our method.

In Figure 2, why are the performance metrics inconsistent? Although the caption specifies the use of AUC, the figure mixes AUC with Bit Acc as metrics for different methods.

Thanks for catching. The caption should be "Performance (AUC or bit accuracy)", not just AUC. We will revise.

It is better to explain why there is a sudden drop when n = 5 for content-adaptive methods like Stable Signature.

• The point at n=5 isn't a drop, but a discontinuity between no removal (leftmost point) and watermark removal (other points).
• The lowest performance at n=5 occurs because small n values lead to imperfect $\delta_w$ approximations, causing larger image modifications and reduced watermark detection. We're happy to provide further clarification if needed."

Why the reported PSNR for Tree-Ring is low (17dB)?

Short Answer: Our method achieves 29.79dB/34.58dB (blackbox/graybox, n=5000) PSNR when removing Tree-Ring watermarks, indicating minimal visible alteration. The 17dB PSNR value mentioned in the paper measures a different aspect.

The 17dB PSNR reported in the paper compares $A$ and $C$, reflecting the cumulative impact of watermarking and removal (Sec 4.1). However, this doesn't represent the degradation caused by watermark removal alone.

To assess removal-induced degradation, we calculate PSNR between $B$ and $C$, yielding 29.79dB/34.58dB (blackbox/graybox, n=5000) for Tree-Ring watermarks.

Tree-Ring, a diffusion-based semantic watermark method, alters image layout significantly. This results in a low PSNR (~16dB) between $A$ and $B$ (Figure 2 NRmv). Consequently, $C$'s PSNR relative to $A$ is also low due to layout differences, despite $C$ closely resembling $B$. This low $A$-$C$ PSNR doesn't accurately reflect the extent of modification during watermark removal.

Comparing perceived degradation like this isn't fair because it's difficult to objectively evaluate the severity of different attacks.

Short Answer: We evaluated both subjectively and objectively. The objective quality evaluation is in Figure 3, which is supplimented by the visualizations (subjective quality evaluation) in your mentioned Figure 9.

Figure 9 supplements Figure 3 in the main text, which already includes objective quality evaluations. We evaluated four metrics: PSNR, SSIM, LPIPS, and SIFID, and plotted the trade-off curves between watermark removal effectiveness (vertical axis) and image quality degradation (horizontal axis) for different distortion methods. Methods closer to the lower-left corner of the graph are better because they achieve significant watermark removal with minimal distortion. Figure 9 visualizes the data points and their parameters used to plot these curves, thus serving as a complement to Figure 3.

The method's requirement for a large number of pre-existing watermarked images poses practical challenges for real-world applications. Attacks such as JPEG compression can directly affect the watermark itself.

• The setting is practical: For example, popular products like Stable Diffusion defaults to adding a fixed watermark to all generated images. Online platforms like Midjourney can add a unique watermark to all images generated by the same user. What's more, these products produce a large number of images everyday. For these popular products, it is possible to obtain tens or hundreds of pre-existing watermarked images.

• Steganalysis requires fewer images than recent works: Many watermark detectors like Tree-Ring have demonstrated robustness against JPEG compression. Recent works have explored adversarial attack-based methods to improve watermark removal. While these methods have successfully removed Tree-Ring watermarks, they require training a surrogate model with 3000 to 7500 images [1, 2]. In comparison, steganalysis requires as few as 500 images for effective removal without significant artifacts, as shown in our experiments.

The contribution of the approach may be somewhat limited.

We argue that our work has made significant contributions. As outlined in our common rebuttal:

• We are the first to introduce a black-box, training-free method that effectively removes and forges Tree-Ring watermarks, which is a SOTA diffusion-based watermarking method.
• We offer a new perspective on understanding watermarks by classifying them and visualizing the patterns of content-agnostic watermarks, providing explainability on why these watermarks are vulnerable.
• We highlight that addressing the content-agnostic issue is a future direction for robust watermarking, as we demonstrated the wide impact of simple steganalysis on several modern detectors, such as Tree-Ring, RAWatermark, and RoSteALS.

References

[1] M. Saberi et al., Robustness of AI-image detectors: Fundamental limits and practical attacks, 2023.

[2] B. An et al., Benchmarking the robustness of image watermarks, 2024.

The method is fundamentally general and not restricted to images. Providing its effectiveness on other modalities, such as audio, would further validate and broaden the impact.

You are correct that the issue extends beyond images. After conducting steganalysis on Audioseal [1] invisible audio watermarks, we found that it also adds content-agnostic watermark that could be removed (<0.76 detection rate). Your suggestion even promotes the significance of our work serving as a reminder of the re-emerging content-agnostic issue.

Why can Tree-Ring change the layout of the generated image?

Adding a Tree-Ring watermark inevitably changes the generation seed. Replacing the low-frequency part of the initial noise with a Ring pattern will cause global changes of the noise pattern, and hence the layout of the generated image will change. However, we have visualized that Tree-Ring also contains a content-agnostic component, and removing this part of the watermark is enough to fool the tree-ring watermark detector, which makes Tree-Ring vulnerable.

The quality of the figures could be further improved.

We will increase the font size in Figure 2-4 to enhance readability, and improve the overall quality of the figures.

References

[1] Robin San Roman et al., Proactive Detection of Voice Cloning with Localized Watermarking, 2024.

Thank you for your response. I would appreciate some further clarifications on the following points:

1. Regarding the setup of audio experiments

Could you specify which dataset you used and whether any preprocessing was applied?

2. On content-adaptive audio watermarks

Have you observed any content-adaptive audio watermarks? Are the conclusions consistent with those on images? Could you share some results?

3. Clarification on Tree-Ring

To confirm, are you suggesting that the tree-ring affects the generated images in both high-level aspects (evident in the layout changes of generated content) and low-level aspects (appearing as a content-agnostic pattern overlaid on the generated images)? If this is the case, I would recommend explicitly stating in the paper that "watermarks added in the latent domain of diffusion models may not always result in a semantic watermark and could remain content-agnostic." Highlighting this risk is crucial for future security of diffusion watermarking.

Thank you for the authors' clarification and extra experiments. My concerns have been addressed. I will raise my score to 8. If possible, please incorporate the extra experiments and discussion into the next version of the paper.

Similar ideas have been explored in existing works [1, 2].

We argue that our work fundamentally differs from [1, 2] with distinctive advantages.

Differences with the spoofing attack in [1]

The spoofing attack described in [1] requires access to the watermark encoder (white-box) and does not generalize to removal, while our steganalysis-based approach unifies removal and forgery without accessing the decoder (black-box). In terms of explainability, our method extends beyond [1] by using steganalysis to extract content-agnostic watermarks. This step enables visualizing to show the existence of the extracted and added patterns during forgery, whereas [1], lacking this step, cannot explain why and which component of the weak overlayed watermark signals enables spoofing. We thus believe the idea explored by [1] is fundamentally different from our work.

Differences with adversarial attacks in [1, 2]

Steganalysis, adversarial, and distortion-based attacks are parallel methods, each fundamentally different from the others. Steganalysis attacks involve extracting hidden information to facilitate removal, adversarial attacks focus on solving optimization problems to harness adversarial examples, and distortion-based attacks typically employ image-processing techniques to create distortions. They are not variations of one another; therefore, we don't think one is a more advanced version of another.

Specifically, steganalysis has advantages over the adversarial attacks described in [1, 2]: our approach can effectively remove watermarks from just 500 images without leaving noticeable artifacts, seamlessly unifying both removal and forgery under whitebox and blackbox scenarios. In contrast, the adversarial attacks in [1, 2] require at least 3000 images for training. The attack in [2] also does not support forgery, as perturbations disrupt the entire latent space. The claim about the potential applicability of AdvCls to content-aware watermarks remains debatable [1, 2], as both work only demonstrated effectiveness in removing Tree-Ring watermarks unless with large perturbations ($\epsilon>10$) [1]. [1] fails on StegaStamp. [2] fails on Stable Signature and StegaStamp.

In conclusion, the uniqueness of steganalysis seamlessly unifies removal and forgery with visualizable explainability under both whitebox and blackbox settings. It requires significantly fewer images than adversarial attacks, further showcasing its advantage over existing approaches.

Steganalysis is not the first blackbox method, as [1] also works under blackbox.

Thanks for pointing out. We will rephrase to clarify that our method is the first that is training-free.

The method lacks novelty, and the vulnerability is trivial.

We argue that our method is novel, and the vulnerability is severe. As outlined in our common rebuttal:

• Novelty
  • We are the first to introduce a blackbox training-free method that effectively removes and forges Tree-Ring watermarks.
  • We are the first to explain what modern content-agnostic watermark patterns look like (such as the Tree-Ring ripples), and why they can be easily removed.
  • We highlight a future direction for robust watermarking: to address the content-agnostic issue by evaluating against steganalysis attacks, similar to how robustness is evaluated against distortions.
• Severity and Impact: Our experiments show that a sufficiently simple and straightforward method easily fools several modern watermark detectors. The vulnerability is therefore severe with wide impact.

Comparison with the watermark benchmark [2].

We compared with attacks in [2] following its Detection setting, reporting TPR@0.1%FPR (low_1000 in its code). The comparison is on Tree-Ring watermarks, which is the watermark that could be successfully removed by both [2] and our work.

As can be seen, our removal lowers TPR@0.1%FPR to near 0, more effective than any attacks in [2].

Include forgery results for other watermarks to provide a more comprehensive evaluation.

• RoSteALS (ROS): highly effective (0.9+ bit acc), suggesting RoSteALS adds linearly content-agnostic watermarks
• RAWatermark (RAW): moderately effective under graybox (0.72 AUC) but behave like removal under blackbox (0.25 AUC)
• DwtDctSvd (SVD): ineffective forgery, suggesting the effective removal could be due to $\delta_w$ disrupting the decoding space (similar to AdvCls-WM1&WM2 in [2])

References

[1] M. Saberi et al., Robustness of AI-image detectors: Fundamental limits and practical attacks, 2023.

[2] B. An et al., Benchmarking the robustness of image watermarks, 2024.

The spoofing attack described in [1] requires access to the watermark encoder (white-box) and does not generalize to removal, while our steganalysis-based approach unifies removal and forgery without accessing the decoder (black-box).

I understand the point of authors about their attack being black-box, in contrast to the spoofing attack from [1]. However, the spoofing attack is very similar to the steganalysis attack from this paper, and while [1] does not discuss on generalization of the attack on watermark removal, I don't see a reason why it wouldn't work.

It requires significantly fewer images than adversarial attacks, further showcasing its advantage over existing approaches.

Reading the authors' response, I accept that their method, while having similarities to the adversarial attack, has its own advantages and disadvantages (only working on content-unaware watermarks).

I still believe that the vulnerability of content-unaware watermarks to this attack is trivial (especially in case of the gray-box setting). However, for the black-box setting, where non-paired samples for watermarked and non-watermarked images are used, there is more novelty and contribution. I appreciate the authors for adding new results for comparison to the attacks from [2], and the forgery results. I will increase my score to 5 (borderline accept).

[1] M. Saberi et al., Robustness of AI-image detectors: Fundamental limits and practical attacks, 2023.

[2] B. An et al., Benchmarking the robustness of image watermarks, 2024.

Dear Reviewers,

Thank you all for engaging so thoughtfully with our work during the discussion period. We greatly appreciate your constructive feedback. We're also grateful that most of you considered raising the scores. We wish you all the best in your research and future career!

Best regards, #9663 authors