Shallow Diffuse: Robust and Invisible Watermarking through Low-Dimensional Subspaces in Diffusion Models

Q1: "The attack experiments are limited, consisting of only four fixed-parameter attacks, which do not demonstrate the method's robustness. For instance, the method can be viewed as a variant of treering, could experiments with additional attacks, such as rotation、regeneration be included?"

A1: We have added 7 additional adversarial attacks, please see Q1 in the global response.

Q2: "The theoretical assumptions of the method are built upon [1], but the experimental results yield a different range of t values compared to the theoretical analysis in [1]. Although this can be explained by the errors introduced by DDIM-Inv, it remains perplexing."

A2: There is another important factor contributing to this gap. The previous results in LOCO-Edit [2] only evaluate rank in the image space of unconditional diffusion models. However, most of our experiments are conducted on latent diffusion models such as Stable Diffusion. For these models, the minimum rank may be achieved at a different timestep compared to vanilla diffusion models. However, we couldn’t reproduce the experiment for Stable Diffusion because calculating the Jacobian matrix is computationally infeasible due to its huge size (3 * 256 * 256 x 3 * 256 * 256).

In the revision, we have included a discussion this factor in the experimental analysis.

Q3: "The method relies on the properties of DDIM and DDIM-inverse, which may lack certain generalizability. It might not perform well for attacks executed in the latent space."

We believe there might be some misunderstandings of our result. In our experiments, we do apply our Shallow Diffuse in the latent space for Stable Diffusion, see line 402. Additionally, we have added ablation studies on different sampling methods, including DDIM, DEIS [3], DPM-Solver [4], PNDM [5], and UniPC [6]. See Appendix C.6 for more details. In short, all these samplers have very similar image generation quality and robustness, demonstrating the generalizability of our approach.

Q4: Typos in the equation.

A4: We have fixed and highlighted it in the revised manuscript.

Q5: Could you specify the parameters used in the Shallow Diffuse method in Section 5, such as the embedding channels and watermark radius?

A5: We have added and highlighted it in the revised manuscript in section C.5 and line 269.

Q6: The experiments in Appendix C only provide results, could you include some analysis?

A6: We have added discussions and highlighted them in Appendix C in the revised manuscript.

[2] Chen, Siyi, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu. "Exploring low-dimensional subspaces in diffusion models for controllable image editing." NeurIPS 2024

[3] Zhang, Qinsheng, and Yongxin Chen. "Fast sampling of diffusion models with exponential integrator." ICLR 2023.

[4] Lu, Cheng, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. "Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps." Advances in Neural Information Processing Systems 35 (2022): 5775-5787.

[5] Liu, Luping, Yi Ren, Zhijie Lin, and Zhou Zhao. "Pseudo numerical methods for diffusion models on manifolds." ICLR 2022.

[6] Zhao, Wenliang, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. "Unipc: A unified predictor-corrector framework for fast sampling of diffusion models." Advances in Neural Information Processing Systems 36 (2024).

Q1: "Besides Tree-Ring and Stable Signature, I think there are more existing watermarking methods that introduce small perturbation to the image to embed watermark, like StegaStamp. To show the proposed method actually preserves image quality, I think the authors should compare the method with some watermarking methods that watermark the image after image generation with a small perturbation."

A1: We have included comparisons with StegaStamp in both user and server scenarios, as detailed in Tables 1 - 4. In summary, the generation consistency of Shallow Diffuse is comparable to that of StegaStamp, as shown in Table 2. However, Shallow Diffuse demonstrates significantly better robustness, particularly against diffusion-based attacks such as DiffPure and IR, as highlighted in Tables 1 - 4.

Q2: "In the robustness part, I think the regeneration attacks and some adversarial perturbation should be evaluated on the proposed method to see whether the proposed method is actually robust under various attacks."

A2: We have added 7 more adversarial attacks, please see Q1 in the global response.

Q3: "Since the authors mention the user scenario, if multiple users rewatermark the same image with the proposed method, can the watermark embeded by the specific user be detected in this circumstance?"

A3: We have added experiments for multi-key identification. please see Q2 in the response to all reviewers and ACs.

Q4: "Compared to Tree-Ring, the technical contribution of the proposed method is limited."

A4: Although our method is developed based on the Tree-Ring framework, there are several significant contributions over Tree-Ring that we want to highlight below:

1. Identifying and addressing fundamental limits of Tree-Ring. Our study revealed that the Jacobian of the posterior mean estimator is full-rank at high-noise timesteps, leading to inherent image distortion when injecting watermarks using Tree-Ring. Our work addressed this limitation. We inject the watermark at shallow timesteps, where the Jacobian is low-rank with a large null space. This ensures most of the watermark's energy resides in the null space, minimizing distortion in image generation.

2. Improved Watermarking Techniques. In Section 3.2, we inject the watermark into the low-frequency region, whereas Tree-Ring uses the high-frequency region. Additionally, in Appendix B, we introduce the concept of channel averaging. With these technical improvements, Shallow Diffuse achieves enhanced robustness and consistency.

3. Theoretical justifications. Tree-Ring is purely an empirical method lacking theoretical justification. In contrast, our approach is backed by rigorous theoretical guarantees of detectability and robustness under appropriate assumptions. This foundation enhances the interpretability and trustworthiness of our method.

[1] Chen, Siyi, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu. "Exploring low-dimensional subspaces in diffusion models for controllable image editing." arXiv preprint arXiv:2409.02374 (2024).

Q1: Based on the new Table 1, it is evident that the generation quality of StageStamp significantly outperforms the proposed method, as indicated by the CLIP score (0.355 vs. 0.328). This raises a critical question: does the improved robustness of the proposed method come solely as a trade-off for lower generation quality?

A1: Thank you for the insightful comments. We address two aspects of your question as follows.

1. CLIP score is not a good indicator of generation consistency. For watermarking applications, our primary objective is to ensure consistency between watermarked and original images rather than generation quality—a property that the CLIP score does not effectively capture. Figure 9 illustrates this by comparing original Stable Diffusion images with watermarked versions generated by StageStamp and Shallow Diffuse. While StageStamp introduces noticeable visual artifacts, Shallow Diffuse produces cleaner, more visually consistent outputs. This discrepancy between visual quality and the CLIP score highlights the limitations of the CLIP metric, which is inherently biased due to its sensitivity to text embeddings [1].

2. Our method improves both robustness and generation consistency over StageStamp. Furthermore, Table 1 demonstrates that Shallow Diffuse achieves CLIP scores and FID values closer to those of the original Stable Diffusion (first and last rows of Table 1). This alignment implies that Shallow Diffuse better preserves the original image characteristics compared to StageStamp, which, despite achieving higher CLIP scores, introduces undesirable distortions for watermarking applications. Additionally, as demonstrated in Table 2, metrics better suited for assessing generation consistency—such as LPIPS, SSIM, and PSNR—indicate that Shallow Diffuse performs comparably to StageStamp.

To fully evaluate whether there exists a trade-off between these factors, we also conducted additional experiments comparing our approach with existing methods. As shown in Figure 4, under nearly identical robustness conditions, Shallow Diffuse outperforms others in terms of generation consistency. This demonstrates that our method achieves simultaneous improvements in both robustness and consistency, without compromising one for the other.

[1] Ahmadi, Saba, and Aishwarya Agrawal. "An examination of the robustness of reference-free image captioning evaluation metrics." Findings of the Association for Computational Linguistics: EACL 2024, pages 196–208.

Q3: In the new experiment, the authors stated that they utilized non-overlapping masks for multi-user rewatermarking and assumed the ability to predefine the number of keys and non-overlapping masks. However, this evaluation is limited to the scenario of two users. In real-world applications, as the number of users increases, how significantly would the identification accuracy and robustness of the proposed method degrade? Furthermore, if the number of users surpasses the predefined limit, does this imply that no additional users can be integrated into the watermarking system?

A3: Thank you for the thoughtful question. We extended our experiments to include 4, 8, 16, and 32 users and compared the results with Tree-Ring. The results are presented in Table 6, and we’ve summarized the table below.

Shallow Diffuse consistently outperformed Tree-Ring in robustness across different numbers of users. Even as the number of users increased to 32, Shallow Diffuse maintained strong robustness under clean conditions. However, in adversarial settings, its robustness began to decline when the number of users exceeded 16. Under the current setup, when the number of users surpasses the predefined limit, our method becomes less robust and accurate.

We believe that enabling watermarking for hundreds or even thousands of users simultaneously is a challenging yet promising future direction for Shallow Diffuse.

| Watermark number | Method          | Clean     | Adversarial average |
|------------------|-----------------|-----------|:-------------------:|
| 2                | Tree-Ring       | 1.00/1.00 | 0.98/0.80           |
| 2                | Shallow Diffuse | 1.00/1.00 | 0.99/0.95           |
| 4                | Tree-Ring       | 1.00/1.00 | 0.96/0.70           |
| 4                | Shallow Diffuse | 1.00/1.00 | 0.99/0.86           |
| 8                | Tree-Ring       | 1.00/0.95 | 0.91/0.47           |
| 8                | Shallow Diffuse | 1.00/1.00 | 0.98/0.80           |
| 16               | Tree-Ring       | 0.96/0.57 | 0.83/0.26           |
| 16               | Shallow Diffuse | 1.00/0.89 | 0.92/0.56           |
| 32               | Tree-Ring       | 0.95/0.44 | 0.80/0.16           |
| 32               | Shallow Diffuse | 0.99/0.89 | 0.90/0.44           |

Q1: "The table in the paper is not very well drawn, it is very difficult to read, especially the header. At the same time, the experimental part is not detailed enough. For example, should the comparison method reproduce the results or use the pre-training model?"

A1: Thank you for your feedback. We have improved the presentation, as outlined in Q3 of our global response. Regarding the comparison setup, we highlight this in lines 401–405. In summary, for the server scenario, all diffusion-based methods are evaluated using the same model, Stable Diffusion 2.1. Non-diffusion methods are applied to images generated by Stable Diffusion 2.1. We control the initial seeds so that the non-diffusion methods use the same set of images as the diffusion-based methods.

Q2: "In Table 1, for the CLIP-Score index, yours is 0.3285, which seems to be the worst. Please explain further."

A2: The CLIP score focuses solely on image quality. For watermarking, however, our priority is the consistency between the watermarked and original images, which the CLIP score fails to capture.

Table 1 shows that the CLIP score and FID achieved by Shallow Diffuse are closest to those of Stable Diffusion without watermarking (see the first and last rows of Table 1). This suggests that images generated by Shallow Diffuse maintain greater consistency with those of the original Stable Diffusion model. In contrast, methods like Tree-Ring Watermarks, while achieving higher CLIP scores, significantly distort the images, which is undesirable. Figure 1 further illustrates this, showing how Tree-Ring Watermarks introduce a bias toward the inserted key.

Q3: "Please explain why the filter size in the Gaussian blurring is 8 × 8 and how the standard deviation is selected."

A3: We apply the same experiment setting as our baseline method Tree-Ring and RingID.

Q4: "As can be seen from Table 2, the PSNR and SSIM of most methods are very low, so it is easy for human eyes to find modification traces, which easily leads to the risk of watermarked images being maliciously broken. Please further explain the visual quality of the generated watermarked image."

A4: We have included additional qualitative comparisons with non-diffusion-based methods in Figure 8. From the results, it is challenging to visually distinguish our method from these non-diffusion-based approaches and clean images. Furthermore, as demonstrated in Figure 1, our method achieves significantly greater visual consistency with the original image compared to Tree-Ring and RingID.

Q1: The presentation of this paper is poor, for instance, the ablation studies (Appendix C) and index of experimental results (Table 4) are incomplete. Therefore, this leads to a shortage of critical ablation studies.

A1: We have improved the presentation in the revised paper. Specifically, we have included discussions on each ablation study in Appendix C, and improved the readability of the tables as suggested. See Q3 of our global response.

Q2: What is the performance of multi-key identification, specifically, is it possible for the Shallow Diffuse to inject multiple watermarks and distinguish between them?

A2: We have added experiments for multi-key watermarking. Please see Q2 of our global response.

Q3: "The image distortions are less than that in previous studies, such as Tree-Ring, where they apply 6 distortions". "Can DiffPure purify the watermarked patterns?"

A3: We have added 7 more adversarial attacks, please see Q1 in our global response to all reviewers. Specifically, we have chosen DiffPure [1] as a specific attack. Shallow Diffuse achieves 1.00 TPR@1%FPR at the server scenario and 0.86 (COCO), 0.9 (DiffusionDB), 1.0 (WikiArt) TPR@1%FPR at the user scenario. Thus, DiffPure is hard to purify the watermarked patterns from Shallow Diffuse.

Q4: "The findings in Table 4 are confusing. It appears that employing channel averaging enhances robustness against image distortions. However, channel averaging involves averaging clean and watermarked images across specific channels. As far as I understand, this process might reduce the robustness of the watermark. Can you explain this observation?"

A4: We do not directly average the clean images and watermarked images in our approach. Instead, we embed the watermark into a single channel while averaging the non-watermarked channels. This design leverages the observation that many image processing operations, such as color jittering or Gaussian blurring, tend to affect all channels uniformly. By isolating the watermark in a single channel, it becomes less susceptible to these transformations. Consequently, channel averaging improves robustness against certain attacks.

[1] Nie, Weili, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. "Diffusion models for adversarial purification." arXiv preprint arXiv:2205.07460 (2022).