DiffusionShield: A Watermark for Data Copyright Protection against Generative Diffusion Models

Thank you very much for the valuable feedback. We hope the following responses can help address your concerns.

Weakness 1 & Question 1: Improvement on the layout by incorporating the analysis on the watermark rate.

Response: Thanks for your suggestion about the paper organization. We updated the main text of Section 4.2 to incorporate the discussion on the influence of watermark rate. In the main text, we point out that, although the low watermark rate might hurt the performance, this problem is mitigated in the multi-user case. Even though the watermark rate for each user is low and they encode different messages and do not share the pattern uniformity, our method can still perform well. Then we refer the readers to read more details about the experiment in Appendix F. The updated part is marked blue in the revision for your convenience.

Weakness 2 & Question 2: More state-of-the-art deep learning-based watermarking models should be considered as the baselines for comparison.

Response: Thanks for your suggestion regarding the inclusion of new baselines. In response, we have expanded our comparative analysis to include three additional baselines: IGA[1], MBRS[2], and CIN[3]. The results are reported in Table 15 below. We can see that the performance of IGA is very similar to HiDDeN and DFD. Although IGA’s bit accuracy is comparable to our DiffusionShield, it demands a significantly higher budget—nearly 10 times the Linf and LPIPS values of our approach. As for MBRS and CIN, despite having budgets similar to DiffusionShield, their bit accuracies are worse than ours, especially on CIFAR100, where MBRS achieves only 87.68% and CIN is 51.13% in bit accuracy. In contrast, DiffusionShield maintains the bit accuracy of 99.80% with a comparable budget. This is because of the higher pattern uniformity of DiffusionShield. In summary, the performances of the new baselines are similar to the previous baseline methods. They either compromise the budget for bit accuracy close to ours, or fail to be reproduced well in the generated images.

[Table 15]

[1] Zhang, Honglei, et al. Robust data hiding using inverse gradient attention.

[2] Jia et al. Mbrs: Enhancing robustness of dnn-based watermarking by mini-batch of real and simulated jpeg compression, ACM MM '21

[3] Ma et al. Towards Blind Watermarking: Combining Invertible and Non-invertible Mechanisms, ACM MM '22

Weakness 3 & Question 3: More comprehensive experiments to further substantiate the effectiveness of the proposed method across multiple real-world scenarios, especially when the protected images are from various classes instead of one class. Response: Thanks for your suggestions on the implementation details. In response to your concerns, we have added one set of experiments using CIFAR-100. In the following Table 16, we have considered that the protected images are from various classes (5, 10 or 15 classes randomly selected in CIFAR-100) and added watermarks to those protected classes. The bit accuracy of the watermark on the generated images from the protected class are shown in the table. It reveals that the watermarks are predominantly detectable in the images across various classes, with limited exceptions noted in scenarios where the watermark budget is minimal (1/255) and only 5 classes are watermarked.

[Table 16]

We are grateful for the useful comments provided by you. We hope that our answers have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

Regrading the results on STL10 and ImageNet-20 datasets:

In response to your concerns, we have also conducted the experiments on STL10 and ImageNet-20 datasets. Given that training a diffusion model on images with higher resolutions requires a very high computational cost, it takes much longer time to get the results. We will update the results in the latest version of the paper once we get them.

Based on the previous results on images with size from 32x32 to 256x256, we can deduce certain trends regarding performance changes of watermarking methods as image resolution increases. We summarize the trend of the change in the bit accuracy of our method on different datasets below. From the table below we can see that when the size of the images is increased across different datasets, there is no significant change in the bit accuracy in our method (4/255). But the baselines (including CIN and MBRS) reduce in a demanding scenario. Therefore, although we haven’t got the results for the experiments with higher resolutions of STL10 and ImageNet20, we believe that following this trend, when the image resolution continues to increase like STL10 and ImageNet (which is usually a more demanding scenario), our DiffusionShield is very likely to keep a higher performance than MBRS and CIN.

Thanks for providing your insightful feedback. We hope that our following answers can address your concerns.

Regarding the performance on CIFAR10:

• 1) Our method does not outperform MBRS and CIN in terms of image quality.

  Thanks for pointing out the concerns. We note that there is a typo about LPIPS budget of our method for CIFAR-10 in the last response. It should be 0.00120 instead of 0.00720, which is actually smaller than the baselines. We have fixed the typo in the following table. In addition, considering that our method provides a flexibility to control the budget of the watermark, we further demonstrate the performance of our methods considering lower L-inf budget, 2/255 and 1/255 in the table below. The new results, presented in the table below, indicate that our method maintains comparable bit accuracy with the baseline methods even at much reduced budgets. Notably, with an L-inf budget of 1/255, our method achieves a bit accuracy of 99.90%, while both the L_2 and LPIPS metrics are significantly lower than those of the baseline methods. These findings underscore the effectiveness of our approach when working with smaller budgets.

• 2) Uniformity doesn't seem to bring the positive correlation between the watermark detection rate and the pattern uniformity.

  The effectiveness of the pattern uniformity is more outstanding when the budget is lower and the scenario is more demanding (e.g. when watermark rate is low). The scenario considered in the experiments on CIFAR10 is relatively easier given that the watermark rate (10%) is relatively high, making the watermarks easier to be learned by the GDM and reproduced in the generated images. That is the reason why the baseline methods can also achieve good performance on the CIFAR10 dataset. But we can see the superiority of pattern uniformity in reaching a lower budget in CIFAR10 from the point 1 above. Also, in the following discussion regarding CIFAR100 we will show that if the scenario is more demanding, the importance of the pattern uniformity will be more obvious.

Regarding the performance on CIFAR100 dataset:

• 1) Why is the performance of CIN peculiar in CIFAR100?

  As we discussed, the reproduction of the watermark is related to factors including pattern uniformity, budget and watermark rate. In our experiments, we consider adding watermarks to the images from one certain class. Thus, the watermark rate for CIFAR100 (1%) is much lower compared with CIFAR10 (10%). Therefore, CIFAR-100 is a more difficult dataset. Both FRQ and CIN fail to protect in this dataset and have only an accuracy similar to random guess result. They do not have very extremely large budget like HiDDeN and do not have good pattern uniformity than DiffusionShield. That is the reason why CIN suffers from an obvious degradation from CIFAR10 to CIFAR100 taking the three factors (watermark rate, budget and pattern uniformity) into consideration.

• 2) The superiority of our method in CIFAR100 stems from the high pattern uniformity.

  Given the inherent complexity of CIFAR100 and the fact that the watermarking rate is limited to just 1%, it becomes significantly challenging for the watermark to be replicated accurately in the generated images. In more demanding scenarios, the advantage of our highly uniform watermark pattern becomes more evident. That is the reason why our method demonstrates a very good performance in CIFAR100. This further highlights the necessity of high pattern uniformity, particularly in practical applications where complex data sets and low watermark rate are involved.

Question 2: The feature distribution of the watermarked images.

Response: Thanks for your suggestion regarding the feature distribution of watermarked images. To address this, we employed a Resnet18 model pre-trained by Contrastive Learning as our feature extractor to check the change of feature distribution after watermarking. We visualize the feature space by t-SNE with the updated results now included in Figure 15 in Appendix J. In this visualization, we plot three clusters: the clean bird class (marked as red dots), the bird class watermarked by DiffusionShield with a budget of 8/255 (represented by blue “+” symbols), and the clean truck class (indicated by green dots). From the figure we can see that, watermarked bird class exhibits negligible change compared to the clean bird class. Each sample in clean bird class shares the same coordinate with a watermarked bird sample. This indicates that the watermark has no influence on the feature space. For comparison, we also included the truck class in our visualization, which demonstrates a distinctly different distribution compared to the two bird classes. In summary, our watermark has almost no influence on not only the visibility but also the underlying feature distribution.

Thanks for your positive comments. Please find our response below for your questions.

Weakness 1: The author can provide more examples/analysis of adding a watermark to latent space and how it will affect the VAE encoding/decoding process.

Response: Thanks for your insightful query regarding the influence of watermark in latent space. To address this, we compared the budget and visualized the change of latent space in updated Appendix J. As shown in the below Table 13, in latent space, the budget of baselines including both DFD and HiDDeN are much larger than our method. Even though the baseline methods have such large budgets, we know that, from Table 3 in Section 4.4, their performance of protection is still worse than ours. (For your convenience, we also repeat Table 3 below. ) Also, we visualize the change of latent space after watermarking in Appendix J and find that the change of DFD and HiDDeN is much more obvious than ours. Regarding the VAE, after we decode from the watermarked latent space, the budget is still small and the watermark is imperceptible in pixel space. This means that our watermark has a much smaller influence on the decoding process compared with baseline methods.

[Table 13]

Weakness 2: The author can provide the cost of watermarking the images and it may not be likely that a content creator has the resources to watermark the images.

Response: Thanks for your helpful question. The training process of watermark patches and classifiers requires about 10 hours on a single V100 GPU card. For content creators, even if they have no access to GPU resources for training, they can reuse the off-the-shelf basic patches and classifier to protect their artworks. This is one of our advantages, flexibility. As we mentioned in Section 3.1, 3.3 and Experiment in Section 4.3, once the watermarks are obtained, DiffusionShield does not require retraining when there is an influx of new users and images. Therefore, a third-party agency can train these patches and classifiers and provide the service. The content creators can entrust the third-party agency to watermark and validate it.

Weakness 3 & Question 1: Analysis on how DiffusionShield can protect against the removal attack

Response: Thanks for your question. In Table 4 in Section 4.6, we have conducted experiments to demonstrate the robustness of our method against various forms of image corruption including Gaussian noise, low-pass filtering, grayscale, and JPEG compression. In order to further clarify your concern about the robustness, we have expanded the scope of our experiments to include image resizing and a deep-learning-based watermark removal method [1]. These tests are conducted using the CIFAR-10 dataset. For the resizing experiments, we altered image sizes from 32x32 to 64x64 (termed "large" in the table), or from 32x32 to 16x16 (termed "small" in the table). During detection, we resize all the data back to 32x32 before inputting them to the detector. In order to improve the robustness, we have incorporated the corresponding disturbances as an augmentation in the training of the watermark. All the results related to robustness are put together in the following Table 14. From the results we can see that DiffusionShield consistently achieves a high bit accuracy and outperforms the two baseline methods in all scenarios except under the Gaussian Noise corruption. In summary, the results from these experiments indicate that our DiffuisonShield method is able to maintain robustness against multiple disturbances.

[Table 14]

[1] Invisible Image Watermarks Are Provably Removable Using Generative AI. Zhao et. al., 2023

We are grateful for the useful comments provided by you. We hope that our answers have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

I have carefully read the authors responses and they have addressed my concerns. I also appreciate the authors effort. I am staying at my rating due to the reason I stated in the strengths.

Thanks for your valuable comments. Please find our response below for your questions.

Weakness 1: Details of parameter settings are best placed inside the main paper. The missing details of the implementation of baseline HiDDeN (whether it has the noise layers) may cause misunderstanding about the fairness of experiment settings.

Response: Thanks for your helpful suggestion. Following your suggestions, we have placed the details of parameters settings in Section 4.1 in the main paper. Regarding the implementation of HiDDeN, our settings are as fair as you expected. Specifically, in our experiments, we did not include the noise layers in HiDDeN except when testing for robustness against noise. This means for all experiments other than those reported in Table 4 of Section 4.6, HiDDeN was used without its noise layer. However, for experiments focused on testing robustness, we did incorporate a noise layer in HiDDeN to maintain fairness. We have updated the details of this baseline setting in Appendix C4 and marked the update part as blue color in the revision for your convenience. We hope this update clears up any confusion regarding the baseline settings in our experiments.

Weakness 2: Experiments on the watermark's robustness against possible attack such as JPEG compression and resize are necessary.

Response: Thanks for your suggestions about the robustness analysis. In Table 4 of Section 4.6, we have conducted experiments to demonstrate our robustness against attack including JPEG compression, Gaussian noise, low-pass filter and grayscale. To further clarify your concern about the robustness, we add more experiments about resize and a deep-learning-based watermark removal method [1]. In the following Table 12, we merge all the results including Table 4 for comparison. These tests are conducted using the CIFAR-10 dataset. For the resizing experiments, we altered image sizes from 32x32 to 64x64 (termed "large" in the table), or from 32x32 to 16x16 (termed "small" in the table). During detection, we resize all the data back to 32x32 before inputting them to the detector. In order to improve the robustness, we have incorporated the corresponding disturbances as an augmentation in the training of the watermark. Notably, DiffusionSheild maintains bit accuracy above 94% and outperforms the two baseline methods in all scenarios except under the Gaussian Noise corruption. In comparison, DFD suffers from a significant reduction in Gaussian noise, greyscale and JPEG compression, and HiDDeN shows a poor performance under low-pass filter, resize, watermark removal attack, and JPEG Compression. In summary, the results from these experiments indicate that our DiffuisonShield method is able to maintain robustness against multiple disturbances.

[1] Invisible Image Watermarks Are Provably Removable Using Generative AI. Zhao et. al., 2023

[Table 12]

Weakness 3: Experiments on high-resolution images are necessary.

Response: Thanks for your suggestion. The experiments of high-resolution images are reasonable and necessary and we do have ImageNet in Table 1 in Section 4.2 and pokemon-blip-captions in Table 3 in Section 4.4. ImageNet is in shape 256x256, and pokemon-blip-captions is in 512x512. These datasets have similar resolution as the dataset in the suggested paper [1], and they are also similar to most of Diffusion Model papers, like [2] [3], whose experiments are mainly conducted with CIFAR-10 in 32x32, ImageNet in 64x64 and LSUN in 256x256, and Latent Diffusion papers like [4], whose experiments are mainly conducted using image with size 512x512 and 256x256. We hope this can clarify your concern about the image resolution.

[1] Guo et al. Practical deep dispersed watermarking with synchronization and fusion, ACM MM'23

[2] Ho et al. Denoising Diffusion Probabilistic Models, NeurIPS’ 20

[3] Nichol et al. Improved Denoising Diffusion Probabilistic Models, ICML’ 21

[4] Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models, CVPR’ 22

Question 1: It seems that each 4x4 basic block can represent 2 bits (if B=4), so how to embed 128bit in CIFAR-10? The message should be (28/4)x(28/4)x2bit=98bit?

Response: Thanks for your question. The resolution of CIFAR10 and CIFAR100 [1] is 32x32. Thus, the message length for CIFAR should be (32/4)x(32/4)x2bit=128bit. We updated this calculation process in the Appendix C.2 for readers’ convenience.

[1] Krizhevsky et al. Learning multiple layers of features from tiny images. (2009): 7.

We are grateful for the useful comments provided by you. We hope that our answers have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

In real world, an image will under many geometric attacks. I notice that you claime "image protectors who own the protected images are aware of the original size of the protected images".
First of all I don't agree with you on this, it is very resource consuming to store his original information for every image. Secondly, even if the owner of the image knows the original size, what if the image has been cropped?

Therefore, your reply still cannot convince me. Based on your inadequate thinking, I'm going to decrease my score.

I'm sorry that I'am still confuse about your reply. Following is my question:

Further discussion about the first question:

"If leftmost 32x4 region is cropped, that leaves only (28/4)x(32/4)*2=112bit. In this case, how to decode 128bit?" Could answner this question? In this case, can only 112bits be extracted?

Further discussion about the second question:

• The purpose of the watermark detector is to decode the watermark, not to detect whether the image has a watermark, right? If so, I suggest to replace "detector" with "decoder", which will be more clarity.
• I still confuse about the 10 classes. If the decoded bit is "1", what the result of the model? and If the decoded bit is "0", what the result of the model? and If the image is without watermark, what the result of the model? Could you provide an example?
• How to understand "and then is classified into $b_{1,m}$ in a patch-by-patch manner." (in page 5). Does it mean the input size of your watermark detector is $8u\times8v$, and the output size of your watermark detector is 64bit? Or does it mean the input size of your watermark detector is $u\times v$, and the output size of your watermark detector is 1bit, and you need to decode 64 $u\times v$ blocks to get 64bit?

About the response Regarding the concern related to storing information for resizing:

• I don't agree "resizing the image back to the original size is a commonly adopted and practical setting in the watermarking process"
• I don't agree "we note that all contemporary watermarking techniques (like the baseline methods DFD[1] and HiDDeN[2]) rely on a consistent image size for effective decoding".

I hardly see any recent work that would require an advance resizing operation in the face of a resize attack. If you continue to insist, please provide the corresponding reference.

About the response Regarding the concern related to the cropping and resizing:

• How would you define a slight crop? Please provide a specific value for cropping as described in [1].
• Please focus on more recent work as we are now approaching the end of the year 2023. "the performance of all watermarked methods (like the baseline methods DFD[1] and HiDDeN[2]) may decrease since they all use a detector with a fixed size"
• "If the images are cropped and resized, the blocks are also cropped and resized which is the reason for potential performance drop. Therefore, we can add crop-resize augmentation to train our block detectors, which can further increase the robustness to cropping." Please provide experimental results to support your claim.

[1] Guo et al. practical deep dispersed watermarking with synchronization and fusion, ACM MM'23

Thanks for your insightful and detailed comments. We hope the following responses can help address your concerns.

Weakness 1 & Question 1: The protection scenario involves considering a class-conditional model or using a classifier to filter the generated images that require protection in unconditional generation, which may be not practical. The effectiveness of the classifier is not demonstrated.

Response: Thanks for your question. We want to clarify that both the class-condition and the classifier in the unconditional case are only used for experimental evaluation. We use them to test the performance of our watermark but do not rely on the classifier or class-condition in real-world scenarios. Below we discuss two viable examples of using the watermark to reveal the infringement in practice:

• First example: Assuming that the unconditional model by data offender generates watermarked and unwatermarked images and the offender releases them. To find the infringement, the data owner does not need to first find the images that are similar to their artworks before watermark detection. They can just use the watermark detector to detect from all the images (both watermark-protected and unwatermarked). Then the watermarked images can be effectively selected. The data owner can choose to engage some human labor to further make sure that the watermarked images are copied from their owned artworks like similar art style and similar object design. Both the watermark and the copied similar things can work as the evidence to reveal and accuse the infringement.
• Second example: The data owner notices some generated images are very similar to their artwork. They doubt that the images come from generated models trained with their protected watermarked protected images. They use the watermark detector to detect the watermarks in the suspect images. If the watermark is detected, this provides the evidence to reveal and accuse the infringement and then protect their copyright.

Besides, we also demonstrate the effectiveness of our classifier in evaluation to show the correctness of our experimental evaluation. We show TPR (True Positive Rate) and FPR (False Positive Rate) of the classifiers in the following Table 9. Here, TPR reflects the classifier's accuracy in correctly labeling generated images from protected classes as protected. Conversely, FPR indicates the proportion of generated images that, although not from protected classes, were mistakenly identified as protected by the classifier. We can see that generally the FPR is very low, meaning that the bit accuracy of unconditionally generated images is mainly measured on the images from the protected classes, thus the results related to the unconditional model are valid.

In summary, in practical applications, we do not need such classifiers and we only use them for experimental evaluation. The demonstrated ability of classifiers shows that our experimental results are valid. We hope this can clarify your concern about practicality.

[Table 9]

Weakness 2 & Question 2: The paper does not include an examination of the method's false positive rate and AUROC.

Response: Thank you for the suggestion to include FPR and AUROC in our watermark detection accuracy assessment. In the following Table 10, we show the TPR, FPR and AUROC of our DiffusionShield with CIFAR-10 dataset. The threshold for the TPR and FPR is 90%, which means that, if more than or equal to 90% of the detected message match the message encoded to the watermarks on the training data, then the image will be considered as an image generated by a model which was trained on the protected data. From the table we can see that our DiffusionShield has a perfect performance in terms of TPR and FPR, which demonstrates DiffusionShield's powerful effectiveness in safeguarding image copyrights.

[Table 10]

Weakness 3 & Question 3: Provide an analysis on the robustness against recent attacks known to compromise imperceptible watermarks [1][2] & More details about the Gaussian Noise added to the images in the experiments in Table 4.

Response: Thank you for your suggestions on the analysis of the robustness of watermarks. In Table 11 we demonstrate the bit accuracy and AUROC of our DiffusionShield together with two baselines methods under the context of watermark removal attempts by [2]. The results show that DiffusionShield is able to survive and achieve a high bit accuracy and AUROC in the experiments on CIFAR-10 and Pokemon. In comparison, the performance of HiDDeN is much worse. Although generally DFD performs well in defending against the removal method, we note that it requires a much higher budget compared to our method, and larger budgets are more difficult to remove. Regarding the experiments in Table 4, the standard deviation of the Gaussian Noise is 0.1 and its effect on the images can be visualized in Figure 13 in Appendix J. From the figure we can see that the Gaussian noise considered in Table 4 has a great influence on the quality of images. Even with such an influence on the images, DiffusionShield can still achieve bit accuracy higher than 80%, and AUROC=1, demonstrating the robustness of DiffusionShield against Gaussian Noise.

[Table 11]

[1] Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks. Saberi et. al., 2023 [2] Invisible Image Watermarks Are Provably Removable Using Generative AI. Zhao et. al., 2023

Weakness 4: Employing the same watermarking pattern for every image simplifies the learning process for potential attackers, making it easier to potentially break the watermark.

Response: Thank you for your helpful observation and question. Although our watermarks share the same patterns for each image, given that the budgets for the watermarks are relatively low, it is still hard for the potential attackers to detect the existence of the watermarks. Even in hidden representation, it is still hard to detect the difference of watermarked images. In Figure 15 in Appendix J, we illustrate and compare the feature representations of a set of unaltered images from one class with their corresponding watermarked versions, along with images from a different class. The feature extractor is from a Resnet18 Model pre-trained by Contrastive Learning. From the figure we can see that the watermark of DiffusionShield makes almost no difference to the features of images, demonstrating that it is not easy for the potential attacker to discover the watermark in feature space and search for a way to break it. Besides, based on our study in watermark removal attack, even though the offender knows there is watermark, it is hard to remove them, which provides a robust protection for the copyright of the artworks.

We are grateful for the useful comments provided by you. We hope that our answers have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.