GaussMarker: Robust Dual-Domain Watermark for Diffusion Models

Q1: However, in table 1, it seems that the propsoed method are more rebust than other watermarking method only under rotaion and C&S attack. Any comments? Moreover, as the GNR is trained in latent space, does it mean that it would be reobust on image space attack like JPEG and Brightness.

A1.1: GaussMarker also achieves better detection performance than Stable Signature, TreeRing and PRC under S. Noise, G. Noise, and Bright attacks, as present in Table 1. A major shortcoming of existing methods is that they do not perform well across different image distortions, whereas GaussMarker exhibits relatively consistent robustness across various image distortions.

A1.2: Yes, GNR can inherit the robustness of LDM, as present in the 1st and 4th rows of Table 6. However, since GNR uses an approximate objective, it may extract a little high detection score from an unwatermarked image sometimes (low TPR@1%FPR but high bit accuracy in the 4th row and S. Noise line of Table 6). Therefore, it is necessary to fuse the frequency score for a lower FPR.

Q2: Even though the proposed method does not achieve state-of-the-art performance under all attack, it achieves competitive results. However, comparing with other methods with similiar performance (RingID, TreeRing, etc), the proposed method needs additonal training.

A2: GaussMarker does need additional training, but the cost is minimal and the training is model-agnostic. As presented in the table below, the training of GNR only needs 72 minutes on 1 V100 32G GPU. Moreover, the time overhead incurred by GNR and Fuser during the detection phase is almost negligible.

Q3: Comparing other attacks, why applying Gaussian Noise attack in a multiple user applicaiton would grealty downgrade the accuracy is not clear, could the author provide some comment on this?

A3: When the number of users increases, higher bit accuracy is required in extracting watermarks from images to ensure high identification accuracy. For example, consider injecting a 3-bit watermark (ignoring unwatermarked images for simplicity). With two users assigned watermarks {0,0,0} and {1,1,1}, an estimated watermark $\tilde{w}$={0,0,1} allows us to correctly identify the first user with just 66.7% bit accuracy.

As presented in Table 1, the bit accuracy under Gaussian Noise is the most unstable. Under Gaussian Noise attack, GaussMarker only obtains 0.989 TPR@1%FPR, which means that nearly 11 watermarked images don't get high bit accuracy. Therefore, it is affected by the number of users greatly.

Q1: Scope of Equation 10: Equation 10 is designed to improve robustness against rotation and cropping. Does it also apply to non-geometric transformations like JPEG compression and blurring? If so, can you provide a formal explanation or additional experiments? If not, can you clarify its limitations?

A1: Equation 10 does not apply to common non-geometric transformations in our experiments. We will add this limitation. However, during detection, GNR takes the signal map of Gaussian noise that is estimated by LDM as input. Therefore, GNR can inherit the robustness of LDM (the 1st row of Table 6).

Q2: Generalization of GNR: In Table 6, GNR consistently improves detection across all distortions, despite being trained only on rotation and cropping. Given that deep-learning models often overfit to specific noise patterns, how do you explain this unexpected gain?

A2: Since the objective of GNR is approximate, as present in Figure 2 (nearly 30%-42%), we use random sign flipping (p=0.35) for GNR training. This prevents GNR from overfitting and ensures its good generalization. Therefore, GNR also enhances the robustness of GaussMarker under other attacks to some extent.

We provide additional ablation results ( TPR@1%FPR / bit acc. on SD V2.1) in the table below to verify this. With random sign flipping, the average TPR@1%FPR and bit accuracy of GaussMarker increases by 5.7% and 12.1%, respectively. Although GaussMarker gains the main improvement under Rotate and C&S attacks, the improvement under S. Noise, G. Noise, and Bright is also significant, especially on the bit accuracy.

Q3: Complementarity of spatial and frequency-domain watermarks: The paper claims that dual-domain embedding improves robustness. Can you provide an analysis or empirical study to justify why spatial and frequency-domain watermarks are complementary?

A3: We believe that the effectiveness of combining is built on the theoretical foundation of ensemble learning [a]. When both frequency-domain watermarking and spatial-domain watermarking can achieve precise and diverse detection results, as present in Table 6, fusing them may lead to more robust detection performance. This has been fully demonstrated by our experimental results.

[a] https://jmlr.org/papers/v24/23-0041.html

Q1: Why does GaussMarker largely outperform other methods when evaluated against the regeneration attack?

A1: This advantage stems from GNR, as shown in the SD V2.1 results presented in the table below. Regeneration using diffusion models mainly involves semantic editing, which may also include spatial-like transformations. Notably, since GNR is trained using the composition of rotation and cropping with random sign flipping (p=0.35) for preventing overfitting, it can learn the invariance to various spatial-like transformations. Therefore, GNR enhances GaussMarker's ability to resist spatial-like transformations introduced by regeneration attacks.

Q2: How do you evaluate the identification accuracy with multiple users?

A2: We adopt an approximate algorithm as detailed in Appendix C, following Gaussian Shading. Specifically, we first compute a theoretical threshold $\tau$ based on the predetermined number of users, $N$ and expected FPR. Given a watermarked image, if its bit accuracy exceeds this threshold, we consider the user generating it will be correctly identified among the $N$ users. A practical multi-user identification extension of GaussMarker is provided in Response A5.

Q3: For rotation attacks, is the GNR capable of generalizing to other useen rotation degrees (angles not used during training the GNR)? My concern is that since the CNN-based networks face challenges in handling the image rotations, how does GNR overcome this?

A3: GNR is trained with dense rotation angles randomly sampled from (-180$^{\circ}$, 180$^{\circ}$), and can converge easily. Therefore, even with unseen rotation degrees, they are likely to be approximated with minor errors by nearby angles.

Q4: The formulation and method description of this paper needs further proofreading and justification.

A4: We will improve the formulation and method description.

Q5: It is confusing whether GNR requires retraining for each signal map. If so, the scalability of GNR is reduced, and how to select which GNR to use when multiple users generate images from the same LDM?

A5: When we have multiple users, we also only need to train one GNR. Each LDM only has one unique model-watermark $w \in$ {0,1}$^l$ (or signal map) with its corresponding GNR. Each user will be assigned a unique key $k \in$ {0,1}$^l$ and a unique user-watermark $w_u \in$ {0,1}$^l$ with $w_u = \text{XOR}(w,k)$. We use the estimated $\tilde{w}$ to estimate the user-watermark $\tilde{w}_u = \text{XOR}(\tilde{w},k)$ for calculating the bit accuracy. This easy extension is similar to Gaussian shading, so we omit that detail from the paper. We can add it if needed.

Q6: The idea of GNR is similar to the robustly trained watermark decoder (in Stable Signature) and is only designed to enhance the robustness of rotation and cropping.

A6: The core ideas are similar, but training GNR is much more efficient. GNR does not require additional datasets for training, while Stable Signature needs to train its watermark decoder and VAE on the COCO dataset. Besides, in addition to rotation and cropping, GNR also includes random sign flipping and is robust to many spatial-like transformations in the pixel space, as detailed in Response A1.

Q7: How do you derive the Equation in lines 222-224 (needs index) from Equation 10?

A7: In Equation 10, GNR takes Gaussian noise as input and outputs its restored version. However, as shown in Figure 2, we find that the approximate invariance exists only in the signal space. Therefore, we redefine GNR to take the signal map of Gaussian noise as input and output the restored signal map, as formalized in the Equation in lines 222-224.

Q8: In table 4, why does SD V 2.1's TPR performance get a significant drop compared to SD V 2.0 in the last two rows (from 0.997 $\rightarrow$ 0.875), while the bit ACC remains the same? It seems that Spatial + GNR is sufficient to get a robust watermarking performance.

A8: This is because the Spatial+GNR sometimes extracts high bit accuracy from unwatermarked images, which lowers the FPR and TPR@1%FPR, especially on stronger models (SD V2.1 is a further fine-tuned version of SD V2.0). Note that bit accuracy is only calculated from watermarked images, so it is less affected. The frequency score helps to reduce the FPR through score fusion.

Q1: About "tunning-free".

A1: We consider GaussMarker to be a tuning-free method. The term "tuning-free" implies that SDs cannot be fine-tuned due to computational costs and the watermarking method can be attached to the model in a plug-and-play manner without touching the model weights. GaussMarker avoids fine-tuning SDs and instead trains only two small modules, which require minimal computational resources, as elaborated in Response 7.

Q2: Generalizability of these two components.

A2: We train a GNR for all SDs due to its model-independent characteristics. We train a Fuser for each SD, as the training data for the Fuser is generated from the corresponding SD.

Q3: Some problems in Figure 3.

A3: Some curves overlap at y=1. We will improve the illustration.

Q4: Why does frequency watermark only matter for V2.1?

A4: The frequency watermark matters more for stronger SDs when using GNR. In our experiments, we find that GNR occasionally extracts a high bit accuracy from unwatermarked images, especially on stronger SDs. This results in the need for a higher threshold to maintain the expected 1% FPR, which in turn lowers the TPR@1%FPR.

For instance, in Table 6, GNR+Spatial demonstrates better bit accuracy but worse TPR@1%FPR compared to Spatial under Gaussian Noise and Salt-and-Pepper Noise on SD V2.1. This significant drop does not occur with SD V1.4 and V2.0. Given the stable FPR of the frequency watermark [a], the frequency score becomes crucial for achieving a lower FPR through score fusion, especially on SD V2.1.

[a] https://arxiv.org/pdf/2305.20030

Q5: The relation between watermark capacity and the number of users in Figure 3.

A5: The watermark capacity and number of users in Figure 3 are not directly related. The watermark capacity, denoted by $l$, refers to the number of bits available for watermarking. The number of users that can be assigned unique watermarks must be much less than $2^l$, and a larger $l$ usually supports more users. In Figure 3(b), with $l=256$, we assess how accurately GaussMarker can identify the correct user as the number of users grows.

Q6: In Figure 3, why increasing the watermark capacity will make the watermark less vulnerable to Random Drop? why increasing the number of users will make the watermark less vulnerable to Gaussian noise?

A6.1: Maybe you want to use 'more' instead of 'less'. As shown in Equation 5, we use a voting strategy (average pooling) to estimate each bit in the watermark $\tilde{\omega} \in$ {0,1}$^l$ based on $cwh/l$ signs in the signal map $\tilde{s} \in$ {0,1}$^{cwh}$. When $l$ increases, fewer signs can be used for voting, which means that sign estimation needs to be more accurate for estimating the $\tilde{\omega}$. Random Drop masks 80% of the image with black pixels, significantly impacting sign accuracy. Therefore, the bit accuracy under this attack is most affected by the watermark capacity.

A6.2: When the number of users increases, higher bit accuracy is required in extracting watermarks from images to ensure high identification accuracy. For example, consider injecting a 3-bit watermark (ignoring unwatermarked images for simplicity). With two users assigned watermarks {0,0,0} and {1,1,1}, an estimated watermark $\tilde{w}$={0,0,1} allows us to correctly identify the first user with just 66.7% bit accuracy.

As presented in Table 1, the bit accuracy under Gaussian Noise is the most unstable. Therefore, it is affected by the number of users most. Note that we set the watermark capacity to $l=256$ in this experiment, thus, the performance under Random Drop is great with its 1.000 TPR@1%FPR. However, under Gaussian Noise attack, GaussMarker only obtains 0.989 TPR@1%FPR, which means that nearly 11 watermarked images don't get high bit accuracy.

Q7: Computation cost.

A7: The cost is minimal, as presented in the table below. These experiments are conducted on 1 V100 32G GPU.

Q8: Ablation study on fidelity.

A8: The ablation results are shown in the table below. Both spatial and frequency watermarks help preserve the original CLIP Score. For FID, the spatial watermark performs better than the frequency one. Combining both methods often requires more editing, leading to slightly worse CLIP Scores and FID for the dual-domain watermark compared to single-domain approaches. Improving the visual quality of watermarked images using GaussMarker could be a direction for future work.