ROBIN: Robust and Invisible Watermarks for Diffusion Models with Adversarial Optimization

Q1. Could the method maintain its efficacy against a combination of attacks and reconstruction attacks?

As suggested by the reviewer, we have evaluated ROBIN under both conbination attacks and reconstruction attacks.

(1) We randomly selected various combinations of the six attacks outlined in this paper. The performance of watermark verification under different numbers of simultaneous attacks is shown in the table below.

Note that due to the inherent potency of the individual attacks, their combination leads to significant image quality deterioration. The resulting images are included in the attached rebuttal pdf. These images are no longer suitable for watermark detection as their integrity has been severely compromised. But ROBIN still demonstrates superior robustness compared to the state-of-the-art method Tree-Ring in such challenging scenarios.

Table 1. Watermark verification (AUC) on different number of random attacks applied at the same time.

(2) We evaluate the performance of ROBIN under different variants of reconstruction attacks [a]. As shown in the table below, ROBIN consistently exhibits stronger robustness under these adversarial conditions.

Table 2. Watermark verification (AUC) under reconstruction attack.

[a] Invisible Image Watermarks Are Provably Removable Using Generative AI. ArXiv:2306.01953 (2023).

Q2. Is the proposed watermarking technique applicable to noise-to-image diffusion models?

ROBIN can indeed be applied to noise-to-image generation models, as our scheme does not rely on the original text prompt input. Given that large-scale pretrained diffusion models are typically conditional generative models, we chose to use the unconditional capability of Stable Diffusion to simulate the noise-to-image generation process for this evaluation.

We evaluate ROBIN on the unconditional generation of Stable Diffusion, where the original text is set to NULL (representing no text prompt). In this setup, the image is generated unconditionally before the watermark injection point. After that, we still utilize our watermarking hiding prompt embedding $w_p$ to guide the generation process and actively erase the watermark. The results in the table below indicate that ROBIN can still function well in noise-to-image generation.

Table 3. Watermark verification (AUC) on noise-to-image generation.

Q3. Provide more insights about how the watermarking at a pixel level contributes to the robustness.

(1) In our scheme, the watermark is embedded in the latent space while the loss function is calculated at the pixel level (Equation 10 in the original manuscript). We believe that this approach, which combines pixel-level alignment with latent space optimization, is beneficial for improving robustness.

(2) This is because different latent representations can map to similar pixel-level expressions, allowing us to find a latent code that maps to visually the same image but also contains robust watermarks. This provides more opportunities to embed strong and robust watermark signals without introducing noticeable visual artifacts. The benefits of this optimization method are evident when we actively aim for concealment, a feature not supported by other watermarking methods.

(3) To further validate our approach, we also tested a variant of the ROBIN scheme where the loss function is computed at the latent level rather than the pixel level. The results presented in the table below demonstrate that latent-level alignment slightly decreases the robustness of the watermark, thereby underscoring the effectiveness of our pixel-level alignment strategy.

Table 4. Watermark verification (AUC) on different optimization settings.

Q4. Clarify more about the distinction between the passive nature of existing methods and the active nature of the proposed method.

We appreciate the reviewers' comments and would like to clarify this point in more detail.

(1) Existing methods achieve concealment passively because they don't take invisibility into consideration during watermark implantation and don't take actions to achieve this goal actively. Users are thus compelled to manually and empirically reduce the watermark strength to achieve stealthiness, often leading to a weak watermark signal and decreased robustness.

(2) In our method, we prioritize invisibility as a goal for watermark embedding and design our approach specifically to achieve this goal. We design a prompt embedding guidance to achieve invisibility, and our optimization of prompt embeddings guides the model to visually conceal the watermark. Consequently, our method can maximize watermark strength while minimizing visual artifacts.

Q1. The optimization of $w_p$ and $w_i$ in lines 13 and 14 seems not to be consistent with Algorithm 1. The definition of $\epsilon_\theta(x_t^*,t,\psi(p),w_p)$ should be moved to the main text.

(1) In the original manuscript, Lines 5 and 6 in Algorithm 1 indeed correspond to the minimization of the loss functions described in Equations 13 and 14. We recognize that the inclusion of additional inputs for the loss calculation in Algorithm 1 may have caused some confusion. Therefore, we will revise the expressions in Algorithm 1 to enhance clarity in the revised version.

(2) We acknowledge the reviewer's suggestion and will move the definition of $\epsilon_\theta(x_t^*,t,\psi(p),w_p)$ from Appendix A.2 to the Methodology section for improved understanding.

Q2. Confusion for the caption and content of Figure 2. An illustration of the overlapped curve in Figure 2a.

(1) The caption of Figure 2 in the original manuscript provides a simplified overview of the commonly used classifier-free guidance method [a], which has become a fundamental part of large-scale text-to-image diffusion models. A detailed explanation of this method has been presented in Appendix A.2 of the original manuscript. For better understanding, we include the definition from the original paper as follows. For classifier-free guidance, the label $y$ in a class-conditional diffusion model $\theta(x_t|y)$ is replaced with a null label $\emptyset$ with a fixed probability during training. During sampling, the output of the model is extrapolated further in the direction of $\theta(x_t|y)$ (Condition) and away from $\theta(x_t|\emptyset)$ (Uncondition)as follows: $Full=Uncondition + s\cdot (Condition - Uncondition)$. The second term of the addition is Guidance.

Figure 2 (in the original manuscript) illustrates how the above components (Full, Uncondition, Condition and Guidance) change during the generation process (Figure 2a) and the impact of frequency domain perturbations on these components (Figure 2b). To enhance clarity, we will incorporate the relevant reference [a] within the Figure 2 caption and include the formula above in the revised version for better understanding.

(2) We appreciate the reviewer's suggestion regarding Figure 2a. According to the above formula, the mean values of Uncondition and Condition are very close, resulting in their almost overlapping curves in Figure 2. We will incorporate a detailed explanation of the overlapped curves in the caption of the figure in the revised version.

[a] GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. ICML, 2022.

Q3. Explanation about the subscript in Table 3 and the results for other metrics.

(1) The subscripts in the original manuscript (Table 3) indicate the standard deviation of five independent experimental runs, each initialized with a different random seed. FID measures the difference between the five independently generated image sets and the ground-truth real image sets, so it has a standard deviation.

(2) Due to space constraints, we have omitted the standard deviations for other metrics such as PSNR and SSIM. These values are provided in the table below. We will clarify the meaning of subscripts and include the standard deviations for all metrics in the revised version.

Table 1. Image quality of five repeated experiments. Subscripts indicate standard deviation.

Q4. Some typographical errors.

We thank the reviewer for identifying the typos. The variable $w_t$ in line 110 should be $x_t$. And $w_t$ in the output line and initializaion of Algorithm 1 should be $w_p$. These errors will be corrected in the revised manuscript. We will also conduct a thorough proofreading of the original manuscript to avoid any other typographical errors.

Q5. Does the optimization create a big bottleneck?

(1) We do not believe watermark optimization to be a big bottleneck. First, the watermark optimization only needs to be done once. The optimized prompt embedding can then be applied universally to all images and text prompts without requiring further optimization. Second, the optimization process can be carried out offline, and the online watermark embedding incurs negligible overhead during image generation.

(2) The table below presents the time costs per image for watermark optimization, watermarked image generation, and watermark verification using Stable Diffusion. The watermark optimization of ROBIN only incurs an average time overhead of 0.112 seconds per image. The time overhead for watermark embedding during image generation is 0.068 seconds, which is negligible compared to the average image generation time of 2.614s using Stable Diffusion. The verification of ROBIN watermarks is 80% faster than Tree-Ring watermarks.

Table 2. Time cost (s) of the watermark operation on each image.

Q6. The watermarking capacity is not mentioned in the empirical section.

As mentioned in Line 187 in the original manuscript, the watermark capacity is 70% of the image frequency domain, corresponding to 30 and 120 concentric rings for Stable Diffusion and ImageNet diffusion model, respectively.

Q1. The robustness improvement over Tree-Ring is minor.

(1) Compared to Tree-Ring watermarks, ROBIN improves the robustness from 0.975 to 0.983 on Stable Diffusion, effectively reducing the error rate by 32% (from 0.025 to 0.017). Achieving further improvements on an already high AUC of 0.975 is inherently challenging.

(2) Additionally, our robustness evaluations were conducted under severe attacks that significantly compromised image quality, making watermark verification much more difficult. Thus, we believe our improvement is significant.

(3) Moreover, ROBIN offers substantial advantages in verification speed (80% improvement in verification time) and image quality (35% improvement in SSIM), compared to Tree-Ring watermarks. These enhancements underscore the substantial benefits of our approach.

Q2. Stable Signature is not compared in Table 2.

As mentioned in Line 198 of the original manuscript, Stable Signatures are specifically designed for latent diffusion models and are therefore incompatible with pixel-level ImageNet diffusion models (Table 2). This limitation is also mentioned in the Introduction section of the original paper of Stable Signatures [a].

[a] The stable signature: Rooting watermarks in latent diffusion models. ICCV, 2023.

Q3. Applicable only to diffusion models with DDIM sampler.

The applicability of ROBIN is not limited to DDIM samplers. Our watermark verification requires a reversible generation process, making it compatible with any reversible samplers such as DPM-Solver [a], DPM-Solver++ [b], PNDM [c], and AMED-Solver [d]. Our experiments employed both DPM-Solver and DDIM, and we anticipate ROBIN's adaptability to future reversible generation algorithms.

[a] Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 2022.

[b] Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv:2211.01095 (2022).

[c] Pseudo Numerical Methods for Diffusion Models on Manifolds." ICLR, 2022.

[d] Fast ode-based sampling for diffusion models in around 5 steps. CVPR, 2024.

Q4. Unclear how the proposed method resists frequency domain attacks.

Following your suggestions, to further assess robustness, we evaluated ROBIN under various low-pass filtering frequency attacks, which interfere with the frequency domain of the image without destroying the main content. The accompanying table presents AUC values on Stable Diffusion under different attack methods, demonstrating the superior robustness of our approach to frequency domain attacks.

Table 1. Performance evaluation under frequency-domain attacks.

Q5. The rationale behind maximizing the watermark signal value to increase robustness.

(1) Our approach adheres to the fundamental principle that stronger signals exhibit greater resilience to noise and interference. Existing attacks typically add small noise signals to the original image to disrupt the watermark without damaging the image. A stronger watermark signal has more "power" to overpower the noise signal, as demonstrated in [a].

(2) In the context of frequency domain watermarking that we use, the strength of the watermark signal is positively correlated with its numerical value. By maximizing the value of the watermark, we increase the watermark strength, thereby improving robustness.

[a] Digital watermarking and steganography. Morgan kaufmann, 2007.

Q6. The initialization and optimization process for the text prompt $w_p$ is unclear, especially when text prompt optimization is a discrete optimization problem.

(1) As stated in line 438 of the original manuscript, we initialized $w_p$ as empty strings to minimize its interference with the generation process.

(2) In our approach, we optimize a continuous prompt embedding rather than the discrete text prompt, which allows it to be optimized via gradient descent.

Model owners can introduce $w_p$ into the generation process as another guidance term independent of the original text for watermarking. We will clarify it in the revised version and replace the words ''prompt $w_p$'' with ''prompt embedding $w_p$'' for clarity.

Q7. Why the original prompt used for image generation is unknown during verification?

We target a more general watermark verification scenario where users might generate images using diffusion models and publish them online. Our goal is to determine whether these published images are authentic, even when users do not provide their diffusion prompts (likely they won't).

Q8. No noticeable improvement in the quality of generated images in terms of Clip score.

(1) In our paper, the image quality is assessed by PSNR, SSIM, and MSSIM metrics rather than CLIP score. And we achieve an improvement of 35% in SSIM over Tree-Ring watermarks.

(2) CLIP score evaluates the alignment between the generated image and its corresponding text prompt and is constrained by the base model's generative capacity. We achieved a relative 8.8% improvement in CLIP score (from 0.364 to 0.396) on Stable Diffusion (SD) compared to the optimal Tree-Ring watermarks, nearing the upper bound of the base model (SD) with a CLIP score of 0.403.

Q1. Clarify the motivation for preserving semantic, given that the sematic alterations caused by Tree-Ring remain faithful to the original text prompt, as evidenced by a negligible drop in FID.

(1) FID cannot evaluate the semantic faithfulness of the generated images to the orignal textual prompt. FID only measures the distributional distance between two unordered sets of images at the pixel level, regardless of the semantic content. So, a minor decrease in FID does not ensure that the generated images remain semantically consistent with the text.

(2) Semantic modifications induced by Tree-Ring watermarks are random and may result in images that deviate from the original textual prompts. This is evidenced by the decline in the CLIP score by 10% (from 0.403 to 0.364) on Stable Diffusion, which quantifies the correspondence between generated images and the given text. Figure 3 in the original manuscript presents examples of such generation failures.

(3) Therefore, we aim to exactly preserve the original semantics to achieve a better lower bound for faithfulness. We also improve the CLIP score by 8.8% compared to Tree-Ring. By generating a watermarked image semantically aligned with the original one, we minimize its impact on user experience (image-text alignment).

(4) In addition, we potentially support the scenario when users may expect watermarked and original Stable Diffusion to produce the same outputs to verify that they are using the correct version, which necessitates semantic preservation.

Q2. Attribution for frequency domain embedding.

In the original manuscript, we have credited Tree-Ring watermarks for the frequency domain embedding in lines 134 and 135. But for clarity, we will move it to line 133 in the revised version.

Q3. How to select the L1 distance threshold for watermark verification.

(1) In practical application scenarios, a fixed threshold is required for per-image watermark detection. We empirically set this threshold as the median L1 distance between 250 watermarked and 250 non-watermarked images. Using the calculated threshold (31.45), we achieve 100% detection accuracy on a separate 1000-image dataset.

(2) For research comparisions, we aim for a thorough evaluation under various thresholds to ensure a larger effective threshold interval. Therefore, we follow Tree-Ring [a] and use the Area Under the Curve (AUC) metric. AUC represents the area under the Receiver Operating Characteristic (ROC) curve, which plots the fraction of true positive results against the fraction of false positive results at various threshold settings. A higher AUC indicates a broader range of effective thresholds for real-world applications, reflecting a higher tolerance for errors.

[a] Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. NeurIPS, 2023.

Q4. Metrics like SSIM may not be entirely fair when comparing with Tree-Ring which doesn't aim to preserve semantic but keep faithfulness. Instead, FID would be a more appropriate metric.

(1) We acknowledge that in generative watermarking, it is not necessary for the watermarked image to exactly match the original image. Ideally, we may find another watermarked image that aligns with the text prompt, even if it differs from the original image. However, achieving this is challenging. Tree-Ring watermarks, as noted in response to Q1, are more akin to random semantic modifications and do not guarantee the same level of text alignment as the original generation (10% reduction in CLIP score).

(2) Instead, we make the simplest assumption that preserving the original semantics provides a better lower bound for faithfulness. Thus, we used PSNR, SSIM, and MSSIM to evaluate the similarity between the images before and after adding watermarks. Our improvement lies in achieving stronger robustness than Tree-Ring watermarks while producing outputs nearly identical to the original image.

(3) Additionally, our method potentially supports adding watermarks to a given image as we can maintain almost identical outputs. In contrast, Tree-Ring watermarks are limited to the generation phase.

(4) As we mentioned in the response for Q1, FID cannot evaluate the semantic faithfulness of the generated images to the original textual description but CLIP score does.

In addition, FID is only calculated on pixel level and is a less reliable metric for semantic integrity, which is crucial in generative watermarking. The table below illustrates the impact of various pixel-level image transformations on FID, including random erasing with ratio 7%, center cropping of 80%, JPEG compression with quality 45 and random rotation of 30 degree. The processed image samples are provided in Figure 1 in the rebuttal pdf.

The results show that JPEG compression, causing negligible visual changes for huamn eyes, significantly affects FID. In contrast, randomly erasing 10% of the image, which substantially alters image semantics, has limited influence on FID.

Table 1. FID under different image transformations.