FreqMark: Invisible Image Watermarking via Frequency Based Optimization in Latent Space

Dear Reviewer TJux,

We sincerely appreciate your thoughtful review and constructive suggestions on our paper. We have carefully considered your comments and have taken them into account.

W1: One of the main weaknesses of the work is that there is no convincing proof or theoretical justification of why hiding watermark bits in the FFT of latent space should be beneficial. The usual argument of using frequency domain for hiding watermark bits makes sense in that the FFT is applied to images (i.e., image pixels) and captures the spatial relationships between the image pixels. What kind of "spatial" relationship is captured by the FFT in the latent space and why is that beneficial for hiding watermark?

A1: The primary distinction between the latent and pixel domains is that the latent domain is the result of information compression and downsampling of the pixel domain. Training on the pixel FFT alters the overall distribution of the image, making it more robust against common attacks. A similar effect occurs in the latent FFT domain. The key difference is that the compressed latent information is restored back to the pixel domain with the assistance of the model's prior knowledge through the VAE decoder. During this process, the watermark information added to the latent FFT interacts and fuses with the semantic information of the image. Consequently, FreqMark exhibits enhanced robustness against regeneration attacks, as detailed in Section 5.3. This also serves as the answer to question 2.

Bit Accuracy on DiffusionDB datasets:

W2: FFT produces complex values, yet there is no discussion of how the watermark bits are embedded on these complex values. In Fig. 1, the outputs of FFT are displayed as magnitude images and if this is what is intended, what happened to the phase of the FFT outputs.

A2: The output of latent FFT is complex, and we optimize both the real and imaginary parts during the optimization process.

The latent FFT is then restored to the pixel domain through IFFT and the VAE decoder. The perturbation in the latent FFT domain alters the feature vector obtained after passing the image through an image encoder (DINO v2 in this case), thus achieving message encoding.

For example, the output vector after perturbation for an image is [-2, 1, -0.5] . We predefine three vector groups as [1, 0, 0], [0, 1, 0], and [0, 0, 1] (assuming a 3-bit message encoding for simplicity). After performing element-wise multiplication, we obtain the results -2, 1, and -0.5. We use the sign of these results as the 0-1 bit encoding, ultimately yielding an encoded message of 010. This also serves as the answer to question 1.

L1: Authors mention that there is room for optimization in the fusion of images and watermarks. It would have been useful to have a more detailed discussion of this aspect of the proposed method.

A3: Experimental results demonstrate that the image quality post watermark addition via FreqMark does not significantly deteriorate compared to the image quality achieved solely through VAE encoding and decoding. Therefore, a VAE with enhanced performance may potentially further improve the quality of the watermarked images.

We plan to further optimize our method from the following aspects:

1. Exploring the necessity of optimizing all elements in the latent FFT could be a focus for further investigations. Achieving better integration might be possible by optimizing only a portion or a specific channel while maintaining performance.

2. The utilization of diffusion models' robust capabilities could potentially facilitate the implantation of watermarks, resulting in superior fusion effects.

3. With the assurance of performance, a strategy could be developed to further blend the watermarked image post FreqMark processing with the original image, potentially leading to enhanced fusion effects.

We genuinely appreciate your valuable suggestions and will continue to optimize and explore our methods with enthusiasm.

Thank you for your consideration! We have further addressed the concerns of other reviewers, which you can directly view.

Dear Reviewer kwvu,

Thank you very much for your careful reading and recognition of our work, we will address each one sequentially below:

W1: Lacks some theoretical analysis. In the experiments, do you need to use the same VAE model for the attack as you have used during watermarking training? If not, then an analysis may be needed as to why it is still robust against attacks with different VAE models.

A1: In the experiment, it is not necessary to use the same VAE model for attacks as the one used for watermark training.

Essentially, adding information will inevitably alter some aspects of the image, and our goal is to ensure that this information remains as intact as possible during attacks. As demonstrated in Section 5.3 and Table 3, different optimization methods exhibit distinct characteristics.

• Directly embedding watermarks in the Pixel or Latent domains makes them highly susceptible to various traditional attacks, as specific bits of information may independently reside in one or more pixels. In contrast, embedding watermarks in the frequency domain effectively resists multiple traditional attacks because this method distributes the information across the entire image.
• Similarly, watermarks embedded in the pixel space are weak against regeneration attacks, whereas embedding watermarks in the latent space significantly enhances robustness against such attacks. This is due to the stronger correlation between the optimized latent embeddings and the image semantics, making the watermark more resistant to disruption by regeneration attacks.

FreqMark embeds watermarks in the latent frequency space, combining these two aspects to produce a synergistic effect, making FreqMark robust against traditional attacks and regeneration attacks.

Furthermore, due to FreqMark's characteristic of training the image itself, it exhibits stronger robustness against regeneration attacks using the same VAE.

Bit Accuracy of regeneration attack using the same VAE

W2: Since the proposed method requires case-by-case optimization, what is the watermarking time for each case, and how does it compare to other competing methods?

A2: We employed half-precision training to reduce watermarking time and GPU memory usage without compromising performance. In our current experiments, using a single A-100 GPU with 40GB memory, and process four 512x512 images in parallel for 400 steps takes about 0.75 minutes per image. Increasing the batch size will improve overall efficiency.

As shown in Figure 7 in the Appendix, FreqMark demonstrates considerable performance at just 200 steps, and even at 100 steps. This allows for the reduction of steps as needed to save time. According to our experiments, processing four 512x512 images in parallel for 100 steps can be accelerated to about 12 seconds per image.

Bit Accuracy with different number of steps

To train the Stable Signature, one must first spend a day training the watermark extractor on 8 GPUs. Subsequently, for any specific hidden message, the stable signature requires approximately 1 minute to fine-tune the VAE decoder.

SSL directly optimizes the image pixels, resulting in faster processing speed. Under the same experimental settings, it takes approximately 1 second to process per image. However, FreqMark demonstrates a significantly stronger robustness advantage than SSL.

We will further attempt to reduce the optimization time without compromising performance.

W3: Why do we need a set of pre-trained N-dimensional direction vectors, but not directly produce the message?

A3: The method of extracting hidden watermark information by predefined vector directions is referenced from SSL. Overall, this approach is more suitable for post-generation self-supervised methods, providing stronger robustness for hidden watermarks while maintaining image quality. Additionally, this method effectively addresses the challenge of obtaining hidden watermark information without the need for additional training of the extraction network.

W4: In Equation (4), why $k∈N$? N is a number, not a set, so $∈N$ is not appropriate. Besides, using $v^N$ to denote a vector set and $v^k$ to represent a vector can lead to confusion. In Figure 2, there are $K+1$ vectors in the set, I think $K+1$ is the bit length.

Thank you for your careful reading and for pointing out the errors. This section of the paper indeed has some incorrect expressions that caused confusion, and we will make the necessary corrections.

The dimension of the vector is N-dimensional. For clarity and expression, the number of pre-defined vectors is defined as K (we will correct Vector 0 to Vector 1 in Figure 2). To avoid interference between different bits of information, the pre-defined vectors should be orthogonal to each other, thus K should be less than or equal to N. This means that a maximum of N bits of information can be embedded for an N-dimensional vector.

Therefore, The correct expression should be that the pre-defined vectors are a set of K N-dimensional vectors $V^N = \\{v^1, v^2, ..., v^K \mid K\leq N\\}$.

And Equation 4 should be corrected to:

$m_d^k = sign(z_{I_{w}} \cdot v^k) = sign(E_{img}(I_{w}) \cdot v^k), v^k \in V^N$

We will continue to carefully review and correct any mistaken and confused statements in the paper.

Thanks again for your suggestions and corrections!

Dear Reviewer BgZJ,

Thank you for your insightful feedback and questions. We address the concerns as follows:

W1: The major contribution of this paper is to introduce a kind of image watermark robust to regeneration attacks. However, this part is not highlighted in its title. The authors are suggested to consider this in the title.

A1: Thank you for your suggestion, we will consider it positively. Yes, FreqMark's ability to resist regeneration attacks is a significant advantage, and we believe the primary contribution of the paper lies in the innovative approach of embedding watermark in the latent frequency space, and the extensive experiments conducted to demonstrate its effectiveness.

W2: The proposed methodology follows an established manner with incremental innovations. The whole approach does not demonstrate enough differences when compared with the previous approaches.

A2: The core idea of FreqMark lies in embedding watermarks in the latent frequency space of images. Frequency domain optimization subtly alters the overall image, enabling the watermark information to be hidden within the entire picture. The latent domain represents the compressed image, and during the process of restoring the image using the decoder, the added watermark information naturally integrates with the semantic information of the image. This results in outstanding robustness when facing regeneration attacks and various other types of attacks.

Bit Accuracy on both ImageNet and DiffusionDB datasets:

Furthermore, our experiments demonstrate the impact of different watermark embedding positions on robustness, and we delve into the advantages and limitations of FreqMark.

Bit Accuracy on DiffusionDB datasets:

FreqMark also boasts several additional advantages. It is a plug-and-play solution that can be directly applied to existing pre-trained models without any network training. During the watermark embedding process, users can freely adjust parameters to suit their specific needs, offering considerable flexibility, which will be discussed in detail in the W3.

It is important to emphasize that FreqMark differs from previous methods.

• FreqMark, as a post-generation method, is fundamentally distinct from merged-in-generation methods like Stable Signature, and it offers greater convenience and flexibility in use.
• Unlike most works such as StegaStamp, FreqMark does not require the training of any network parameters.
• Compared to other network-training free methods like SSL, FreqMark substantially enhances robustness through its unique watermark embedding approach.

W3: For the second contribution, the authors claim that their proposed framework is flexible and then say such flexibility guarantees a trade-off between the bits number, image quality, and robustness. However, why can such properties be considered as flexibility?

A3: There are three crucial attributes for invisible image watermarking—bits number, image quality, and robustness. Based on information theory, these attributes form an impossible trinity, meaning it is impossible to achieve all three simultaneously. For instance, if a method achieves high bits number and image quality, it inevitably cannot achieve high robustness; similarly, if it aims for high bits number and robustness, it cannot maintain high image quality. Thus, users need to make trade-offs among these attributes based on different application scenarios.

Specifically, the flexibility of FreqMark is reflected in the following tow aspects:

1. Merged-in-generation methods fix the model's capabilities during the training process, which prevent users from choosing between three attributes. FreqMark grants this choice to users, allowing them to freely adjust parameters during the watermark embedding process to achieve the desired balance.
2. Users can independently choose the VAE model and image encoder. Additionally, the pre-defined vector can be customized by users, serving as a key that adds an extra layer of security to the watermark.

W4: The whole pipeline presented in Figure 2 lacks novelty. This is just a very common encoder and decoder framework. The authors need to better consider their contributions for this part.

A4: Figure 2 illustrates the entire process of watermark embedding and extraction, while the main idea of the paper is depicted in Figure 1. We still believe that presenting the overall process is necessary, as it helps readers better understand our method. Although Figure 2 reflects the encoder-decoder framework, it differs from other methods in the following ways:

1. FreqMark uses only pre-trained encoders and decoders without requiring any network training. In Figure 2, we use a "snowflake" icon to denote the frozen neural networks.
2. FreqMark achieves watermark embedding by training perturbations in the latent frequency space. In Figure 2, we use a "fire" icon to indicate the trained perturbations.

Thank you for your suggestion. We will consider improving or adding image to better highlight our contributions.

Dear Reviewer BZ7h, Thank you for your suggestions. We address the concerns as follows:

W1: In my opinion, the FreqMark is an incremental variant of the StegaStamp [1]. ... The resemblance of equation (10) in this manuscript to the loss function equation (2) in [1] begs the question of novelty.

A1: In fact, there are significant differences between FreqMark and StegaStamp, and FreqMark is not an incremental variant of StegaStamp. We have discussed StegaStamp in the Related Work section and elaborated on the differences between the two kind of approaches in lines 22 to 27 of the Introduction section.

• FreqMark leverages the representational capacity of pre-trained networks to optimize the latent FFT of images, without the need for training any additional networks, adopting a case-by-case watermarking approach. In contrast, StegaStamp involves training an encoder and decoder under various perturbation augmentations.
• FreqMark only incorporates two types of noise as augmentations during the training of image perturbations, it maintains excellent robustness under various attacks.
• The loss functions of FreqMark and StegaStamp hold distinct meanings. FreqMark employs an image encoder to encode the image into a representative vector, which is then multiplied by a predefined vector group and the watermark image's representative vector. The encoding is determined by the positive or negative nature of the result, which is entirely different from StegaStamp.

We conducted a comparison under the same bit number(100) and essentially consistent image quality.

The findings indicate that FreqMark presents a balanced performance and exceptional robustness against diffusion attacks. Additionally, its network independence provides great flexibility and security. Users can dynamically choose the optimal image quality and encoding bit number, and customize their decoding vector group, thereby augmenting the security of the watermark decoding process.

W2: The authors also needed to compare against the state-of-the-art Tree-Ring watermark [4].

A2: Tree-Ring integrates the watermarking process and generation process, encoding 1-bit message (with or without watermark) through its approach, which is an in-process method. FreqMark is a post-processing watermarking technique capable of encoding multi-bit message, thus offering a broader range of applicability and a higher amount of encoded message. Therefore, following the metric setup of Tree-Ring, we compared the TPR@1%FPR of FreqMark and Treering under various attacks.

The experimental results demonstrate that the robustness performance of both methods is very close. However, it is important to note that FreqMark encode 48-bit message, which is a significantly larger amount of information compared to Treering's 1-bit encoding.

W3: As noted in [3], there is no single perceptual metric that is an objective measure of image quality...

A3: Thank you for your suggestion. We have added CLIP-FID and L2 as new image quality metrics, with the results as follows:

The experimental conclusions regarding image quality are consistent with the results in the paper, demonstrating that FreqMark exhibits outstanding robustness performance while maintaining acceptable image quality.

W4: 500 images is too small a sample size for the tested FPR thresholds.

A4: We utilized 5,000 watermarked images to plot the FPR-TPR curve, and some key results are as follows:

The experimental results are close to those in the main text. We will update the experimental results in the paper accordingly.

W5: VAE regenerations are far weaker compared to diffusion regenerations.

A5: We have gradually increased the diffusion steps. The results are as follows:

It can be seen that FreqMark maintains good robustness even at higher diffusion steps.

W6: As observed in [3], the use of publicly available VAEs to encode/decode watermarks is easily defeated...

A6: The characteristic of FreqMark that trains the image itself rather than the network can ensure strong robustness when facing regeneration attacks using the same VAE.

This is because the perturbation on the latent FFT changes the overall distribution of the image latent, making the watermark message affect the entire image globally. Therefore, it is difficult for perturbations on the image's latent to damage the watermark message. A similar phenomenon is observed in the pixel FFT of the image.

Q1: Which version of Stable Diffusion was used for the regeneration attack?

A7: Following the experimental setup of [2], we employ stable diffusion 2-1 to conduct diffusion attacks.

First I want to thank the other reviewers and authors for their detailed replies. Two items:

1. Another reviewer also questioned the motivation of using the FFT in the latent space. I am still not completely convinced by the author response as there are many adversarial attacks which specifically target the latent representation (see Adv-Emb attacks in [3].
2. Could the authors provide the TPR/FPR for the varied diffusion steps?

The new comparisons against Tree-Ring (another FFT watermark) and StegaStamp highlight the utility of FreqMark. I am increasing my score to a 5, and looking forward to further response.