GaussianMarker: Uncertainty-Aware Copyright Protection of 3D Gaussian Splatting

Dear Reviewer BKx3,

We will address your comments below and in the revised paper.

Weakness:

W1: Generalizability of decoders.

We agree that the generalization ability is very important for practical use. In our designs, we employ a two-layer protection approach to achieve generalizable and specific protection. We have two decoders in 2D and 3D, respectively. The 2D decoder is never trained per scene for generalization ability. In our setting, it is pre-trained and can be used across images rendered from different 3DGS models. This can provide protections for vast number of Gaussians distributed across the internet. Our second decoder is in 3D and is trained per scene. This is the second protection wall for 3D assets with more specific protection. The 3D asset owners can train their 3D decoder per scene based on the scene property. We believe such two-layer protection with both general and specific protection can make 3D assets safer.

W2: More complex, dynamic scenes.

In Table 1 of our main paper, we present evaluations of our method across three datasets, including MipNeRF360, widely regarded as the most challenging static scene dataset. To address your concerns, in this rebuttal, we further test our watermarking strategy on the Simplicits [1], a latest framework for simulating complex motions. In this place, we just need to simply modify the distortion layers in the original designs by incorporating various geometry distortions. From the left part of rebuttal Figure 4, our method still shows robustness in such new and complex scenarios. Besides, it also partly shows that our approach can be extended to dynamic 3DGS scenarios.

W3: Adversarial perturbations.

Following your suggestions, we have designed more experiments to evaluate adversarial robustness. A malicious user can employ an adversarial attack, such as PGD, to compromise hidden messages in rendered images. As shown in the right part of rebuttal Figure 5, adversarial attacks can indeed reduce bit accuracy, even with minimal visual distortion (PSNR > 30). However, the defense against such adversarial attacks can be easily achieved via adversarial training by generating adversarial samples during the training for HiDDeN decoder. Under adversarial training to the message decoder for 2D images, the bit accuracy can be significantly improved, while the image quality is preserved with a PSNR value larger than 30.

W4: Potential negative impacts.

Although our method demonstrates robustness against many sophisticated distortions, additional legislative strategies should be implemented to combat such malicious attacks beyond technological measures. We will explore extending our work by integrating legislative actions to provide comprehensive copyright protection for model owners.

Questions

If our understanding is accurate, BKx3’s primary concern is whether our watermarks can maintain geometric consistency, which is essential for robust extraction from different viewing angles. Our specific designs have equipped our digital watermarks with such consistency. Our method embeds watermarks into the 3DGS model by adding perturbations to 3D Gaussians with high uncertainty. As shown in Figure 3 of our main paper and supplementary material, these areas cover most object boundaries in the 3DGS scene, regardless of viewing angle. Rebuttal Figure 6 further illustrates geometry consistency by displaying the incorporated perturbations. These perturbations effectively cover the scene's general geometric structure. The geometry of these perturbations remains consistent across different camera angles and can be transmitted into the rendered images, enabling detection by the 2D message decoder.

The scenarios envisioned by BKx3 can be achieved by some classical information-hiding techniques [2], which use specific strategies to hide images or even videos in the target data and can employ a specific decoder to extract such hidden data. We try to address the newly raised questions related to this envisioned scenario below：

Viewing angles, information capacity, and comparison to traditional watermarking.

During training, careful selection of training and testing data guarantees that camera angles cover most scene perspectives, enabling successful watermark detection from various viewpoints. The information capacity relies on the model parameters of the decoder since this approach is conducted in an over-fitting manner. Compared with traditional watermarking, the envisioned method can decode different types of watermark data, but it relies on a specific decoder.

The 3D robustness.

Information hiding seldom considers robustness. However, this also belongs to a kind of fine-tuning approach, which means the signals are embedded by fine-tuning the original model. In this situation, based on Table 2 of the main paper, such a fine-tuning strategy may show degraded 3D robustness.

Trade-off.

As discussed in the rebuttal Table2 and the previous methods [2], stronger 3D perturbations can enhance watermarking decoding accuracy, while it may affect image quality. Thus, for the envisioned scenario, there should also be a similar trade-off.

However, we have to say our method is for digital watermarking, which is a different topic against information hiding, despite their somewhat correlations. We will investigate how to incorporate your suggestions into our future work and feel happy to discuss this point with you during the discussion session.

Limitations

We will further incorporate more limitations and potential impacts into our final version by considering the valuable suggestions from each reviewer.

[1] Simplicits: Mesh-Free, Geometry-Agnostic, Elastic Simulation.

[2] StegaNeRF: Embedding Invisible Information within Neural Radiance Fields.

Thanks for pointing out this. We agree that a general encoder-decoder framework in image watermarking is indeed highly desirable. However, the unique nature of 3D neural representations presents distinctions that require a different approach when compared to image watermarking.

In 3D neural representations, trainable parameters or networks are utilized to represent 3D scenes. This fundamental difference makes it challenging to directly apply the encoding techniques used in image watermarking to encode those parameters or networks. Consequently, some changes are necessary for watermarking neural representations.

These changes inevitably lead to certain optimizations during the message embedding for neural representations like previous CopyRNeRF for NeRF or our approach for 3DGS. We agree that this may potentially impact the generalization ability. To address this, our current strategy focuses on minimizing the time costs of the message embedding. By reducing the time required for embedding, we can partly mitigate the issues arising from these additional optimizations.

We have conducted extensive experiments here. Our results demonstrate that our method achieves significantly shorter message embedding times. However, such message embedding sometimes may need about 70 hours in established pipelines for neural representations. This improvement not only enhances efficiency but also helps to preserve the generalization capability of the watermarking technique.

Besides, we have to point out that the images used for training the 2D message decoder are all from the COCO dataset. This dataset does not have any correlations to the images used for the optimization of 3DGS models. Thus, it means that our 2D message decoder has never seen the images used for the optimization of 3DGS models, and can be used as a generalized message decoder on any novel 3D scenes.

What you mention is very insightful. We will incorporate those suggestions into the future work of our final version.

Dear Reviewer VdSM,

Thank you for your valuable feedback and constructive comments. We will address your comments below and in the revised paper.

Weakness

W1: Parameters for uncertainty calculation and partitioning.

As we have mentioned in the main paper (Section 4.1), the model parameters $\theta$ are used to compute the uncertainty, including all Gaussian parameters $\mu$, $R$, $S$, $c$, and $\alpha$. The uncertainty is estimated by calculating the Hessian matrix as an approximation of the Fisher information. This process can be simplified by computing the gradient for each Gaussian (main paper, lines 148-152) as $\mathbf{H}\left[\mathbf{I} \mid \mathbf{V}, \boldsymbol{\theta}^*\right]=\nabla_{\boldsymbol{\theta}} f\left(\mathbf{V} ; \boldsymbol{\theta}^*\right)^T \nabla_{\boldsymbol{\theta}} f\left(\mathbf{V} ; \boldsymbol{\theta}^*\right)$. Since Fisher information is additive (main paper, lines 153–155), the uncertainty of each Gaussian can be summed by adding the uncertainty of each parameter: $\mathbf{H}[\mathbf{I}|\mathbf{V},\mathbf{\theta}^*] = \mathbf{H}[\mathbf{I}|\mathbf{V},\mathbf{\theta^*_{\mu}}] + \mathbf{H}[\mathbf{I}|\mathbf{V},\mathbf{\theta^*_{R}}] + \mathbf{H}[\mathbf{I}|\mathbf{V},\mathbf{\theta^*_{S}}] + \mathbf{H}[\mathbf{I}|\mathbf{V},\mathbf{\theta^*_{c}}] + \mathbf{H}[\mathbf{I}|\mathbf{V},\mathbf{\theta^*_{\alpha}}]$. The high uncertainty 3D Gaussians are partitioned based on the Equation (7).

W2: The desify function 𝑔(⋅).

As we have mentioned in the main paper (lines 163-170), we retain the integrity of the original 3D Gaussians, denoted as $\mathcal{G}$. We densify the 3D Gaussians with high uncertainty, and the newly added 3D Gaussians are our proposed 3D perturbations denoted as $\mathcal{\tilde{G}}$. We embed $\mathcal{\tilde{G}}$ into $\mathcal{G}$ to get the watermarked Gaussians: $\mathcal{\hat{G}} = \mathcal{G} \cup \mathcal{\tilde{G}}$. We keep the integrity of the original Gaussians $\mathcal{G}$ and add 3D perturbations $\mathcal{\tilde{G}}$ on the original 3D Gaussian, similar to the 2D watermarking methods apply an invisible 2D perturbation messages on the original cover images: $x_{w} = x_{o} + \delta$ (main paper, lines 178-180).

W3: It is unclear whether the 2D and 3D watermarks are embedded into the same model parameters

The 2D and 3D watermarks are all embedded into the same model parameters. As mentioned in lines 207-210 of the main paper, our training contains two phases. In the first phase, we distill the messages into the model parameters. After this distillation, the messages have already been able to be extracted from 2D rendered images. In the second phase, we only train the 3D message decoder to ensure that the messages embedded in the first phase can be directly extracted from the 3D assets.

W4: Distilling watermarking knowledge.

As we have mentioned in lines 211-213 of the main paper, previous methods have shown that the 2D knowledges can be distilled into the 3D radiance fields via additional settings. In our designs, we distill the 2D knowledges from a pre-trained HiDDeN decoder into the Gaussian parameters as the embedded watermarks. Based on our proposed two training phases, such distilled knowledges can be extracted from the 2D and 3D domains.

Questions

Q1: Message extraction methods

In our experimental setting, for a fair comparison, “3DGS with message” and “3DGS with fine-tuning” methods all use the same 2D and 3D message decoders to extract messages. The 2D message decoder is a pre-trained HiDDeN decoder and can extract messages on the watermarked rendered images (main paper, lines 176–177). The 3D message decoder is based on PointNet architecture and can extract messages on the watermarked 3D Gaussians (main paper, lines 196–197).

Q2. Four types of 3D editing methods are listed in the experiment, which parameters of 3DGS are affected by these distortions?

The four types of 3D editing (3D Gaussian noise, translation, rotation, crop-out) occur in 3D space and are used to modify the positions $\mu$ of the 3D Gaussians, while other Gaussian parameters remain unchanged. We treat the 3D Gaussian mean $\mu$ as the point position and other parameters as the associated point features (main paper, lines 193-196), so that the PointNet-based 3D message decoder can extract the hidden messages directly from the 3D Gaussians and even they are spatially transformed or distorted.

Dear Reviewer QvjQ,

Thank you for your valuable feedback and constructive comments. We appreciate your suggestion about considering more sophisticated scenarios, and we will address your comments below and in the revised paper.

W1: Model fine-tuning and auto-encoder attack.

By following your suggestions, we have designed experiments for both the model fine-tuning scenario and the auto-encoder attack scenario.

For the model fine-tuning, we consider a challenging case where attackers have direct access to the original non-watermarked images used for the creation of the 3DGS. Based on this assumption, we implement the attack by eliminating the message loss and then fine-tuning the model solely through perceptual loss.

As shown in Figure 3, left part of the rebuttal paper, the bit accuracy still remains relatively high even when such fine-tuning compromises the model quality. It indicates that model fine-tuning cannot significantly reduce bit accuracy without undermining image quality.

For the autoencoder attack, we consider two kinds of VAE attacks: VQ-VAE [1], and VQ-GAN [2]. We use different compression rates to alter image quality and evaluate the robustness of the watermarking.

As shown in Figure 3 of the rebuttal paper, the message decoding can maintain relatively high accuracy and achieve good reconstruction quality (PSNR > 31), when the two kinds of auto-encoder use a low compression rate. Then, under a high compression rate, these two auto-encoder attacks can result in lower bit accuracy and degraded image quality. Although these two kinds of auto-encoder attacks can undermine the watermark in the rendered images, as shown in the Figure 3, right part of the rebuttal paper, such degraded image quality may lead to multiview inconsistency, making the sharing of the rendered contents difficult.

W2: Retrain a 3DGS with Bob's rendered images.

We appreciate your insightful feedback. As shown in the table below, our experiments on retraining a new 3DGS model using Bob's rendered images reveal that, even with a slight reduction in message decoding accuracy, the fully converged retrained model can still maintain a relatively high bit accuracy.

We appreciate all your valuable suggestions and look forward to discussing them further with you during the discussion session.

[1] Neural Discrete Representation Learning.

[2] Taming Transformers for High-Resolution Image Synthesis.

Dear Reviewer fyQG,

We will address your comments below and in the revised paper.

W1. One concern about this paper is its novelty.

Thanks for raising the concern. While our approach incorporates elements from classical techniques, our contributions extend beyond these traditional frameworks. Besides the first exploration to protect the 3DGS via digital watermarking, our primary contributions also lie in developing a framework based on uncertainty for embedding invisible perturbations into the 3DGS model, and message extraction from both 2D images and 3D Gaussians.

W1-1. Use uncertainty to achieve invisible watermarking of 3DGS

One major contribution is the application of uncertainty estimation in invisible watermarking of 3DGS, an exploration that diverges from its traditional use in active learning [1]. We find that uncertainty estimation can inherently identify model parameters that are more tolerant to perturbations, making it well-suited for the purpose of invisible watermarking in 3DGS.

Without considering the uncertainty, naively applying the existing watermarking strategies can cause noticeable distortions. As shown in rebuttal Figure 1, compared with our method, "3DGS with message" and "3DGS with fine-tuning" all show poor reconstruction quality and reduced bit accuracy. This further demonstrates the superiority of the proposed uncertainty-based strategy. Besides, the HiDDeN decoder mainly focuses on the decoding of information along the boundary areas. From the left part of rebuttal Figure 2, such boundary areas are all with high uncertainty values. This can also partly show the correlation between the uncertainty and the message embedding.

W1-2. Message extraction from both 2D images and 3D Gaussians.

Our second contribution is a method that enables copyright message extraction from both 2D images and 3D Gaussians. Existing watermarking techniques for 3D assets (e.g., NeRF) are limited to extracting messages from 2D rendered images, as directly extracting messages from the neural networks used for scene representation in NeRF proves challenging. Consequently, owners of the 3D assets can only assert their ownership through these rendered 2D images, rather than from the underlying 3D neural representation itself. Our approach allows direct message extraction from 3D Gaussians using PointNet. This provides model owners with a novel means of claiming ownership directly from their 3D assets, rather than relying solely on rendered images. This advancement offers a more robust and versatile approach to protecting the copyright of 3DGS model owners.

W2: The proposed method utilizes the 3D Gaussians with high uncertainty to embed watermarking. What if an attacker also uses this feature?

In response to your concern about this potential risk, we have conducted additional experiments specifically attacking 3D Gaussians with high uncertainty values to assess the robustness of our approach. We assume a scenario where an attacker directly manipulates the high-uncertainty 3D Gaussians. As demonstrated in the table below, our method maintains relatively high bit accuracy despite this attack.

This resilience partly stems from the fact that the uncertainty distribution of watermarked 3D Gaussians can differ from that of the original 3D Gaussians, as mentioned in the main paper (lines 168–170). If the attacker conducts the operation based on the original Gaussians, it is still challenging to attack the watermarked Gaussians with new properties. Additionally, our 3D message decoder randomly samples 3D Gaussians (main paper, lines 197-199) so that even if some of the high-uncertainty Gaussians are deleted, messages can still be extracted from other 3D Gaussians. However, it is still suggested that the watermarking strategy be kept private.

W3: The influence of the parameter uncertainty threshold should be included.

In our experiments, we set the average uncertainty value as the default threshold. By following your suggestion, we show more results to verify the sensitivity of the uncertainty threshold. As shown in right part of the rebuttal Figure 2, a lower threshold can enhance the bit accuracy, though it slightly compromises image quality. Conversely, a higher threshold results in better image quality and a more lightweight model but also slightly compromising message decoding accuracy. However, in both situations, the compromises are moderate.

W4. The results with different bit lengths are missing.

As we have discussed in the main paper (lines 273-275), we evaluate the message decoding capacity by setting the bit length to 48 bits, aligned with the maximum length used in 3D model watermarking methods [34, 4]. Shorter message bit lengths typically yield higher decoding accuracy. We evaluate the 16-bit and 32-bit message decoding accuracy on the Blender dataset and show the results in the below table.

[1] Unifying approaches in active learning and active sampling via fisher information and information-theoretic quantities.

We are pleased to hear that our revisions have addressed your concerns. Your constructive comments help us improve the quality and robustness of our work.

We will incorporate all your valuable suggestions into our final versions. If you have further questions, please feel free to raise them. We would be grateful if you could consider raising your scores. Thanks very much.

Best Regards,

Authors