IPR-NeRF: Ownership Verification Meets Neural Radiance Field

Dear Reviewer,

We appreciate your thoughtful review of our NeRF model implementation and thank you for raising pertinent questions. We would like to address your concerns and provide further clarification on the normalization layers and other aspects of our manuscript.

(a)Clarification on Normalization Layers:

• We sincerely apologize for the oversight. Yes, it is correct that the original NeRF model [1] did not have normalization layers. In our work, we modified the NeRF model by appending normalization layers to all Multi-Layer Perceptron (MLP) layers. This modification aimed to embed a digital signature, optimized through sign loss objectives, thereby enhancing Intellectual Property Rights (IPR) protection in a white-box scenario.
• Our experiments (Table 2) demonstrated that the introduction of these normalization layers did not adversely impact the rendering performance. We will include this description in the revised version of our manuscript to ensure reproducibility.

[1] https://arxiv.org/pdf/2003.08934.pdf

(b)Response to Figure 5 Distribution:

• Although a substantial portion of the distribution of the protected model and the random images overlaps, there is still a difference of 0.2979 and 2.77 in the computed average maximum mean discrepancy and average Wasserstein distance across all scenes. The details of each scene are as follows:

• The trained diffusion-based watermark detector can only extract a watermark from the overfitted distribution of the protected model. As a result, it is unable to extract the watermark from the distribution of random images and the unprotected model in which the extracted image is essentially a noise image similar to watermark image extracted by 2D steganography method (see Figure 1 of the main paper).
• This is because the trained diffusion-based watermark detector was exclusively and strongly overfitted to the distribution of the protected model, effectively preventing false positive detections. We apologize for any confusion caused by our writing and will address this issue by refining our manuscript in the revised version.

(c)Apology for Grammatical Mistakes:

• Turning to the matter of grammatical errors, we extend our sincere apologies for any oversights. We have revised our manuscript, focusing specifically on rectifying grammatical mistakes, including those highlighted by your insightful feedback.

We value your constructive feedback and assure you that our revised manuscript will comprehensively address and clarify the points you have raised. Your input is invaluable, and we are committed to ensuring that our work meets the rigorous standards of the conference.

Thanks for the clarification on the implementation details of the normalization layers. It is a crucial point in your work.

You'd better show the false positive rate (FPR) and false negative rate (FNR) using the proposed decision boundary ($w_\text{SSIM}$ > 0.75), rather than the average mean discrepancy or Wasserstein distance. The choice of evaluation metric makes the judgment of the proposed method difficult to assess its effectiveness. The authors just mentioned "preventing a false positive detection" but didn't say how much to prevent that.

Clarification on the Implementation Details of the Normalization Layers: Thank you, this is to confirm that we have included the normalization layer implementation details in the revised manuscript to ensure reproducibility.

False Positive Rate (FPR) and False Negative Rate (FNR) using the Proposed Decision Boundary: As suggested, we show the false positive rate (FPR) and false negative rate (FNR), along with the true positive rate (TPR) and true negative rate (TNR) for completeness, employing the proposed decision boundary ($w_\text{SSIM} > 0.75$) as below,

It indicates that the trained watermark detector will only effectively detect watermarks from the protected model. We will include this result in the paper.

Note that, the True Positive (TP) for the protected model represents the correct detection of the watermark in the watermarked scene. On the other hand, the TP for both the unprotected model and random images indicates the correct non-detection of the watermark in unmarked scenes.

We are delighted to have addressed your concerns and would like to express our gratitude for your recognition of our revised version. Your thoughtful and constructive feedback has been integral to making these improvements possible.

We have updated the Section 3.1 (black-box) and Section 4.3 (False positive detection). Specifically, we’ve mentioned in the updated Section 3.1:

• This work was inspired by (Ho et al., 2020) to propose a diffusion-based method to learn the detector d. This is because the nature of diffusion models to gradually add noise to the image provides natural robustness against various forms of image degradation attacks as the main objective of diffusion models is denoising. Thanks to the powerful denoising ability of diffusion models, our idea is to first convert the views x into Gaussian noises, then we denoise from these Gaussian noises into the watermark w. As a result, we can seamlessly translate between x and w through d.

Then in Section 3.2 (white-box), we added

• To achieve this, normalization layers were appended to the MLP implementation.

Also, we have updated the Section 4.3, as well as Appendix A.5 with experiment details:

• Both the protected and unprotected NeRF models in this experiment employed scenes from the NeRF-Synthetic 360 dataset, including Lego, Chair, and Drum, as well as scenes from the LLFF-Forward dataset, namely Fern, Fortress, Room, and Flower. To further understand why the false positives did not happen, we compute the histogram of 100 different views from each scene of protected/unprotected and 100 random images.
• A new confusion matrix (Figure 14) and Figure 15 (Qualitative result of the various recovered watermarks from the rendered image of the protected model, rendered image of the unprotected model and random images) are added to further clarify the False Positive Detection Prevention.

In the appendix, new experiment using Unbound-360 Dataset is also included in Appendix A.10 to clarify the impact of dataset selection on ownership verification.

• We conducted an experiment using the 'Garden' and 'Kitchen' scenes from the Unbound-360 dataset. We found that the quality of the recovered watermark still maintained a high standard with an SSIM of 0.95 overall (Kitchen: watermark SSIM 0.9625; Garden: watermark SSIM 0.9526).

Finally, the font in all figures has been adjusted to ensure better clarity and comprehension.

Dear Reviewer,

Thank you for your thoughtful comments and concerns regarding the technical contribution of our work. We appreciate the opportunity to clarify the key points you raised.

(a) Technical Contribution to Black-box Protection:

• We propose diffusion models as the foundation for a watermark detector, enabling the extraction of embedded watermarks within the rendered images of NeRF models. Our methodology has demonstrated noteworthy resilience against noise, a point substantiated in our research. More importantly, our work stands as a pioneering effort in uncovering IPR protection strategies for NeRF models, a pursuit of paramount significance for the 3D vision community. The escalating commercial use of NeRF, particularly in applications like the metaverse, underscores the urgency for a robust NeRF IPR protection framework.

• While our black-box protection methodology hinges on diffusion models, it's crucial to highlight the non-trivial nature of jointly optimizing watermark injection into the rendered scenes of NeRF (Eq 8). This optimization task becomes even more intricate when striving for a high watermark recovery rate. Consequently, the primary contribution of our work lies in pioneering the application of diffusion models for black-box NeRF protection. We demonstrate the feasibility of the joint optimization process (Eq 8) for embedding watermarks into rendered scenes and extracting embedded watermarks using diffusion models (Eq 4 and Eq 5), a valuable insight for the 3D community. In essence, our work contributes to the evolving landscape of NeRF IPR protection. Also, it serves as a testament to the viability and applicability of diffusion models in the joint optimization process. We believe these advancements are crucial steps toward a more secure and resilient NeRF IPR protection framework.

(b) Technical Contribution to White-box Protection:

• We acknowledge the existence of DeepIPR [3] but contend that its technical contribution might be deemed insufficient. Upon reviewing previous works on IPR protection for NeRF (e.g. StegaNeRF [1] and CopyRNeRF [2]), we observed a predominant focus on black-box protection strategies. This reliance on black-box protection becomes a point of vulnerability. In scenarios where black-box protection is compromised or when black-box ownership verification is not verified, the owner of the protected NeRF model may find it challenging to assert ownership in cases of unauthorized misuse. Given the substantial resources required to train a high-performance NeRF model, this ambiguity surrounding ownership could lead to significant financial losses.

• Considering these factors, we have proactively integrated the DeepIPR technique, widely recognized as a robust white-box protection framework against various IPR attacks in the neural network IP protection research field. By doing so, this work offers complete IPR protection for the NeRF model. The comprehensive IPR protection framework proposed in our work not only addresses potential compromises in any one protection aspect but also aims to elevate the overall security of NeRF models. Moreover, the intent in applying DeepIPR's white-box protection extends beyond the immediate scope of our work. We aspire to raise awareness among future researchers in the IPR domain of NeRF models. We recommend and demonstrate that both white-box and black-box protection strategies be considered to ensure the safety and effectiveness of NeRF model protection. In this way, we envision future work on NeRF IPR protection to be comprehensive in its approach, covering both white-box and black-box dimensions.

[1] https://arxiv.org/abs/2212.01602
[2] https://arxiv.org/abs/2307.11526
[3] https://www.computer.org/csdl/journal/tp/2022/10/09454280/1uqBfxOEjny

(c) Fidelity Analysis:

• Thanks! We appreciate the suggestion. And Yes! Our proposed method does not affect the fidelity of the rendered images. We have included a comprehensive fidelity comparison between our proposed method and the existing methods, namely StegaNeRF[1] and CopyRNeRF [2] in our revised manuscript, as well as below.

We hope these clarifications demonstrate the significance of our contributions and address your concerns. We appreciate your valuable feedback.

We have updated the Section 3.1 (black-box) and Section 4.3 (False positive detection). Specifically, we’ve mentioned in the updated Section 3.1:

• This work was inspired by (Ho et al., 2020) to propose a diffusion-based method to learn the detector d. This is because the nature of diffusion models to gradually add noise to the image provides natural robustness against various forms of image degradation attacks as the main objective of diffusion models is denoising. Thanks to the powerful denoising ability of diffusion models, our idea is to first convert the views x into Gaussian noises, then we denoise from these Gaussian noises into the watermark w. As a result, we can seamlessly translate between x and w through d.

Then in Section 3.2 (white-box), we added

• To achieve this, normalization layers were appended to the MLP implementation.

Also, we have updated the Section 4.3, as well as Appendix A.5 with experiment details:

• Both the protected and unprotected NeRF models in this experiment employed scenes from the NeRF-Synthetic 360 dataset, including Lego, Chair, and Drum, as well as scenes from the LLFF-Forward dataset, namely Fern, Fortress, Room, and Flower. To further understand why the false positives did not happen, we compute the histogram of 100 different views from each scene of protected/unprotected and 100 random images.
• A new confusion matrix (Figure 14) and Figure 15 (Qualitative result of the various recovered watermarks from the rendered image of the protected model, rendered image of the unprotected model and random images) are added to further clarify the False Positive Detection Prevention.

In the appendix, new experiment using Unbound-360 Dataset is also included in Appendix A.10 to clarify the impact of dataset selection on ownership verification.

• We conducted an experiment using the 'Garden' and 'Kitchen' scenes from the Unbound-360 dataset. We found that the quality of the recovered watermark still maintained a high standard with an SSIM of 0.95 overall (Kitchen: watermark SSIM 0.9625; Garden: watermark SSIM 0.9526).

Finally, the font in all figures has been adjusted to ensure better clarity and comprehension.

We are glad that our responses helped and would like to thank you for raising the score of our paper.

Dear Reviewer,

Thank you for your valuable comments and thoughtful inquiries concerning our work. We appreciate the opportunity to respond to your concerns and are committed to providing additional clarity in our revised submission.

(a)Scalability to explicit NeRF representations:

• Although our proposed method was designed for implicit NeRF representations, it can be directly applied to explicit NeRF representations. This is because our black-box diffusion-based detector does not depend on the architecture of NeRF models but rather on the rendered images. Additionally, our white-box protection scheme only adds/modifies the normalization layer into the MLP layers. We will conduct a further investigation on directly applying our proposed methods to explicit NeRF representations in our future work.

(b)Clarification on "False Positive Detection Prevention" Section A.5 in Manuscript:

• We apologize for any confusion caused by the oversight in our writing. For clarification, both the protected and unprotected NeRF models employed scenes from the NeRF-Synthetic 360 dataset, including Lego, Chair, and Drum, as well as scenes from the LLFF-Forward dataset, namely Fern, Fortress, Room, and Flower. We are committed to incorporating this important information into our revised manuscript to enhance the clarity and quality of our work.

(c)Clarification on the Impact of Dataset Selection on Ownership Verification:

• For clarification, different datasets would not require to have different optimal thresholds. We are sorry for the misunderstanding. This section primarily centers around the crucial quality of extracted watermarks for asserting ownership, for which we chosen the threshold = 0.75 (SSIM). This choice is made empirically, aligning with Figure 13, where the watermark is still distinctly discernible for ownership claim. Additionally, our experimental findings (refer to Table 2) consistently demonstrate that the quality of the extracted watermark consistently surpasses 0.95 (SSIM) across diverse scenes within the NeRF-Synthetic and LLFF-Forward datasets.
• Additionally, we conducted an experiment using the 'Garden' and 'Kitchen' scenes from the Unbound-360 dataset [1], as suggested. We found that the quality of the recovered watermark still maintained a high standard with an SSIM of 0.95 overall (Kitchen: watermark SSIM 0.9625; Garden: watermark SSIM 0.9526). We have included these additional experimental results in our revised manuscript for better clarification.

[1] https://jonbarron.info/mipnerf360/

Your constructive feedback is invaluable, and we assure you that we will address these issues promptly in our revised manuscript to enhance the clarity and quality of our work. We appreciate the opportunity to address these concerns in our revised manuscript.

We have updated the Section 3.1 (black-box) and Section 4.3 (False positive detection). Specifically, we’ve mentioned in the updated Section 3.1:

• This work was inspired by (Ho et al., 2020) to propose a diffusion-based method to learn the detector d. This is because the nature of diffusion models to gradually add noise to the image provides natural robustness against various forms of image degradation attacks as the main objective of diffusion models is denoising. Thanks to the powerful denoising ability of diffusion models, our idea is to first convert the views x into Gaussian noises, then we denoise from these Gaussian noises into the watermark w. As a result, we can seamlessly translate between x and w through d.

Then in Section 3.2 (white-box), we added

• To achieve this, normalization layers were appended to the MLP implementation.

Also, we have updated the Section 4.3, as well as Appendix A.5 with experiment details:

• Both the protected and unprotected NeRF models in this experiment employed scenes from the NeRF-Synthetic 360 dataset, including Lego, Chair, and Drum, as well as scenes from the LLFF-Forward dataset, namely Fern, Fortress, Room, and Flower. To further understand why the false positives did not happen, we compute the histogram of 100 different views from each scene of protected/unprotected and 100 random images.
• A new confusion matrix (Figure 14) and Figure 15 (Qualitative result of the various recovered watermarks from the rendered image of the protected model, rendered image of the unprotected model and random images) are added to further clarify the False Positive Detection Prevention.

In the appendix, new experiment using Unbound-360 Dataset is also included in Appendix A.10 to clarify the impact of dataset selection on ownership verification.

• We conducted an experiment using the 'Garden' and 'Kitchen' scenes from the Unbound-360 dataset. We found that the quality of the recovered watermark still maintained a high standard with an SSIM of 0.95 overall (Kitchen: watermark SSIM 0.9625; Garden: watermark SSIM 0.9526).

Finally, the font in all figures has been adjusted to ensure better clarity and comprehension.

Dear Reviewer,

Thank you for your insightful comments and queries regarding our work on applying the diffusion model for black-box protection in NeRF. We appreciate the opportunity to address your concerns and provide additional clarification.

(a) Elaboration on Applying the Diffusion Model:

• Our initial findings using simple solutions such as DeepStega [1] and HiDDen [2] as well as more advanced methods such as StegaNeRF [3] demonstrated a few issues: (i) Ineffective watermark extraction on the rendered scene [1,2] due to the embedded watermark being smooth out during rendering (please refer to Figure 1 in the main paper); and (ii) The extraction of watermarks from the rendered scene is not robust to the introduction of noise [3], as it was trained under noise-free conditions (please refer to Figure 6 and Figure 7 in the main paper)

• As a result, we propose to use diffusion model as the foundation for our black-box approach. First, the nature of diffusion models to gradually add noise to the image provides natural robustness against various forms of image degradation attacks as the main objective of diffusion models is denoising (please refer to Figure 6 and Figure 7 in main paper). This resilience is a result of the inherent characteristics of the diffusion model. Second, the process of watermark embedding has minimal impact on the rendering quality of the NeRF models (please refer to Table 2 in main paper). Third, it can retrieve embedded watermarks within the rendered images of NeRF models (please refer to Table 2 in main paper).

• We sincerely apologize for the lack of an explanation of our motivation for applying the diffusion model as the black-box protection, we will add the above description to our revised manuscript.

(b) Simple Alternative Solution for Black-Box Protection of NeRF:

• Yes, some examples of simple alternative solutions are DeepStega [1] and HiDDen [2], which involve directly watermarking the training images before model training. However, as illustrated in Figure 1 of the main paper, the embedded watermark could not be extracted as it was smoothed out during training and rendering. Hence, these simple alternative methods are deemed not suitable for black-box protection of NeRF models.

[1] https://proceedings.neurips.cc/paper_files/paper/2017/file/838e8afb1ca34354ac209f53d90c3a43-Paper.pdf
[2] https://dl.acm.org/doi/10.1007/978-3-030-01267-0_40
[3] https://arxiv.org/abs/2212.01602

(c) Vulnerability of StegaNeRF to Gaussian Noise:

• The vulnerability of StegaNeRF [3] to Gaussian noise is due to the ideal conditions of the watermark detector's training procedure, which does not consider ``noise'' as a form of attack. Therefore, the watermark detector in StegaNeRF is implemented as a simple autoencoder-based detector. Consequently, when noise is added to the rendered scene as a form of removal attack, the features of the rendered image are compromised, causing the embedded watermark unable to be extracted effectively.

(d) Addressing Font Clarity in Figures:

• We sincerely apologize for the oversight regarding the unclear font in the figures, including legends and axis labels. To enhance the reader's viewing experience, we will enlarge the font in these elements to ensure better clarity and comprehension in the revised manuscript.

We hope these responses adequately address your concerns, and we are committed to implementing the necessary improvements in our revised manuscript.

We have updated the Section 3.1 (black-box) and Section 4.3 (False positive detection). Specifically, we’ve mentioned in the updated Section 3.1:

• This work was inspired by (Ho et al., 2020) to propose a diffusion-based method to learn the detector d. This is because the nature of diffusion models to gradually add noise to the image provides natural robustness against various forms of image degradation attacks as the main objective of diffusion models is denoising. Thanks to the powerful denoising ability of diffusion models, our idea is to first convert the views x into Gaussian noises, then we denoise from these Gaussian noises into the watermark w. As a result, we can seamlessly translate between x and w through d.

Then in Section 3.2 (white-box), we added

• To achieve this, normalization layers were appended to the MLP implementation.

Also, we have updated the Section 4.3, as well as Appendix A.5 with experiment details:

• Both the protected and unprotected NeRF models in this experiment employed scenes from the NeRF-Synthetic 360 dataset, including Lego, Chair, and Drum, as well as scenes from the LLFF-Forward dataset, namely Fern, Fortress, Room, and Flower. To further understand why the false positives did not happen, we compute the histogram of 100 different views from each scene of protected/unprotected and 100 random images.
• A new confusion matrix (Figure 14) and Figure 15 (Qualitative result of the various recovered watermarks from the rendered image of the protected model, rendered image of the unprotected model and random images) are added to further clarify the False Positive Detection Prevention.

In the appendix, new experiment using Unbound-360 Dataset is also included in Appendix A.10 to clarify the impact of dataset selection on ownership verification.

• We conducted an experiment using the 'Garden' and 'Kitchen' scenes from the Unbound-360 dataset. We found that the quality of the recovered watermark still maintained a high standard with an SSIM of 0.95 overall (Kitchen: watermark SSIM 0.9625; Garden: watermark SSIM 0.9526).

Finally, the font in all figures has been adjusted to ensure better clarity and comprehension.