Geometry Cloak: Preventing TGS-based 3D Reconstruction from Copyrighted Images

Dear reviewer,

Thank you for your valuable feedback and suggestions.

Response to W1: Extending to other methods

Our method is designed to utilize the explicit geometry feature in GS-based single views to 3D methods, which is fragile and susceptible to disturbances in the reconstruction process. Thus, our method can work on various GS-based single views to 3D methods as the explicit geometry feature is a necessary element for 3DGS. Before our submission, other GS-based single images to 3D methods were not formally peer-reviewed. While we primarily focus on TGS as it is peer-reviewed during our research, our approach can be extended to other recent methods. To illustrate this, we evaluate our technique on the recently proposed LGM [1], accepted by ECCV 2024. The results are appended in Fig. R1 and Tab. R1 of the rebuttal page. By simply adapting key design elements, our approach shows promising results on LGM as well, underscoring its potential in manipulating other GS-based single-image-to-3D reconstruction methods.

Response to W2: Typos

We appreciate you pointing out these typos. The typos in our paper will be carefully corrected.

Response to Q1: Observation viewing direction

1. In the paper, we present the results from the top view. In theory, our method can be applied to arbitrarily selected viewing directions. The algorithm will not fail because we are simply mapping the projected point cloud at direction θ, then using this projected point cloud to present the watermark. We present the results of embedding watermarks at different views in Fig. R5 and Tab. R1 (right). The experimental results indicate that our method can work well when embedding watermarks in different directions.

2. Our watermarks are embedded at a certain location, which is sufficient for copyright verification of 3D models. As provided visual results of the watermark from a certain angle from Fig. R5 (Left), viewing the 3D model from other unsuitable perspectives will result in low-quality fragmented information. Besides, to verify copyright, users can verify from any angle, as long as it is identitable and consistent with the pre-embedded watermarks that can be matched. We will incorporate these results in our next version for better clarification of the property of the geometry cloak.

Response to Q2: Multiple characters

1. Although the geometry cloak was not specifically designed for multiple characters, our method can still ensure identifiable results when the watermarks are multiple characters. We provide visual results in Fig. R5 (left), which remain promising when there are fewer than four characters. This suggests that the geometry cloak holds promising potential regarding watermark capacity.

2. Besides, compared to previous methods of image cloaking [2,3] against the diffusion model, which can only use artist-style as watermarks, the watermarks embedded via our method are much more identifiable.

Response to Q3: Generalized way to do watermarking/cloaking

1. We have discussed how our method can be generalized to other single image to 3D reconstruction algorithms in response to W1. Preliminary experimental results on LGM [1] are provided in Fig. R1 and Tab. R1, indicating our method is promising in manipulating other GS-based single-image-to-3D reconstruction methods.

2. To the best of our knowledge, this is the first paper to reveal a vulnerability in the GS-based 3D reconstruction and effectively manipulate the generated results using this vulnerability. The results generated via GS-based methods can be manipulated through adversarial attacks on geometric features, which is an issue worthy of attention as 3DGS are rather popular today. More studies should be conducted to protect image/model owners' copyrights and prevent potential malicious attackers. Our method provides a novel perspective on adversarial perturbations, enabling copyrighted images to prevent misuse by GS-based methods. which could potentially encourage related research in enhancing the robustness of 3D reconstruction and addressing AI privacy/safety issues.

Response to Q4: Details of obtaining masks

We used SAM to obtain a segmentation of the images to ensure that the perturbations were only added to the objects. We will provide a detailed generation process in our next version of the paper, and experimental codes of geometry cloak will be released to ensure reproducibility.

We look forward to discussing with you in the upcoming discussion phase to clarify things we may neglect.

[1] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. In ECCV 2024.

[2] Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples. In ICML 2024.

[3] Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models. In USENIX 2023.

We would like to clarify the misunderstanding of the settings of LGM and point cloud encoder. LGM[1] and Gamba[3] do have an encoder to get point cloud.

Settings of LGM [1]

For LGM, it takes 4 input views obtained from ImageDream/MVDream. Therefore, to verify the effectiveness of our method in undermining the generated 3D results, we directly optimize adversarial perturbations on these four input images. LGM does have a point cloud encoder part to get the point cloud, as 3DGS needs a point cloud to represent the 3D scene. Specifically, please refer to Line 109 of LGM/core/model.py: https://github.com/3DTopia/LGM/blob/main/core/models.py

109        pos = self.pos_act(x[..., 0:3]) # [B, N, 3]
110        opacity = self.opacity_act(x[..., 3:4])
111        scale = self.scale_act(x[..., 4:7])
112        rotation = self.rot_act(x[..., 7:11])
113        rgbs = self.rgb_act(x[..., 11:])
114        gaussians = torch.cat([pos, opacity, scale, rotation, rgbs], dim=-1) # [B, N, 14]

In our rebuttal settings, we directly target and perturbed the pos property of Gaussians, Line 109 pos = self.pos_act(x[..., 0:3]) # [B, N, 3], which is the center of 3DGS. (i.e., point cloud).

Adapting to LRM [2] and Gamba [3]

For NeRF-based methods like LRM [2], as it does not require explicit geometry features, our method may not be suitable. We didn't experiment with Gamba as it has not been formally peer-reviewed. However, it also has a point cloud encoder to estimate the point cloud (Line 130, position).

https://github.com/kyegomez/Gamba/blob/main/gamba_torch/main.py

130     # Position, opacity, color
131     position = self.position_layer(features)
132     opacity = self.opacity_layer(features)
133     color = self.color_layer(features)

Gamba is still required to estimate the point cloud.

Thus, we do not need to change our motivation statement and algorithm description. The GS-based method must have a point cloud encoder to represent the 3D scene.

[1] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV 2024.

[2] LRM: Large Reconstruction Model for Single Image to 3D. ICLR 2024

[3] Gamba: Marry Gaussian Splatting with Mamba for Single-View 3D Reconstruction. arxiv 2024

Dear reviewer,

Thanks for your valuable feedback.

Our experimental findings indicate that explicit geometry features can be effectively utilized to protect ownership in GS-based tasks, despite TGS being a recent development. Due to the explicit feature of GS, GS-based image-to-3D methods inherently require similar explicit geometry features. This insight suggests that our approach can be extended to these methods, potentially enhancing security and ownership protection in single-image-to-3D applications and inspiring future research in this area.

We strongly agree with you on the concerns of the generalization ability. We will incorporate these additional experiments about our effectiveness on other GS-based approaches in the final version based on your valuable suggestions. We look forward to addressing your further concerns during this discussion.

Thanks for your rebuttal, which clarifies my questions regarding viewing direction and multiple characters watermarking pattern. However, I still have significant concerns regarding the generalization capabilities of the current version of the work.

The authors suggest that their method can generalize to LGM by simply adapting key elements. However, the rebuttal materials do not provide sufficient details about the specific algorithmic adjustments required for LGM. LGM employs a fundamentally different architecture to achieve single image to 3D generation. Specifically, it first utilizes off-the-shelf models like ImageDream or MVDream to synthesize multi-view images, followed by an asymmetric U-Net architecture with cross-view self-attentions to construct Gaussian features. Given these foundational differences, it is unclear how the authors' approach could be seamlessly adapted to LGM[1]. For instance, the method described in the paper relies on a point cloud encoder from TGS and uses the Chamfer distance between two-point clouds as a loss to craft adversarial perturbations (c.f. Algorithm 1 and Figure 2). These critical components do not translate directly to LGM[1], as it does not incorporate point cloud encoders. If authors wants to expand their method to other single image to 3D models, they would have to fundamentally change their motivation statement and algorithm description, which will make a huge modification to their current version of paper.  

In the rebuttal, the authors mention that their method utilizes geometry features in GS-based single views to 3D methods, which I have reservations about this. Their initial claim is that the method is specifically tailored for Triplane Gaussian Splatting. It is important to note that different approaches to single-image-to-3D conversion employ significantly diverse network architectures (e.g., LRM[2] uses transformers, Gamba[3] uses Mamba), as a result, their geometry feature space are highly distinct. Ensuring image copyright protection while considering generalization across these varied models is indeed challenging, and it may not be accurate to make such a broad generalization claim for their methodology design.

TGS[4] is a recently accepted paper in this field, but it is not a representative technique, and its reconstruction results are not among the best. The likelihood of it being widely used in real-world applications remains highly questionable. The key point here is that applying adversarial attacks to a specific model is too limited to be considered a broadly useful copyright protection method. Similarly, designing adversarial attacks tailored to a specific method may be too narrow in scope for a NeurIPS paper.

Reference [1] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV 2024.

[2] LRM: Large Reconstruction Model for Single Image to 3D. ICLR 2024

[3] Gamba: Marry Gaussian Splatting with Mamba for Single-View 3D Reconstruction. arxiv 2024

[4] Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers. CVPR 2024

Dear reviewer,

Thank you for your recognize and valuable suggestions.

Response to W1: Impact on image quality

Our geometry cloak is designed to be invisible so legitimate users can have visually consistent results with the original image quality. All perturbations are controlled within a certain budget $\epsilon$. Hence, we neglect to provide the impact on image quality. For a more comprehensive evaluation of the method's impact on image quality, we provide quantitative metrics of the similarity between protected and unprotected images as below:

Compared to previous methods of adversarial attacks on image features, our proposed geometry cloak ensures higher invisibility while effectively disturbing the reconstruction results. Besides TGS, our method also keeps the invisibility for LGM[1], as shown in Tab. R1 of the rebuttal page.

Response to W2: Viewing direction and Adv. triplane

(View-specific PGD) We conduct experiments from other perspectives (side/front). The experimental results indicate that the reconstructed results can still be effectively manipulated from these perspectives, demonstrating the effectiveness of our method. We also provide results of embedding multiple letters further to demonstrate the view-specific PGD performance (Fig. R5 and Tab. R1).

(Effectiveness of point cloud + triplane latent features ) Following your suggestions, we conduct experiments by combining attacking point cloud and triplane latent features. The results in Fig. R2 (a) indicate that attacking only the tri-plane will affect the visual quality of the reconstruction. There is no significant change after combining attacks on the point cloud. Future work could focus on studying the components in the 3D reconstruction process that are vulnerable to disturbances.

Response to W3: Minor Issues

We appreciate you pointing out these typos. These typos in our paper will be carefully corrected in the next version of our paper.

[1] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. in ECCV 2024.

Dear reviewer,

Thank you for your recognition and valuable suggestions.

Response to W1: Extending to other methods

Our method is designed to utilize the explicit geometry feature in GS-based single views to 3D methods, which is fragile and susceptible to disturbances in the reconstruction process. Thus, our method can work on various GS-based single views to 3D methods as the explicit geometry feature is a necessary element for 3DGS. Before our submission, other GS-based single images to 3D methods were not formally peer-reviewed. While we primarily focus on TGS as it is peer-reviewed during our research, our approach can be extended to other recent methods. To illustrate this, we evaluate our technique on the recently proposed LGM [1], accepted by ECCV 2024. The results are appended in Fig. R1 and Tab. R1 of the rebuttal page. By simply adapting key design elements, our approach shows promising results on LGM as well, underscoring its potential in manipulating other GS-based single-image-to-3D reconstruction methods.

Response to W2: Impact on image quality

Our geometry cloak is designed to be invisible so legitimate users can have visually consistent results with the original image quality. All perturbations are controlled within a certain budget $\epsilon$. Hence, we neglect to provide the impact on image quality. For a more comprehensive evaluation of the method's impact on image quality, we provide quantitative metrics of the similarity between protected and unprotected images as below:

Compared to previous methods of adversarial attacks on image features, our proposed geometry cloak ensures higher invisibility while effectively disturbing the reconstruction results. Besides TGS, our method yields consistent invisible results in LGM[1], demonstrating the generality and concealment of our method (as shown in Tab. R1 of the rebuttal page).

Response to W3: Computational resources

Our method only optimizes the invisible cloak, which does not require a lot of computational resources (~8GB GPU memory). We provide the convergence curve of the loss during the optimization process on the rebuttal page. In less than 50 secs, with a single V100 GPU, we can achieve protection for each image. The computational resources will be incorporated in the next version of our paper.

[1] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. In ECCV 2024.

Dear reviewer,

Thank you for your recognition and valuable suggestions.

Response to W1: Extending to other methods

Our method is designed to utilize the explicit geometry feature in GS-based single views to 3D methods, which is fragile and susceptible to disturbances in the reconstruction process. Thus, our method can work on various GS-based single views to 3D methods as the explicit geometry feature is a necessary element for 3DGS. Before our submission, other GS-based single images to 3D methods were not formally peer-reviewed. While we primarily focus on TGS as it is peer-reviewed during our research, our approach can be extended to other recent methods. To illustrate this, we evaluate our technique on the recently proposed LGM [1], accepted by ECCV 2024. The results are appended in Fig. R1 and Tab. R1 of the rebuttal page. By simply adapting key design elements, our approach shows promising results on LGM as well, underscoring its potential in manipulating other GS-based single-image-to-3D reconstruction methods. For NeRF-based methods like LRM [2], our work could inspire more research to study the vulnerability layers in these frameworks.

TGS combines explicit geometry features and implicit tri-plane representations to achieve accurate and detailed reconstructions, representing a cutting-edge approach to 3D object reconstruction from single-view images. Before our submission, LGM had not been officially peer-reviewed. Thus, we choose TGS as our experiment subject to verify the effectiveness of the geometry cloak. However, this does not affect the generality of our approach in GS-based methods. As discussed in the paper, GS-based methods require explicit geometric features to represent 3D models. The process of obtaining this geometric feature is easily manipulable with proper adversarial perturbations. We will extend the experimental results in LGM to our paper to further clarify the effectiveness of the geometry cloak.

Response to W2: Robustness against image compression

To verify the robustness of our method, we experiment with some common image operations, including Gaussian noise, JPEG, etc. Under mild operations, the geometry cloak cannot be removed, and the geometric features of the generated 3D result are still disturbed. With higher operations, the geometry cloak can be affected. However, the quality of the view image has been severely compromised, leading to poor visual effects in the reconstructed results, making these protected images unusable.

We present more results about robustness against common image operations in Tab. R1 of the rebuttal page.

[1] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. In ECCV 2024.

[2] LRM: Large reconstruction model for single image to 3d. In ICLR 2024.

Dear reviewer,

We express our gratitude for your recognition and valuable suggestions.

Response to W1: Extending to other methods

Our method is designed to utilize the explicit geometry feature in GS-based single views to 3D methods, which is fragile and susceptible to disturbances in the reconstruction process. Thus, our method can work on various GS-based single views to 3D methods as the explicit geometry feature is a necessary element for 3DGS. Before our submission, other GS-based single images to 3D methods were not formally peer-reviewed. While we primarily focus on TGS as it is peer-reviewed during our research, our approach can be extended to other recent methods. To illustrate this, we evaluate our technique on the recently proposed LGM [1], accepted by ECCV 2024. The results are appended in Fig. R1 and Tab. R1 of the rebuttal page. By simply adapting key design elements, our approach shows promising results on LGM as well, underscoring its potential in manipulating other GS-based single-image-to-3D reconstruction methods.

Response to W2: Tendency of performance degradation

We are very grateful for you pointing out this phenomenon.

Tab.1 in the main paper aims to demonstrate that even with larger budgets, other perturbation methods (Gaussian noise and adv. image features) still do not effectively undermine the reconstruction results. To understand the tendency of perturbation, we provide more experimental results under a wider range of budget $\epsilon$.

Besides this, we provide visual results of perturbed results in Fig. R2 (b-c) and curves for different $\epsilon$ in Fig. R3 of the rebuttal page.

The experimental results reveal a sensitivity to smaller budget $\epsilon$ (<2) values, while larger budget $\epsilon$ (>2) values demonstrate insensitivity due to the already severe disruptive effects. We recognize and thank your suggestions that understanding the tendency of performance degradation is crucial; these results will be extended to the next version of our paper.

Response to Q1: Slope of performance deviation

We provide the slope of performance deviation in Fig. R2 of the rebuttal page, which also exhibits a negative performance slope.

Response to Q2: Results under stronger perturbation

We provide visual results when a random noise ($\epsilon$ =16, 32) is applied in Fig R2 (c) of the rebuttal page. Even with this larger budget $\epsilon$, The geometry pattern of the reconstructed 3D model has no obvious change. This indicates that the process of obtaining geometry features through TGS is robust to noise. We will incorporate these results into our paper.

Our method shows unobvious changes in disturbance results after $\epsilon$ > 4 (as discussed in W1). We present the performance slope in Fig. R3 of the rebuttal page.

Response to Q3: Ablation study of PGD

We demonstrate the convergence of loss under different values of epsilon in PGD (Fig. R3). We also provide the results of PGD attacks from different perspectives (Fig. R5 and Tab. R1) and multiple characters as watermarks (Fig. R5) on the rebuttal page. We look forward to discussing with you in the upcoming discussion phase to clarify things we may neglect.

[1] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. In ECCV 2024.

I am satisfied with the authors' response and appreciate their effort to address my concerns regarding the effectiveness of noise perturbation depending on the privacy budget.

Additionally, It is impressive that performance decreases as perturbation increases. While the authors notice that noise to the Tri-Plane do not produce the expected results, the reason for this phenomena remains unexplained. Although they have indicated this as future works, understanding the phenomena seems crucial in 3D reconstruction since learned Tri-Plane appears to also contain geometry information. Given these issue, I maintain my original score.