Machine Unlearning in 3D Generation: A Perspective-Coherent Acceleration Framework

Thank you for your constructive comments! We have carefully considered your suggestions and would like to provide the following clarifications. We'll update the preprint in later revisions.

---

Response to the weaknesses

W1. Although this paper is almost the first in the field of machine unlearning in novel view synthesis, the authors have not compared with any baselines, which is not very solid.

As you rightly pointed out, our work is the first to explore 3D unlearning, and thus our approach can be seen as a flexible framework that integrates various 2D unlearning methods with dynamic skipping acceleration to achieve effective unlearning in diffusion-based 3D models. As mentioned in our second response, we have demonstrated how this framework can be adapted to other 3D models, and we are actively exploring the transferability of more 2D unlearning methods. While the number of unlearning baselines presented in this work is currently limited, our goal is to establish a benchmark that will serve as a foundation for future research and inspire the community to further investigate novel unlearning strategies in both 2D and 3D settings. We believe this will open new avenues for innovation in the field, despite the current scope of our comparisons being limited.

W2: The authors does not show 3D meshes or rendered videos in supplementary results. I think using solely single-view images to demonstrate the results are not very good for demonstrate the full 3D results of this work.

Thank you for your feedback. In Figure 2 of the appendix, titled "Demonstration of Multi-perspective Effects on the Forget Set for Different Unlearning Tasks," we provide the multi-view results for each unlearning task shown in Figure 3 of the main text. Due to space and aesthetic considerations, we did not include the multi-view images in the main body. Additionally, in Appendix L176, Section 2.3: 3D Reconstruction Demonstrations, we have included screenshots of the 3D depth-rendered videos. Unfortunately, the video itself was not uploaded with the supplementary materials due to oversight. However, please rest assured that we can provide the video in the future, demonstrating the rendered 3D video and its unlearning 3D consistency.

---

Response to the questions

Q1: How about the failure cases? What is the typical failure cases look like?

A1: Indeed, there can be some failure cases, particularly when the quality of the override image is low. In such cases, the depth and spatial information from the viewpoint are unclear, and the semantic scalability is weak, which makes successful unlearning more challenging. For instance, when we attempted to unlearn by placing a vase on a side-view distorted chair, we found it difficult to achieve optimal results despite multiple adjustments. We will include a detailed analysis of these failure cases in the supplementary materials.

Q2: How long does it take to generated a single image?

A2: As shown in table below the baseline represents the time (1.1000 seconds) taken by Zero123 to sample an image without using dynamic skipping via interpolation. The other columns show the sampling times with different skip steps, along with the speedup ratios compared to the baseline.

Sample Time (s) for Different Methods

---

Response to the limitations

L1. The evaluation is insufficient. In table 2, the authors only compared using 10 hand-crafted cases. I think a large scale benchmark will be more satisfactory.

Thank you for your valuable feedback. We acknowledge that the evaluation in Table 2 is based on a limited set of 10 hand-crafted cases. However, it is important to emphasize that our work is the first to explore 3D machine unlearning, which introduces novel challenges and contributions to the field. In addition to the experiments in Table 2, we have also conducted further experiments to assess the generalization ability of our method across different 3D tasks, including transforming a Minion into a black-and-white style, converting a cartoon green airplane into a solid white airplane, blurring the background of the Minion (to achieve privacy protection by limiting a certain viewpoint), unlearning the airplane into a dumbbell, and placing a blanket on a sofa, among other tasks. We will provide additional visual results in the future. These additional experiments demonstrate the robustness of our approach in diverse settings. While we agree that a large-scale benchmark would provide further insights, we focused on a smaller, controlled set initially to validate the core principles. We are planning to extend our evaluation with a larger-scale benchmark in future work to strengthen the validation of our method.

L2. The authors proposed 4 lossed in total. However, how to balance these loss functions is not clearly explained or supported by experiments in the paper.

• Balancing Loss Functions: The two parameters $(\lambda)$ and $(\mu)$ were omitted in Equation (14), with their settings provided in Appendix Table 1: Hyperparameters for Fake Score and Generator. The equation should be updated as: $\mathcal{L}{fn} = \lambda \mathcal{L}{fn\ remain} + \mu \mathcal{L}_{fn\ forget}$

• Loss Function Combination: The loss functions are balanced through the adjustable hyperparameters $(\lambda)$ and $(\mu)$, which control the relative importance between the generator's loss (which maintains the generation quality of the remaining categories) and the fake score network's loss (which drives the forgetting of the target category). Specifically, $(\lambda)$ emphasizes the quality of remaining category generation, while $(\mu)$ controls the strength of the forgetting process.

• Alternating Optimization: To ensure balance, we adopt an alternating optimization strategy, as shown in Appendix Algorithm 2. The fake score network is updated first, followed by the generator, ensuring an effective balance between the loss functions of both.

Additionally, $(\lambda)$ and $(\mu)$ are insensitive to different cases; after simple tuning, they yield stable results across multiple cases.

Thank you for your constructive comments! We have carefully considered your suggestions and would like to provide the following clarifications. We'll update the preprint in later revisions.

---

W1. The presentation of the core method could be improved with a figure summarizing all key components.

Thank you for your suggestion. Indeed, we currently do not have a comprehensive diagram that presents all the key components. However, in Algorithm 2: Unlearning via Dynamic Acceleration with Remain and Forget Losses in the appendix, we demonstrate how all the critical components are interconnected. Based on your feedback, we will add a diagram to visually present the entire framework. This diagram will include all the requirements mentioned in Algorithm 2, as well as the balancing relationships among the four loss functions, and it will also incorporate Algorithm 1: Dynamic Skipping via Interpolation from the appendix.

---

W2: I wonder if the proposed unlearning method in 3D generation can be applied to other 3D generation models except Zero123/Zero123-XL. Because there are many other 3D generation models, for example: transformer-based 3D generation models, LRM, LGM, GS-LRM, and etc; diffusion-based 3D generation models, SV3D, and etc.

Thank you for your suggestion. Due to time constraints, we have not included sufficient experiments in this area. In fact, we have migrated to Free3D:

To validate the portability, we have conducted adaptation experiments on the Free3D model. We found that Free3D's viewpoint control mechanism essentially relies on generating structured conditional signals (such as pose encoding and ray embedding) from the input camera angles (e.g., azimuth and elevation) through deterministic transformations (e.g., trigonometric functions, spherical coordinate mapping). This characteristic is highly compatible with the angle-based input control approach used in our Unlearn framework.

Based on this, we extracted the UNet backbone network from Free3D and integrated it into our training framework. During the training process:

• We followed the original preprocessing pipeline of Free3D, combining multiple viewpoint angles and converting them into contextual conditions (including cross-attention inputs and ray embeddings) that match its format.

• We used a multi-view alignment strategy for supervision, where the source object (e.g., Minion) → target object (e.g., Doraemon) is aligned. This forces the model to output reasonable images of the target object from any viewpoint when given the source object as input.

After training, we re-integrated the fine-tuned UNet into the original Free3D inference pipeline. The experimental results show:

• When the source object is input, all generated viewpoints consistently reflect the appearance of the target object, and viewpoint consistency is well-maintained, successfully achieving "unlearning."

• For other objects that were not part of the training (e.g., cars, chairs), the generated quality was comparable to the original model, with no performance degradation.

Additionally, our dynamic skipping acceleration strategy based on viewpoint similarity also demonstrates good generality. Since Free3D uses iterative samplers like DDIM, and viewpoint changes are continuous, this acceleration method can be directly applied to its inference process, significantly improving efficiency while maintaining generation quality.

In the future, we will supplement the results with visualizations using the Free3D model.

---

W3: More visualization results should be provided. The current version is not convincing enough. For example, the generated results for different unlearning tasks provided in Figure 3 only provide one single view. Since this paper aims to handle the unlearning problem in 3D generation, the authors should also provide the rendered results (i.e. rendered videos with higher resolution) on different views to prove the multi-view consistency.

Thank you for your feedback. In Figure 2 of the appendix, titled "Demonstration of Multi-perspective Effects on the Forget Set for Different Unlearning Tasks," we provide the multi-view results for each unlearning task shown in Figure 3 of the main text. Due to space and aesthetic considerations, we did not include the multi-view images in the main body. Additionally, in Appendix L176, Section 2.3: 3D Reconstruction Demonstrations, we have included screenshots of the 3D depth-rendered videos. Unfortunately, the video itself was not uploaded with the supplementary materials due to oversight. However, please rest assured that we can provide the video in the future, demonstrating the rendered 3D video and its unlearning 3D consistency.

---

W4: The number of unlearning baselines is too few to prove the superiority. In Table 2, the authors only provide the comparison between the model with or without the proposed unlearning method. Is there any 2D unlearning methods that can be transfered to Zero123/Zero123-XL directly?

As you rightly pointed out, our work is the first to explore 3D unlearning, and thus our approach can be seen as a flexible framework that integrates various 2D unlearning methods with dynamic skipping acceleration to achieve effective unlearning in diffusion-based 3D models. As mentioned in our second response, we have demonstrated how this framework can be adapted to other 3D models, and we are actively exploring the transferability of more 2D unlearning methods. While the number of unlearning baselines presented in this work is currently limited, our goal is to establish a benchmark that will serve as a foundation for future research and inspire the community to further investigate novel unlearning strategies in both 2D and 3D settings. We believe this will open new avenues for innovation in the field, despite the current scope of our comparisons being limited.

---

Q1 [Extension to other 3D generation models?] Please refer to W1.

Q2 [More qualitative results.] Please refer to W3.

Q3 [Unlearning baselines.] Please refer to W4.

---

Thank you for your constructive suggestions and recognition of our work! We have carefully considered your advice and would like to provide the following clarifications.

---

About the problem setup

Q1. Is unlearning conducted case by case? Namely, will a separate unlearning be needed for each 3D generation?

A1: Yes, the unlearning process for each class needs to be handled individually, but in practice, a single representative image from the class can serve as the "representative" for the entire class.

If a front-facing Minion image in a standing pose is used as the forget image and a corresponding pink-stylized Minion as the override image, the model will transform any Minion, regardless of variations (e.g., eye count, height, body shape), into the pink style, thus achieving machine unlearning.

As shown in Fig.2 of the main text, unlearning the model with the front-facing standing Minion (from Fig.3) as the forget image and the pink Minion as the override image allows different Minion variations to transform into the pink style using this unlearned model.

Unfortunately, we didn’t explicitly mention the generality of this approach. We tested five additional Minion images with varying features, and all successfully transformed into the expected pink style, with overalls and eyes turning pink. This demonstrates the method's effectiveness across different input images of the same class. We will provide supplementary visual results in future submissions.

Q2: For an unlearning, how many "target" images will be given? Based on Eq. (3), it seems like mulitview images need to be provided?

A2: This part may not be clear enough. In fact, during unlearning, we require one "forget image," one "override image" (i.e., "target"), and multiple sampled angles. Since the zero123 model can generate synthesized images from a single image across multiple viewpoints, although we only have one target image, we actually obtain as many target images as the number of sampled angles.

As mentioned in Question a, unlearning can be achieved with just one override image (i.e., the target) and one forget image. The forget image represents the class that we want the model to "forget," while the override image (target image) represents the class that the forget image will eventually be transformed into.

The remain image is primarily used to prevent the model from becoming overly similar to the target image, so as not to affect the generation of other non-target images. The dataset for the remain set comes from Shapenet's rendered results, which include images from multiple categories and different viewpoints.

Q3: L237 mentioned We conduct experiments on three types of data but where are the results for the 2nd dataset:

(L238) Rendered 3D objects from Objaverse 1.0, including sculptures, traffic barriers, and fire hydrants

A3: The experimental results involving the rendered 3D objects from Objaverse 1.0 (including sculptures, traffic barriers, and fire hydrants) are qualitatively demonstrated in Fig. 3 and quantitatively analyzed in Table 2.

This dataset serves two primary purposes in our experiments:

1. Acting as the forget images for 10 unlearning tasks

2. Comparing the performance of Zero123 unlearning with and without dynamic skipping via interpolation, as measured by SSIM and LPIPS on both forget and remain sets.

Taking the yellow car transformation task (the first task in Table 2 and Fig. 3) as an example:

• The forget set metrics evaluate the unlearning effectiveness by comparing the output (from original car images fed into the unlearned model) against yellow car ground truth using SSIM and LPIPS.

• The remain set metrics assess model preservation capability by averaging SSIM and LPIPS scores between outputs (from the other 9 categories' input images) and their respective ground truths.

This experimental design demonstrates that our approach successfully achieves target unlearning while maintaining generation quality on unseen data.

Q4: I am quite confused about the 3rd dataset:

(L329) A subset of five Objaverse models rendered from 24 viewpoints, each with 35 images, totaling approximately 4,200 ground-truth images.

A4: We would like to clarify the following:

• We selected five 3D models from the Objaverse dataset and rendered each from 24 viewpoints, spaced 15° apart, covering a full 360° rotation. Each viewpoint generated 35 images, totaling ~4,200 ground-truth images. Apologies for the previous confusion.

• The relevant results are in Supplementary Materials (L184, Sec. 2.4, Effect of View-consistent Acceleration without Unlearning), where we validate dynamic skipping via interpolation. This method decouples acceleration from unlearning, showing that even without unlearning, generation time can be reduced and image quality improved.

• A comparative analysis with varying reference viewpoints and skip steps shows that our framework produces higher-quality, more efficient images than Zero123. Qualitative results are in Appendix Figure 4, and quantitative results in Appendix Table 4.

We hope this clarification resolves any misunderstanding. Thank you for your understanding and valuable feedback.

Q5: In Tab. 1, does 'X Ref Angles' refer to the given "target" images? If so, why is the performance of '8 Ref Images', i.e., most reference images, the worst? Does this mean that the proposed approach is ineffective in unlearning?

A5: Yes, 'X Ref Angles' refer to the given "target" images. And we found that using 4 reference angles yields better results than using 8. This phenomenon can be explained from the following perspectives:

• Authenticity of Reference Angles. The 8 reference images are synthesized from a single view using Zero123, not real multi-view ground truth. Since Zero123 may generate inaccuracies for distant viewpoints, these extra reference angles could introduce noise rather than useful information.

• Stability of Local Viewpoints. With 4 reference angles, adjacent reference views are 90° apart, allowing reliable interpolation by combining information from both angles during inference. With 8 reference angles, adjacent views are only 45° apart, leading to smaller geometric differences and redundant information, which complicates optimization.

• Rationality of Supervision Signal and Overfitting Risk. As mentioned in L61, full multi-view supervision is unnecessary, as the model should perform inference rather than rigid fitting. Four reference angles provide sufficient geometric priors, whereas more closely spaced reference angles may cause the model to overfit to the distribution of synthetic images (e.g., texture details) while neglecting critical inter-view geometric relationships, ultimately degrading reconstruction quality.

In conclusion, a smaller number of highly relevant reference angles (4) is more effective than a larger number of noisy viewpoints (8).

---

About dynamic skipping via interpolation (Sec. 3.2)

Q1: Can authors provide both quantitative and qualitative results for the following experiments (no learning needed)

A1: Yes, we have already conducted experiments to assess the fidelity of the proposed dynamic skipping, and the details are provided below:

• Table 1 shows static skipping results using the Min10 dataset, which includes multi-view images of 10 Minion shapes. A front-view forget image was paired with a corresponding pink-styled override image. The setup includes nine configurations with 3, 4, and 8 reference angles and step sizes of 8, 12, and 16, plus one baseline (non-accelerated), all trained for 8 epochs. Training time per epoch and the acceleration ratio relative to the baseline are provided.

• Due to an oversight, we did not provide a detailed explanation of the $t_{upper}$ and $t_{lower}$ values in equation (10). When calculating the data presented in Table 1, we set $t_{upper} = t_{lower}$, and both were set to the number of steps skipped in Table 1. Based on the image quality of the unlearned results (calculated by comparing to the ground truth using metrics such as SSIM, LPIPS, and delta PSNR), we found that, with 4 reference angles and 12 skipped steps, we achieved the optimal unlearned image generation effect while reducing training time compared to the baseline.

• In Appendix L36, Section 1.2, we discuss optimizing dynamic skipping via interpolation. With 4 reference angles, we use a threshold strategy: if the angle difference between the sample and any reference is ≤ 20°, we set skip steps to $t_{upper} = 16$; otherwise, $t_{lower} = 12$. A detailed explanation of the choice of the 20-degree threshold can be found in Section L36 of the appendix. The goal of the dynamic approach is to further improve both the generation quality and training efficiency while achieving superior unlearn results compared to the baseline, thus reducing the overall training time.

• In Appendix L184, Section 2.4, we focus on dynamic acceleration effects. Applying this framework to generation, results in Figure 4 and Table 4 show that leveraging 3D angle continuity and neighboring angles improved reliability, quality, and reduced generation time. Additional data further validate the performance gains from dynamic acceleration. We have included additional data to further validate the improvement in performance brought by dynamic acceleration.

---

This section specifically focuses on dynamic skipping via interpolation, which is independent of the unlearning process.

I am quite confused about the author's message. In the Appendix Sec. 1.2.2, it clearly states that Experiments were conducted on the Yellow Car unlearning task (L68). So the experiments are conducted 1) only on a single object; and 2) coupled with the unlearning process instead of pure "dynamic skipping via interpolation". This contradicts the above statement.

We sincerely appreciate your positive feedback and recognition of our work. Your acknowledgment of the importance of machine unlearning in 3D generation is truly encouraging, and we are grateful that you highlighted the timeliness and relevance of this research. We also value your comments on our extension of the SFD framework to 3D generation and the proposed acceleration strategy for multi-view generation. Your feedback reinforces the significance of our approach, and we are glad that it resonates with you.

---

W1: Lack of Validation on Unseen Data. The paper lacks evaluation on an independent validation set. It is recommended that the authors include an additional unseen set (e.g., GPTEval3D) beyond the defined remain $D_r$, $D_o$, $D_f$ sets to assess whether the model’s general generation capabilities are adversely affected.

Thank you for your suggestion. The evaluation on an independent validation set is given in the main paper and supplementary. Below, we give additional details on the quantitative results in Table 2 and the qualitative results in Figure 3.

• First, it is important to clarify that the purpose of $D_r$ - remain set is to ensure that, during the unlearning process, we not only forget the features of the "forget image" class but also ensure that the generation of other classes is not affected. The remain set dataset comes from Shapenet's rendered results, covering multiple categories and different angles of images.

• In Figure 3, all input images are rendered from Objaverse 1.0, and the remain set does not contain any images rendered from Objaverse 1.0. Taking the Cherry to Banana task as an example, $D_f$ is a photo of a cherry, $D_o$ is a photo of a banana, and remain set consists of images from Shapenet. Therefore, for the Cherry to Banana unlearning task, all input images except for "cherry" (such as car, chair, sculpture, etc.) belong to the "unseen set."

• Additionally, we appologize for some ambiguity in the presentation of Table 2, which may cause misunderstanding. We would like to clarify that the data in the remain set is used to evaluate the model's ability to maintain the unseen set. For the Cherry to Banana task, the remain set row in Table 2 shows the following results (the original one is tested on Zero123):

In the table, these values were obtained by inputting the images from the other 9 unlearning tasks (car, chair, sculpture, etc.) into the unlearned model (the cherry-to-banana model), and comparing the generated images with the ground truth for these input images (car, chair, sculpture, etc.). In this task, compared to directly inputting the unseen set (car, chair, sculpture, etc.) into the original model (Zero123), which resulted in SSIM = 0.787 and LPIPS = 0.279, the values show minimal change. This demonstrates our success in maintaining the unseen set effect.

---

W2: Corruption in Figure 3. In Figure 3, the Minion character appears corrupted across all settings. It would be helpful if the authors could explain the cause of this behavior and discuss potential solutions or limitations.

Thank you for pointing this out. This issue stems from the experimental setup. We have since re-adjusted the parameters for generation to ensure consistent lighting and shading on the back of the Minion character. In our experiments, the randomness of the generative model can sometimes lead to inconsistencies in details, particularly in terms of lighting and shape. We will upload the modified generation results shortly and will continue to explore ways to further reduce randomness in the generation process to ensure high-quality, stable outputs.

---

W3: Generality of the Method. The generalization ability of the proposed method remains unclear. It would strengthen the paper to include experiments on unlearning a broader set of images from the same class (e.g., more instances of Doraemon) to assess class-level forgetting performance.

Thank you for your suggestion. We have indeed conducted experiments to demonstrate the generality of our method. However, due to space constraints in the main text, we have presented only a subset of the results. Here are the additional details:

Figure 2 demonstrates the unlearning process applied to the front-facing Minion image from Figure 3, with the goal of transforming it into a pink suspenders style. Due to oversight, we did not provide additional explanation regarding this generality. In fact, we inputted five other Minion images with different body types and facial expressions, and the results showed that these Minions also successfully transformed into the desired pink suspenders style, with the suspenders, eyes, and other parts turning pink. This demonstrates the effectiveness of our method in handling different input images from the same class.

Additionally, for the ninth experiment in Figure 3 (Minion with Backpack), we conducted multiple tests by inputting Minion images with varying numbers of eyes, body types, and facial expressions. The results showed that all of these inputs successfully added a red backpack to the Minion. Due to oversight, this part of the visualization was not included in the supplementary materials. We plan to upload the results of the Minion with Backpack task and the remaining images of the pink Minion transformation with different Minion inputs in the future, or we will supplement these results in the revised version to further support the generality and effectiveness of our method.

Furthermore, we will also add additional visual results for other unlearning tasks to better demonstrate that our framework can achieve forgetting for a given class of images using just a forget image and an override image.

---