UniRestore3D: A Scalable Framework For General Shape Restoration

We appreciate Reviewer 64j6 for recognizing the contributions of our paper and offering valuable comments. Our responses to the feedback are provided below.

---

W1: Key challenge for building a general shape restoration dataset

"What is the key challenge for building the large scale dataset for shape restoration? The process mainly consists of randomly adding different type of pre-defined perturbation on existing large-scale 3D object datasets. It is surprising (or maybe I missed some work?) that no one ever did this before."

WA1: There are indeed existing datasets aimed at addressing specific subtasks, such as noise-free shape completion (AutoSDF, AdaPointTr) and noisy shape completion (3D-EPN, PatchComplete). However, these datasets face a key challenge: their heterogeneity (e.g., using different representations, varying resolutions, different methods for computing the intact representations, as summarized in the table below) prevents them from being combined for joint training and evaluation, unlike image datasets. To address this challenge, we devoted significant effort to constructing a comprehensive dataset specifically for this purpose. This involved unifying representations across different subtasks, designing tailored construction methods to ensure the synthetic data closely resembles real-world scenarios, combining basic subtasks to enhance data diversity and task complexity, and implementing these processes at scale. This extensive effort enabled us to explore the application of shape generative models in general shape restoration more effectively.

[1] Amanatides, J., & Woo, A. (1987). A fast voxel traversal algorithm for ray tracing. In Eurographics (Vol. 87, No. 3, pp. 3-10).

[2] Xu, H., & Barbič, J. (2020). Signed distance fields for polygon soup meshes. In Graphics Interface 2014 (pp. 35-41). AK Peters/CRC Press.

---

W2: Key challenges for building the framework and insights for why it has good performance.

As mentioned in the introduction section, "These diverse tasks require different model capabilities, making it challenging to design a unified model for general shape restoration" ......

We divide this part into three smaller questions to answer individually, hoping this will address your concerns.

W2.1: Required model capabilities for general shape restoration and how they are addressed in the proposed framework.

As mentioned in the introduction section, "These diverse tasks require different model capabilities, making it challenging to design a unified model for general shape restoration" - how does the proposed framework addressed this capabilities requirement?

WA2.1: We think that a model capable of general shape restoration should possess the following capabilities. Below, we outline these capabilities along with our framework's corresponding solutions:

1. Semantic understanding of severely damaged and noisy defective shapes (achieving reasonable completion based on the understanding of the defective shapes even without relying on additional text or image inputs);
   • We train within a conditional generative model framework using a large and diverse set of paired data.
2. Robustness to shapes with varying degrees of degradation;
   • We employ patch-wise encoding in both intact and defective shape encoders.
3. Robustness to irregular, high-intensity noise in defective shapes;
   • We use a unified representation for intact and defective shapes, explicitly aligning the encoding of the defective shape encoder with that of the intact one. This enhances the quality of the shape representation while pre-learning denoising capabilities, improving robustness to noise.
4. Producing shape restoration results that are diverse and realistic while maintaining high fidelity w.r.t. the input shape;
   • We propose a scalable training strategy that facilitates accepting high-resolution defective shapes as inputs.
5. The ability to generate high-quality restoration results;
   • We propose a hierarchical shape generative model as the foundational architecture.

We appreciate Reviewer XP48 for recognizing the contributions of our paper and offering valuable comments. We are especially grateful to Reviewer XP48 for the careful review of the evaluation details across different benchmarks, highlighting the challenges caused by the lack of standardization in existing benchmarks. We also appreciate the recognition of the importance of our proposed benchmark and the effort we put into evaluating our method on relevant benchmarks. Our responses to the feedback are provided below.

W1: Clearly state the shape representation and its resolution early in the writing.

The shape representation (TSDF) is mentioned for the first time on page 4. Your readers might have very different backgrounds and very different understanding of 3D-Shapes. Some might think you work on 3D-Pointclouds, others think you work with meshes, yet others might think you for sure use binary occupancy grids or unsigned distance fields neural representations or whatever. In the related works you put yourself next to all kind of methods without pointing out this major difference. In my humble opinion the choice of shape representation is a central feature, which defines it's usefulness for different problems. It should be communicated very clearly - if not in the Title at least in the Abstract or at least in the Introduction. Also your resolution of 256 is really nice and also could be mentioned earlier.

WA1: Thank you for your suggestion! We have emphasized the shape representation in the revised abstract. We also emphasize the benefit of accepting high resolution TSDF grids as inputs in the revised related work.

---

W2: More details on training & inference

No information about training time of the modules is given. No information of inference time and memory requirement for inference are given. (and in Questions: Please provide training hardware, memory and time requirements. Same for inference.)

WA2: Thank you for pointing out this omission. We have provided relevant information in the A.3 section of the revised appendix.

---

W3: Feasibility on real-time applications

Diffusion based method is not usable for online real-time applications (during 3D-scanning).

WA3: Currently, due to speed limitations, our method is indeed not suitable for real-time applications. However, our pipeline is compatible with diffusion acceleration techniques (e.g., better samplers, diffusion model distillation) and other modeling optimizations (e.g., flow matching). Combining these approaches in the future could enable online shape restoration of 3D reconstruction or provide shape priors for real-time 3D reconstruction.

---

Q1: Concern about claim of being able to condition on "text or images"

Paper claims that the denoiser can be conditioned on "text or images", which was never demonstrated. Claim must be removed or proven in the given context. If it is not used it should be removed from the method (especially equation 1). If it is used its effect must be evaluated. This is my biggest concern currently.

QA1: We indeed used text condition on our evaluation on the 3DQD benchmark. Since the 3DQD benchmark evaluates category-conditional models, not using category condition could lead to ambiguities in the restoration results for categories with similar structures (e.g., tables, chairs, sofas). By incorporating category condition through text condition, our model avoids such ambiguities.

We did not use image condition in any experiments, as we believe it becomes difficult to determine whether the model's restoration relies primarily on the image information or its understanding of the defective shape. Accordingly, we have removed the phrase 'can be conditioned on images' from the paper and retained only 'text.'

We appreciate Reviewer mwfZ for recognizing the contributions of our paper and offering valuable comments. Our responses to the feedback are provided below.

---

W1: lack of details of network architectures

While a high level description of the H-VAE encoder, H-VAE decoder, and H-LDM are provided, there is no description of the actual architectures for these modules ......

WA1: We have added a section of the model architecture in the updated appendix, we hope this resolves your concerns.

---

W2: training time details

A scalable training strategy is posed as a contribution; however, there does not exist any evidence that the proposed strategy is more efficient ......

WA2: Thank you for your suggestion. We can more intuitively demonstrate the importance of the proposed scalable training strategy by comparing the training speeds under two different settings. As shown in the table below, taking single-stage H-LDM as an example, if we do not pre-train an encoder to compress the defective shapes to be restored during training, the training speed slows down by nearly 20 times. This results in unacceptable training times and makes it difficult to train with larger-scale data.

---

W3: Generalization or robustness to unknown categories

In Table 1, the model obtains a 2-5x worse MMD/AMD on ABO across the different tasks and about a 100x worse MMD/AMD on GSO. This large drop in performance is observed even for the noisy refinement task which should be an easier task than noisy completion."

WA3: Achieving robust restoration on unknown categories with severe incompleteness is highly challenging. In our benchmark, we do not utilize text or image inputs; the model must implicitly recognize the semantics of defective shapes and perform restoration based on this understanding. This is inherently infeasible for unknown categories with large missing regions and not the focus of our work. We expect improvement on unknown categories by enlarging the dataset to cover more categories or by fine-tuning text-to-3D models pre-trained on larger-scale datasets. In contrast, one of our goals is to validate the effectiveness of unifying data sources of multiple subtasks compared to single-task data. This has been demonstrated for both known categories and in-the-wild categories in our ablation studies.

However, after careful examination, we discovered that during evaluation on the GSO data, there was an issue with the metrics due to differences in object pose orientation between our predictions and the ground truth. We have corrected the GSO metrics in Table 1. Thank you very much for catching this and we sincerely apologize for not noticing this earlier. With the corrected metrics, the GSO scores are better, and the performance drop in noisy refinement and shape super-resolution is indeed significantly less than in the more challenging noise-free and noisy shape completion subtasks.

Furthermore, the shape refinement task is not always easier. During data construction, we randomly combined different basic shape restoration types. The noisy refinement subtask is randomly paired with different resolutions, and in cases of low resolution (e.g., $32^3$), severe impairment may occur at lower resolutions, making it highly challenging to restore. As shown in the table below, on the shape refinement subtask of the GSO data, our method performs worse on low-resolution defective shapes than on high-resolution ones.

Thank you for providing such in depth responses. I have some additional comments below pertaining to the answers that have been provided.

---

W2: training time details
Please include this table either in the ablations section of your main paper or in the appendix. This table is the only result that provides clear evidence for your claimed contribution of "scalable training", so this is an important result to include in the paper.

---

W3: Generalization or robustness to unknown categories
Thank you for providing the updated numbers on the GSO dataset. The new results look significantly better and address my concerns in regard to generalization. Additionally, it was an oversight on my part that the noisy refinement task also contained varying resolution inputs. I originally assumed they were all at the 256^3 resolution, but the varying input resolution does help explain the potential difficulty in the refinement task. I suggest including the performance break down table of resolution for the noisy refinement task in the appendix, as I believe this is an interesting experiment on how performance varies with varying levels and types of shape degradation.

---

W4: Fairness of experiments on 3DQD and PatchComplete
Thank you for clarification on the claims for why your model is better over existing architectures. I acknowledge that designing a model architecture which enables support for training on higher resolution data and significantly larger datasets are certainly improvements over existing architectures which cannot support this.

---

Q1: Details about feature alignment
Please add this detail to the paper. It is not clear that this is how the feature alignment loss is being computed from the loss function alone.

---

Overall, I have decided to raise my score as I believe the contributions presented (e.g., new dataset, model which can generate high resolution completions, and support for training on large scale datasets) are clear benefits over the existing work on shape restoration. Please do still update the final version of the paper to include the suggestions I have mentioned above though.

We appreciate Reviewer Bu6F for recognizing the contributions of our paper and offering valuable comments. Our responses to the feedback are provided below.

---

W1: Experimental results across different datasets

W1.1: "There are no visual comparisons for other methods, making it hard to compare the improvements."

W1.2: "All data is from the proposed dataset, how about results on real noisy 3D scans?"

W1.3: "How about the results on other datasets?";

W1.4: "Does the network overfits on training set?"

WA1: We conducted experiments on three distinct benchmarks, each involving datasets with sources different from our training dataset and covering various experimental settings. We regret any misunderstanding Reviewer Bu6F might have had regarding the experiments in the paper, and we provide further clarification below. As illustrated in Figure 2, we categorize shape restoration into four basic types: Noise-free Completion, Super-resolution, Noisy Refinement, and Noisy Completion.

• Sec 5.1, Table 1, Figure 5: These present the results tested on our newly proposed dataset/benchmark. This benchmark is designed to evaluate performance in the new general shape restoration setting, which includes all four basic types mentioned above. It requires a single model to handle these shape restoration scenarios simultaneously. Moreover, the test data (from GSO and ABO) is distinct from the training data (from Objaverse and ShapeNet).
• Sec 5.2, Table 2, Figure 6: These show the performance of our method on noise-free completion, a single shape restoration setting tested on an existing benchmark independent of our proposed dataset.
• Sec 5.3, Table 3, Figure 7: These demonstrate the performance of our method on noisy shape completion, another single shape restoration setting tested on an existing benchmark, also independent of our proposed dataset.

More specific clarifications addressing the four sub-questions:

• WA1.1: In Figure 6 and Figure 7, we provide qualitative comparisons between our method and other baselines, clearly illustrating the strengths of our method.
• WA1.2 & WA1.3: Table 2 and Table 3 present the quantitative results on different benchmarks (i.e., datasets). Notably, Table 3 includes results on real noisy 3D scans from the ScanNet dataset, showcasing the effectiveness of our method in real-world noisy scenarios.
• WA1.4: The quantitative and qualitative results presented in Table 1–3 and Figure 5–7 are all based on validation or test sets. We believe this demonstrates that our model does not exhibit significant overfitting, as the performance holds consistently on unseen data.

---

W2: Lack of an overview figure

There is no overview figure of the whole method, from Figure 3-4, it is still hard to follow the method design.

WA2: In Figure 3, we illustrate the model’s inference pipeline, which primarily consists of the noise-robust encoder for defective shapes and the conditional generative model for generating intact shapes. The training process for our model comprises three key stages: (1) Training the VAE on intact shapes; (2) Training the noise-robust encoder for defective shapes; (3) Training the conditional generative model H-LDM. Figure 4 provides a detailed depiction of the first two stages of training. The training process for the conditional generative model (H-LDM), while crucial, follows standard methodologies and was therefore omitted from the figure for brevity.

We are open to feedback on specific aspects of Figures 3 and 4 that might fall short in providing a comprehensive overview of the method design. This will help us refine and improve the presentation for greater clarity and completeness.

---

W3: About problem setup and model capabilities.

For different noises, like low-resolution, noisy completion, noisy refinement etc., do they share the same pipeline/network?

WA3: Yes, the primary motivation of our work is to enable a single pipeline/network to handle the restoration of shape defects caused by various scenarios, achieved through the combination of dataset and model design tailored for this purpose.

I respectfully disagree with the assessment of Reviewer Bu6F. While I understand the importance of rigorous evaluation, I believe the authors have adequately addressed the concerns raised in the original review and subsequent discussions.

Specifically, the authors have provided clear explanations and clarifications to the questions raised by Bu6F. The new benchmark, which is a significant contribution of this paper, is not directly compatible with older works. This becomes clear when considering the efforts described by the authors (appendix) to run their method on the old benchmarks. Including results from prior works in Table 1 would not contribute new insights over what was evaluated on the old benchmarks already.

I encourage Reviewer Bu6F to reconsider their stance, given the authors' efforts to address the raised issues, the novelty of the proposed benchmark and the method which out-scales and outperforms prior work. A more constructive approach would involve providing specific suggestions for improvement, rather than maintaining a negative stance despite the author's efforts.

Ultimately, I believe the paper's contributions warrant a fair and more favorable evaluation.