Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modeling

We would like to thank for your comments. Below, we address each of your concerns in detail.

(W1&Q1) The motivation of using sparse voxels: The motivation for using sparse voxels is both well-founded and straightforward. Dense grids face significant limitations in terms of memory and computational efficiency. Direct optimization in dense grids becomes infeasible due to out-of-memory issues when the resolution exceeds $256^3$, making it practically impossible to operate at resolutions like $1024^3$.

Our sparse voxel representation does not compromise fine details. In the ideal case, a dense grid activates voxels along the surface of the raw mesh—precisely what our encoder achieves with sparse cubes. Thanks to the high efficiency of our encoder, we can afford much higher resolutions than dense grids, resulting in superior reconstruction and generation performance.

(W2) The reliability of the flood-fill algorithm: Apart from our flood fill approach, the previously most common method for determining inside/outside regions in non-watertight remeshing is computing the winding number. However, winding number requires the input mesh to have correctly oriented normals—an assumption that is often violated, especially in large-scale mesh datasets from the internet where normal directions are frequently incorrect or inconsistent. In contrast, our flood fill method only requires the initial seed point to be outside the surface (e.g., placed in a corner), making it a more stable and controllable solution.

(W3&Q4) The metric that fall behind: ANC on Objeverse is one of the few metrics where our method does not consistently outperform others. However, ANC evaluates absolute normal consistency against the ground-truth normals, which are often noisy or incorrect. Our method, on the other hand, corrects the normal orientations via flood fill. Therefore, this metric may not accurately reflect the actual surface quality. Moreover, as shown in the visual results from Objaverse in Fig. 5, our method preserves significantly more geometric details than CraftMan—beyond what is indicated by the quantitative metrics.

(W4) Fine details and small or hidden components: As highlighted in our paper, e.g., the teaser figure demonstrates rich details and hidden components, we argue that our method more effectively preserves fine details, including those in occluded regions. Furthermore, as shown in Fig. 5, our approach retains small and hidden structures without loss compared with the existing SOTA method. These components are already well captured during our multi-step process (Steps 1–3). Compared to prior methods using 2D supervision like Trellis, our approach demonstrates significantly better preservation of hidden parts, as the 2D rendering loss used provides very weak supervision for unobserved areas. Unlike many existing open-source and even commercial text-to-3D models, our generated meshes are strictly watertight, making them suitable for practical applications such as 3D printing.

(W4&Q5) Visulization of XCubes: We already provided the visualization results of XCubes in the last row of Fig. 5 of the main paper.

(Q2) Flood-fill for the object w/o watertight closure: We address this challenge using a coarse-to-fine multi-scale flood-fill strategy. Even when the target object is not fully watertight during early training stages, the coarsest scale provides a sufficiently smoothed and regularized representation that enables robust separation between interior and exterior regions. This coarse-level initialization ensures reliable flood-fill propagation, which is then progressively refined at finer scales.

(Q3) Refining a globally accurate SDF: Our globally accurate SDF is in fact obtained through local optimization over high-resolution sparse cubes, rather than by fitting a global function directly. Therefore, it does not suffer from the loss of fine surface details.

Thanks to the high resolution of our sparse representation, local SDF refinement is both efficient and precise. Moreover, the sparse cubes are tightly aligned with the raw mesh surface, effectively covering the local geometry. Optimizing these cubes is essentially equivalent to optimizing accurate local SDFs, which preserves detailed structures.

(Q4) Decoupled SDF prediction: As with other large-scale generative models, conducting a comprehensive ablation study on each individual loss term is infeasible. Nevertheless, we argue that decoupling the sign and magnitude of the SDF is well justified for two reasons:

• Empirical success in 3D vision. Prior work on implicit surface learning (e.g., Points2Surf [1]) has demonstrated that separating the prediction of sign (inside vs. outside) from the prediction of distance yields significantly better results than treating the SDF as a single end-to-end regression target.
• Faster convergence for classification vs. regression. Statistical learning theory shows that, under low-noise (margin) conditions, classification can achieve strictly faster minimax convergence rates than regression. In particular, Audibert and Tsybakov [2] construct plug-in classifiers that attain “fast” and even “super-fast” rates—up to $O(n^{-1}$)—for the excess Bayes risk when the conditional probability function satisfies the Tsybakov margin assumption.

We hope that our responses adequately address your concerns and demonstrate the soundness of our approach.

References:

[1] Erler, P., Guerrero, P., Ohrhallinger, S., Mitra, N. J., & Wimmer, M. (2020). Points2Surf: Learning implicit surfaces from point clouds. In ECCV (pp. 108–124). Springer.

[2] Audibert, J.-Y., & Tsybakov, A. B. (2007). Fast learning rates for plug-in classifiers.

Thank you to the authors for their efforts in addressing my concerns. The responses have clarified several issues, including the reliability of the flood-fill algorithm, the preservation of fine details, and the visualization results. However, I am not sure if I understand the explanation of how sparse sampling can reliably capture critical information within the target. In addition, not sure about the authors' claim that existing evaluation metrics fail to accurately reflect the effectiveness of their method.

We thank Reviewer 6geV for the thoughtful and constructive feedback, and for recognizing the value of our work.

Below, we provide our detailed response:

(W1) Reproducibility of custom CUDA kernels: As researchers, we wholeheartedly agree that open-source is essential for the community. We are currently seeking internal approval and, once granted, will release the code publicly. Moreover, in the camera-ready version, we will include a detailed description of our CUDA kernels to ensure full reproducibility.

(W2) Optimization cost in remesh steps 3–4: Our reported timing already includes the optimization overhead for steps 3–4. Thanks to the cubes’ high initial resolution—where they closely align with the target surface—each cube requires only a few updates:

• Step 3: converges in fewer than 10 gradient updates.
• Step 4: converges in approximately 100 gradient updates.

This efficiency enables the full per-object optimization to complete in roughly 30 seconds.

(W3) Comparison to SparseFlex: We appreciate the reviewer’s suggestion. SparseFlex is an concurrent arXiv preprint (uploaded on March 27, 2025), just prior to our NeurIPS submission deadline, so we originally placed the quantitative comparison in the supplementary material. In the camera-ready version, we will move this comparison into the main manuscript for greater visibility.

Below, we report Chamfer Distance (CD) and runtime under challenging evaluation settings:

As shown, our method achieves a comparable Chamfer Distance while using ~6× fewer tokens and delivering ~6× faster encoding–decoding, consistent with the qualitative results presented in the supplementary material.

(Q1) Details about remeshing optimzation: We jointly optimize per-vertex SDF offsets and positional deltas, enforce sign consistency on the SDF (so that positive/negative labels never flip), and constrain each vertex’s movement to within half of the original cube edge—provably preventing any inverted faces. We will make these details explicit in the camera-ready version.

(Q2) Typo in Eq. 5: We apologize for the typo in Eq. 5: the expression $\delta(x) \sim \delta(x)$ was included by mistake and will be removed. We did not rely on this assumption.

(Q3) Typo in Reference [28] (TripoSR): We thank the reviewer for catching this error. In the camera-ready version, we will correct the citation to:

''TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models.''

(Q4) Formatting in Eq. 3: We appreciate the reviewer’s careful attention to detail. We will boldface 'x' in Eq. 3 in the camera-ready manuscript.

I appreciate the response from the authors, which addresses my concerns largely. Please carefully include these quantitative comparisons in the camera-ready version. After also reading other reviews and rebuttals, I think this is a good paper to present in NeurIPs.

We appreciate the time and effort dedicated to evaluating our work, and we have addressed each point raised with detailed clarifications

(W1) Additional Baselines: We have compared our method against the state-of-the-art remeshing techniques available at the time of submission (e.g., Dora). In this rebuttal, we also include results for Mesh2SDF [1] on the most challenging cases. We would be happy to add further comparisons if the reviewers can suggest any other readily available, advanced remeshing algorithms.

(W2) Ablation of each in the preprocessing:

We perform an ablation study on Steps 3 and 4 to isolate the contributions of deformation optimization and rendering-based refinement:

In addition to consistent quantitative gains across all metrics, enabling both steps leads to substantial visual improvements—often beyond what the metrics alone can capture in terms of perceptual quality. Due to limitations of rebuttal this year, we cannot include these visuals here; however, we will present comprehensive side-by-side comparisons in the camera-ready version.

(W3) Comparsion with the Trellis' VAE: We would like to clarify that TRELLIS's VAE is based on sparse transformers rather than sparse convolutions. Its architecture is fully composed of Swin attention blocks, which are not scalable or trainable at high resolutions (e.g., 1024³), due to the prohibitive memory and computational cost. This makes direct comparison with our approach technically infeasible.

Our VAE, in contrast, offers significant advantages: It achieves extremely high compression ratios while maintaining superior reconstruction quality and effective learning for downstream generation tasks. Specifically, it supports decoding from a latent resolution of $64^3$ to $1024^3$, significantly surpassing the $64^3→256^3$ decoding capability in TRELLIS.

We did not include an explict concept comparison with the TRELLIS VAE because the two are fundamentally different in terms of representation space, training objectives, and architecture. For example:

• Our VAE learns native geometric representations, whereas TRELLIS relies on lossy DINO features.
• TRELLIS lacks sparse downsampling and upsampling modules, making it infeasible to handle high-resolution sparse inputs. Dense occupancy grids at this scale become too large to train/inference and result in out-of-memory (OOM) issues.
• We use supervision in the 3D space while Trellis use 2D supervsion from the image space.

Lastly, we would like to emphasize that our remeshing pipeline is itself a key contribution of this work, and forms an integral part of the overall design.

(W4) Detailed architecture: The detailed design of our VAE is provided in the supplementary material. As noted in the paper, the architecture of the generative component is identical to that of TRELLIS, and we adopt the same two-stage design.

(MW1) Where to apply hole-filling: We apply hole-filling to the outputs of the second stage—that is, the final generated results. While this does not affect the VAE evaluation metrics, it can influence the visual quality of the generated meshes.

(MW2) Resolution bottleneck and failure cases: The resolution of SparCubes is currently limited by the nvdiffrast rendering backend. In particular, nvdiffrast imposes a fixed upper limit on the number of mesh fragments due to its internal mesh ID encoding. Therefore, we could not directly applied it to the resolution of $2048^3$. However, this is a constraint of the underlying renderer rather than our framework design.

As for failure cases, SparCubes may struggle with meshes that contain extremely occluded internal parts. In such cases, Step 4 may not be effective due to the lack of visibility during optimization. While Steps 1–3 can still provide a coarse reconstruction of these hidden areas, the fidelity may not match that of the visible surfaces.

However, compared to Trellis, SparCubes demonstrates significantly better internal structure reconstruction. This is because our Steps 1–3 are unaffected by occlusion and do not rely on 2D rendering loss for supervision. For example, the internal details of the robot in the teaser figure are well preserved in our results, whereas Trellis—being solely guided by 2D supervision—tends to lose such internal structures.

(MW3) Ablation of each training loss We validated the effectiveness of each loss term under a small-scale dataset prior to the scaled-up training. As with many large-model frameworks, conducting a full ablation study may be impractical due to computational constraints.

Technical limitations

• We observed that, apart from the commonly known challenge of handling highly occluded internal structures—which remains difficult for most methods—our approach does not exhibit any significant additional limitations in terms of the remeshing quality compared to existing methods.
• Similar to existing methods, our use of Marching Cubes can result in meshes with a high number of faces. In certain application scenarios, this may require additional post-processing, which is also a common open problem in the community. Nonetheless, we believe our compact latent representation could benefit downstream retopology tasks, making this a promising direction for future exploration.

Reference:

[1] Peng-Shuai Wang, Yang Liu, and Xin Tong. 2022. Dual Octree Graph Networks for Learning Adaptive Volumetric Shape Representations. ACM Trans. Graph. (SIGGRAPH) 41, 4, Article 103 (July 2022), 14 pages.

Dear Reviewer,

We hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, we wanted to follow up to ensure that we have addressed all your concerns satisfactorily. If there are any additional points or feedback you would like us to consider, please feel free to let us know.

Your insights are invaluable to us, and we remain eager to clarify or improve any aspect of our work based on your comments.

Thank you again for your time and effort in reviewing our paper.

Authors

We sincerely appreciate your valuable feedback and recognization of the contributions to our work. Due to the limitations of NeurIPS this year, we are unfortunately unable to upload images.

(W1) Stability of step 3: We deeply appreciate the reviewer’s concern about the stability of our per-object optimization in Step 3. In this phase, we simultaneously refine each vertex’s signed distance function (SDF) value and its positional offset—an essential step for achieving precise watertight reconstruction. To guard against any mesh inversions or “face flips,” we enforce strict sign consistency on the SDF values, ensuring that vertices originally classified as inside or outside never change their label. Additionally, we cap each vertex’s movement to at most half of the original cube edge length, a mathematically proven safe bound that prevents any inverted topology. Thanks to our high-resolution initialization, where sparse cubes already densely cover the raw mesh surface, this carefully constrained optimization converges extremely quickly—typically < 10 steps.

To confirm the robustness of this design, we ran Step 3 one hundred times with different random seeds. We observed that the Chamfer Distance (×10⁴) varied by less than 0.01, demonstrating that our method is highly insensitive to stochastic initialization and reliably reproducible. We hope this extended analysis reassures the reviewer of the stability of our approach. Per your suggestion, we will include detailed convergence discussion of the optimization in the camera-ready manuscript.

(W2&Q3) Using multiple view images in Step 4 is necessary, as single-view rendered images cannot optimize the occluded regions. As a result, details in those unseen areas will be poor if only using single-view image.

(Q1) Additional datasets: We apologize for the unclear description of our “in-the-wild” dataset. This collection combines the most challenging examples from multiple sources—including difficult cases from GSO and scanned or artist-created samples gathered online. In the camera-ready version, we will present a detailed breakdown of the dataset’s composition and selection criteria.

(Q2) Seed among different benchmarks: We used the same random seed across all benchmarks to eliminate potential bias. Moreover, we repeated each experiment 100 times and observed that the Chamfer Distance ($\times 10^4$) varies by less than 0.01 across runs, demonstrating that our results are robust to seed selection.

Thank you again for your insightful feedback; we hope our clarifications and planned enhancements address your concerns and further strengthen our submission.

Dear Reviewer FtUJ, Did the author's response resolve your questions? If necessary, please have further discussion, and please give the final justification and rating after final confirmation. Best regards, Area Chair

I would like to thank the authors for their rebuttal. I have also looked at the discussions and reviews from other reviewers which has also helped me get a better understanding of the work. I thus continue to maintain my scores.