TetSphere Splatting: Representing High-Quality Geometry with Lagrangian Volumetric Meshes

W1: Simple study on tetrahedron inversion

In Appendix D of the revised version, we provide a 2D illustration showing how inversion can occur and include a simple study in 3D demonstrating tetrahedron inversion and how our regularization alleviates this issue. In this study, we fit a target shape of a sphere that is smaller than the initial TetSphere. As shown in the figure in Appendix D, without regularization, the rendering loss alone pushes the surface vertices of the initial TetSphere directly onto the target sphere's surface, resulting in a significant number of inverted tetrahedra. In contrast, our volumetric regularization prevents this issue and avoids tetrahedron inversion.

W2: Surface extraction from Tetspheres

We extract the surface triangles from each TetSphere to generate surface spheres, and then perform a union of all these surface spheres to obtain the final surface mesh. While there is no theoretical guarantee of manifoldness in the resulting mesh from union operation, our TetSphere optimization is highly regularized to maintain geometric quality. As a result, we did not encounter non-manifold issues in our experiments. We have added this description in Appendix I.

W3: Topology correctness and analysis on Euler characteristic

We appreciate your suggestions. However, we did not claim in the paper that our method can guarantee topology correctness. In fact, we acknowledge this as a potential limitation and have discussed it in Sec.6, the Conclusion section. Specifically, the union operation of multiple TetSpheres can result in topology changes that may compromise topology preservation.

Regarding the analysis of the Euler characteristic, it is important to note that many of the ground-truth shapes used in our study are noisy and contain holes that are artifacts of the data, consistent with the dataset used in [DMesh, Son et al., 2024] and evidenced by the rendered results of the ground truth shapes in Fig 5. As such, reporting the differences in Euler characteristics between the reconstructed shapes and the ground-truth shapes would not be informative and could be misleading, as it would reflect the inherent noise in the ground-truth data rather than the performance of our method.

W4: Mathematical definitions of used metrics

Thanks for the suggestion. We have included a section in Appendix C providing mathematic definitions of ALR, MR, and CC Diff used across the paper.

W5: Metrics on mesh quality

We have added a comparison on the number of triangles in Table 5 of the revised version. Although our method does not have the smallest number of triangles, it achieves the highest mesh quality among all methods. Specifically, our method outperforms NIE and 2DGS, which have approximately twice and four times the number of triangles, respectively, demonstrating superior mesh quality despite the sparser representation.

Regarding the Edge Chamfer Distance, we already included this metric in Table 5 of our original submission, where our method achieved the best result among the compared methods. Additionally, we noted that the TriangleQ metric used in CWF [Xu et al. 2024] is identical to the ALR metric used in our paper. We have added references to this in Appendix C for clarity.

W6: Intersection between Tetspheres

The TetSpheres in our optimization are indeed independent branches. Whether a unioned shape is necessary depends on the downstream application task. For rendering tasks, obtaining a unioned shape is not required. However, when a unioned shape is needed, we resolve intersections between TetSpheres by performing a mesh boolean operation, as detailed in Appendix I.

Our silhouette coverage algorithm ensures that the initial TetSpheres are overlapping with each other. While our method does not explicitly guarantee that the final mesh must be a fully connected one, the approach has proven effective in practice: In our experiments, the final mesh obtained after the union operation typically results in a connected shape. Ensuring a strong, theoretical guarantee of mesh connectivity could be an interesting direction for future work.

W7: Citation problem

Thank you for pointing out. We have corrected them.

Q1: Topological and manifold guarantees and code.

Thank you for highlighting these important points. As noted in our responses to Weaknesses, we did not claim in the original paper that our method provides strong topological and manifold guarantees. In fact, we acknowledged these as limitations and potential areas for future research.

We plan to publish the codebase upon acceptance. We hope our responses help clarify these points and assist you in increasing your rating.

Thank you for the positive feedback! For surface extraction from TetSpheres, we compute the number of occurrences of each tetrahedron's four faces. Faces that occur twice are interior faces, while those that appear only once are surface faces. Here is the pseudocode reformulated in a numpy-like style for clarity:

def extract_surface_vertices_and_faces(tetrahedron_faces):
    """
    Extract surface vertices and surface faces from a tetrahedral mesh.

    Parameters:
    tetrahedron_faces: Tx4 numpy array
        A matrix where each row represents the vertex IDs of a tetrahedron.

    Returns:
    surface_vertices: numpy array
        A sorted array of unique vertex IDs that lie on the surface.
    surface_faces: numpy array
        A matrix of surface face vertex IDs, remapped to a contiguous range.
    """
    # Step 1: Generate all triangular faces of tetrahedra
    triangular_faces = stack_rows(
        tetrahedron_faces[:, [1, 2, 3]],
        tetrahedron_faces[:, [0, 3, 2]],
        tetrahedron_faces[:, [0, 1, 3]],
        tetrahedron_faces[:, [0, 2, 1]]
    )

    # Step 2: Sort vertex IDs in each triangular face to make ordering consistent
    sorted_triangles = sort_rows(triangular_faces)

    # Step 3: Identify unique faces and count their occurrences
    unique_faces, face_indices, face_counts = unique_rows(sorted_triangles, return_indices=True, return_counts=True)

    # Step 4: Select faces that occur only once (surface faces)
    surface_face_mask = (face_counts == 1)
    surface_faces = unique_faces[surface_face_mask]

    # Step 5: Extract unique vertex IDs from surface faces
    surface_vertices = unique_elements(surface_faces)

    # Step 6: Map surface vertex IDs to a contiguous range
    vertex_id_mapping = create_mapping(surface_vertices)
    remapped_surface_faces = map_indices(surface_faces, vertex_id_mapping)

    # Step 7: Return surface vertices and remapped surface faces
    return surface_vertices, remapped_surface_faces

W1 & Q2: Trade-off between the number of tetrahedra and the number of TetSpheres

The number of TetSpheres in our method is determined by the initialization algorithm described in Sec. 4 and Appendix G, controlled by the scaling and offset parameters $\alpha$ and $\beta$ that define the initial radius of each TetSphere. Intuitively, larger values of $\alpha$ and $\beta$ result in a sparser distribution of TetSpheres. We use alpha = 1.2, and beta = 0.07, which result in an average of M = 213 TetSpheres for the shapes used in multi-view reconstruction.

In the revised version, we have added two ablation studies: one analyzing reconstruction performance with respect to different numbers of TetSpheres (Appendix B.1) and another examining the number of tetrahedra per TetSphere (Appendix B.2). When the number of tetrahedra per TetSphere is fixed and the total number of TetSpheres varies, we observe that increasing the number of TetSpheres improves both reconstruction accuracy (as measured by Chamfer distance and Vol. IoU) and surface quality (ALR). However, beyond a certain threshold, these metrics show minimal further improvement. Conversely, when the number of TetSpheres is fixed and the number of tetrahedra per TetSphere increases, reconstruction accuracy does improve, but the gains are less significant compared to increasing the number of TetSpheres. For a more detailed discussion and results, please refer to Fig. 9 and Fig. 10.

W2: Intersections between tetrahedra

Whether a unioned shape is necessary depends on the downstream application task. For rendering tasks, obtaining a unioned shape is not required. However, when a unioned shape is needed, we resolve intersections between TetSpheres by performing a mesh boolean operation, as discussed in Appendix I of the revised version. We note that this operation affects a minimal number of triangles relative to the total surface, and we observe little difference in overall mesh quality after performing the union, as intersections occur in only a limited portion of the surface.

Q1: The proposed optimisation does not appear to prevent non-uniform deformation of TetSpheres. However, the resulting reconstruction ALR is higher than in other methods, suggesting that the tetrahedra within each TetSphere largely preserve their original volume. Is there any factor specifically contributing to the high ALR?

The ALR is determined by calculating the average ratio of a triangle’s area to its perimeter, with a higher ALR indicating that the triangles are more regular and closer to being equilateral. Initially, the triangles on the TetSphere are isotropic, meaning they are already near-equilateral. Our regularization terms are designed to penalize any nonsmooth deformation gradients of the tetrahedra. Consequently, this approach inherently maintains the isotropy of the triangles to a significant extent.

Q3: More details on the image-to-3D and text-to-3D generation

We have added captions with additional information about the experimental setup to the figures in the revised version.

W1: Novelty and contribution compared to work such as [Nicolet et al. 2021] and [Palfinger 2022]

Thank you for highlighting the relevance of [Nicolet et al. 2021]. In addition to the volumetric nature of our TetSphere, a key distinction of our approach compared to [Nicolet et al. 2021] and its subsequent works is the implementation of multiple tetrahedral spheres instead of just one. In the multi-view reconstruction examples presented in our paper, the average number of TetSpheres used is approximately 210. This significantly alters the complexity of the fitting process and introduces new methodological advantages:

Both [Nicolet et al. 2021] and our approach employ regularizations to prevent extreme deformations that can compromise geometric quality, thereby inherently limiting each sphere's expressivity to a certain extent. [Nicolet et al. 2021] address this limitation by using intermediate remeshing to accommodate topological changes and manage drastic deformations when the target shape significantly differs from the sphere. However, remeshing results in the reparameterization of the texture, which poses challenges in the context of 3D shape generation pipelines. Maintaining a consistent texture parameterization throughout the optimization process is crucial, as the texture image itself is an optimization variable (see Appendix H). Additionally, remeshing can introduce undesired meshing complications when the surface undergoes topological changes. To prevent topological changes during each iteration, a single primitive must be initialized with a topology that matches the target, as demonstrated in Fig. 7 of the original paper.

By contrast, our method eliminates the need for intermediate remeshing by utilizing multiple TetSpheres alongside stronger regularization terms, including biharmonic energy and non-inversion constraints. By leveraging multiple TetSpheres, each TetSphere undergo a smaller deformations, ensuring robust shape reconstruction. This approach also adeptly reconstructs complex shapes with numerous holes, as demonstrated in the sorter and dress examples in Fig. 8.

Even with the use of multiple spheres, our method achieves comparable wall-clock time with [Nicolet et al. 2021] and [Palfinger 2022], as we parallelize the optimization process for all TetSpheres. On an A100 GPU, the average running time is approximately 4 minutes.

W2: Evaluations compared to [Nicolet et al. 2021] and "Continuous remeshing for inverse rendering [Palfinger 2022]"

In the revised version, we show both quantitative and qualitative comparisons with [Nicolet et al. 2021] and [Palfinger 2022] in Table 4 and Fig. 8 in Appendix A. We used their publicly available codebases and performed multi-view reconstruction on our dataset as described in Sec. 5.2.

Our method outperforms these approaches, particularly for shapes with complex topologies. Both [Nicolet et al. 2021] and [Palfinger 2022] use a single sphere for reconstruction, which limits their ability to handle objects with multiple holes, such as the dress and the sorter. For the dress example (the second and the third columns in Fig. 8), both baseline methods exhibit unrealistic closure of the open areas. In contrast, our approach accurately represents the thin regions and preserves the open areas due to the use of multiple TetSpheres. For the sorter example (the fourth and the fifth columns), the results from [Nicolet et al. 2021] exhibit folded, overlapping surfaces with noticeable crumpled regions. This is due to their surface-based deformation regularization, which does not prevent the interior volume from shrinking to zero. On the other hand, our tetrahedral-based method, with its volumetric regularization, mitigates these issues.

W3: The argumentation for the use of a volumetric mesh instead of a triangle mesh needs to be justified further through experiments.

Incorporating a volumetric mesh along with volumetric regularization terms is both essential and well-founded. Regularization approaches that concentrate only on surface deformation (e.g. [Nicolet et al. 2021]) can result in undesired folded and overlapping surfaces because they neglect to address or regularize the interior volume. This issue is illustrated in Fig. 8. On the other hand, volumetric regularization intrinsically reduces these artifacts by penalizing nonsmoothness of the volumetric deformation gradient and preventing volume inversion. The advantages of this approach are demonstrated in our results.

W1: Density of TetSpheres and advantanges over dense marching cubes.

Our method employs a Lagrangian representation, whereas the Marching Cubes algorithm is typically used with an Eulerian framework. Consequently, each method retains the inherent characteristics of its respective representations, which we have detailed in the introduction. One notable advantage of using a Lagrangian representation is its ability to represent a shape with fewer parameters compared to its Eulerian counterpart.

Regarding the density of TetSpheres, it is not inherently necessary for achieving high surface quality. In the revised version, we have included two additional experiments to support this claim: (a) a comparison of the average number of triangles in the reconstructed shapes, as shown in Table 5, where our method, while not producing the highest triangle count, still achieves superior mesh quality; (b) an ablation study comparing the impact of the number of TetSpheres vs. the number of tetrahedra within each TetSphere, as detailed in Appendix B.2 and Figure 10 (b). While increasing the number of tetrahedra within each TetSphere can lead to performance improvements, these gains are relatively minor compared to increasing the overall number of TetSpheres, demonstrating that effective coverage is more crucial than the density of each TetSphere.

W2: TetSphere’s advantage over NeuS/VolSDF from the quality perspective

Besides manifoldness and faster optimization speed, TetSphere also outperforms Eulerian methods such as NeuS/VolSDF in terms of the regularity of surface triangles. This is demonstrated by the single-view reconstruction results presented in Table 2, where the baseline methods SyncDreamer and Wonder3D, which use NeuS as their geometry representation, are compared. Our approach outperforms these methods in terms of the Area-Length Ratio (ALR), indicating more regular and well-formed surface triangles and, consequently, superior triangle quality.

W3: Explanation of chamfer distance

In our paper, we specifically refer to mesh quality using the three metrics of ALR, MR, and CC Diff which assess triangle regularity, manifoldness, and structural integrity and coherence. This is consistent with prior work such as FlexiCubes [Shen et al., 2023b] and DMesh [Son et al., 2024]. Chamfer distance, on the other hand, is used as a metric for reconstruction accuracy rather than surface quality, as it measures the distance between sampled points on the reconstructed mesh and the ground truth mesh, without evaluating the regularity or quality of the surface itself. While our method may not achieve the lowest chamfer distance, it still delivers competitive reconstruction accuracy, as evidenced by the Volume IoU metric. Additionally, our method achieves significantly superior performance in terms of mesh quality as reflected in the ALR, MR, and CC Diff metrics.

Q1: Gradient on invisible TetSphere interior and manifoldness

The interior TetSphere indeed only receives gradients from regularization terms. The volumetric regularization helps guide the internal structure to remain consistent and avoid non-manifold artifacts.

Q2: Missing work of Mosaic SDF

We have added a discussion on Mosaic SDF in the Related Work section. Specifically, Mosaic SDF is designed for 3D generation tasks where ground-truth shapes are provided, as it requires a surface input. In contrast, our method only requires multi-view images as input, enabling it to support reconstruction tasks without the need for pre-existing surface information.

Q3: Clarification needed on the definition of variables

In the revised version, we have added explanations for the variables used: N represents the number of vertices for each TetSphere, and T denotes the number of tetrahedra within each TetSphere. The variable n on line 309 was a typo and has been corrected to M. We appreciate your attention to detail.

Thanks for the additional results. I have increased my rating given the clarifications from the authors.

W1.1: adding such a discussion could be interesting! I have not seen any work able to achieve this yet.

W2: This sufficiently addressed my concerns.

Q1: Potential failure cases and conceptual or technical challenges

As mentioned in our response to the weaknesses, our method requires a sufficient number of TetSpheres to ensure that the output shape is of high quality. This necessitates a close match between the resolution of the initial TetSphere distribution and the level of detail required by the target geometry. Failure cases can arise in reconstruction tasks when the number of input views is extremely sparse, when the views do not sufficiently cover the target object (e.g., missing key angles or areas), or when the views lack detail due to low resolution or poor alignment. In such scenarios, the number of TetSpheres may be insufficient to faithfully cover the entire target shape, leading to incomplete or lower-quality reconstructions.

Q2: Key trade-offs over previous representations.

Our method can be viewed as balancing the trade-off between the number of primitives used for shape representation and the complexity of each individual primitive. In general, using fewer primitives requires each primitive to be more complex to adequately capture the shape, necessitating intricate regularization and processing schemes. For example, [Nicolet 2021] employs a single surface sphere with additional remeshing steps to manage large deformations and maintain mesh quality. On the other hand, using simpler primitives, as seen in methods like DMesh (which relies on surface triangles) and Gaussian Splatting (GS, which uses point clouds), requires a large number of primitives to capture intricate shapes. While this approach reduces the need for complex regularization within each primitive, it sacrifices overall mesh quality due to limited interactions between the primitives.

Our method, leveraging structured TetSpheres, strikes a balance between these extremes. By using multiple simple yet structured volumetric primitives, we enable strong regularization within each TetSphere without necessitating complex deformation schemes or intermediate remeshing. This allows for more robust and high-quality reconstructions, maintaining a balance between computational efficiency and reconstruction fidelity.

W1.1: The use of bi-harmonic energy on deformation gradients

The incorporation of biharmonic energy in our method is inspired by its proven efficacy in geometry processing, as highlighted by prior research [1]. Furthermore, penalizing the nonsmoothness of transformations (or deformation gradient) across a domain is a well-explored area that has demonstrated its utility in various geometry processing tasks such as shape deformation [3], deformation transfer [2], and mesh alignment [4].

For the surface regions where observations are available, the surface deformation is primarily influenced by the rendering loss. Conversely, in regions lacking direct observations, our regularization term plays an important role by penalizing high-frequency deformations resulting from nonsmooth deformation gradients. This approach ensures reliable results.

The biharmonic energy offers stronger regularization compared to first-order harmonic energy, as demonstrated in [1]. Although theoretically higher-order smoothness energies can be considered, in practice, they often lead to numerical instability, as discussed in [5]. Therefore, biharmonic energy strikes an balance by providing effective regularization while avoiding numerical issues.

[1] On Linear Variational Surface Deformation Methods, Botsch et al., 2008

[2] Deformation Transfer for Triangle Meshes, Sumner et al., 2004

[3] Modeling of Personalized Anatomy using Plastic Strains, Wang et al., 2021

[4] Optimal Step Nonrigid ICP Algorithms for Surface Registration, Amberg et al., 2007

[5] Libigl, https://libigl.github.io/

W1.2: How important is the non-inversion term, and how brittle are parameter choices?

The non-inversion term plays a critical role by preventing surface flipping and overlapping. In Section 5.4 and Fig. 6, we analyze the effects of energy coefficients and illustrate how changes to these coefficients affect reconstruction results using an Armadillo shape. We found that larger coefficients yield smoother surfaces, whereas using excessively small coefficients or omitting the non-inversion regularization can result in tetrahedron inversion, ultimately causing artifacts on the surface.

W1.3: Experimental comparison with [Nicholet et al. 2021]

We have added a comparison with [Nicholet et al. 2021] in Appendix A in the revised version. Table 4 and Fig. 8 show that our method outperforms [Nicholet et al. 2021], particularly for shapes with complex topologies. Unlike these single-sphere, surface-regularized approaches, which struggle with objects featuring multiple holes (e.g., the dress and sorter), our method, using multiple TetSpheres and volumetric regularization, preserves thin regions and open areas without unrealistic closures. Additionally, our volumetric approach effectively prevents artifacts like folded, overlapping surfaces and crumpled regions seen in [Nicholet et al. 2021], highlighting the benefits of a volumetric method for achieving high-quality reconstructions.

For a more throughtout discussion about [Nicholet et al. 2021], please refer to [AA94] W1.

W2: Limitations could be discussed more clearly.

We have added a section in Appendix K regarding the overlaid piecewise meshes rather than fitting a single tetrahedra mesh. We have also added a paragraph in Appendix I describing how to obtain a unioned shape from TetSpheres. We note that this operation affects a minimal number of triangles relative to the total surface, and we observe little difference in overall mesh quality after performing the union, as intersections occur in only a limited portion of the surface.

W3: More studies into cases where results are unsatisfactory.

In the revised version, we have added multiple ablation studies (Fig. 8, 9, and 10). Regarding the comment on fitting a single sphere globally, the ablation study on the number of TetSpheres used for reconstruction, detailed in Appendix B.1, explores how the number of TetSpheres affects reconstruction performance, with qualitative results shown in Fig. 9.

When an appropriate number of TetSpheres is used, our method achieves the best results. Cases where the number of TetSpheres is too small relative to the complexity of the shape – such as attempting to use only a single global sphere for highly detailed models like those shown in Fig. 9 – may result in unsatisfactory reconstructions.

We hope these additions and clarifications address your concerns and more clearly highlight the strengths and limitations of our approach.