DMesh: A Differentiable Mesh Representation

We appreciate your detailed comments and positive evaluation about our work. Please let us know if we addressed your questions correctly.

---

Q1. Degraded efficiency of the method due to WDT

A1. As pointed out, the main computational bottleneck in the current implementation is the WDT construction. Please see Global Response A1 for detailed information. Removing the necessity for WDT is one of our future research directions.

---

Q2. Showing the deformation process of a mesh with different topologies

A2. Thank you for the suggestion. Please see Global Response A3. We will include this in the revised version of the paper.

---

Q3. Ratio of non-manifold errors

A3. Please see Global Response A2 for details on the ratio of non-manifold errors.

---

Q4. Robustness against noises and holes in point cloud reconstruction

A4. As pointed out, we did not test our method on point clouds with noises or holes. Currently, our method may not be suitable for such challenging cases because our reconstruction is driven solely by Chamfer Distance loss. Consequently, holes or noises in the input point clouds may persist in the reconstructed mesh. Other point cloud reconstruction methods often impose assumptions about the mesh topology to address these issues. However, our approach does not impose any topological constraints, which is contradictory to these methods.

Additionally, developing a robust point cloud reconstruction method is beyond the scope of this paper, as the main contribution lies in introducing the differentiable mesh formulation, not the reconstruction itself. Nevertheless, we acknowledge the importance of this aspect and consider it an interesting direction for future research.

---

Q5. Limitations in processing indoor/outdoor scenes

A5. Yes, we will include a discussion of the limitations related to processing indoor and outdoor scenes in the paper, as these data are more challenging due to noises, holes, and complicated topology. Thank you for the suggestion.

We appreciate your detailed comments and positive evaluation about our work. Please let us know if we addressed your questions correctly.

---

Q1. Significant computational cost

A1. Yes, in the current implementation, we can handle up to 20K points efficiently, but going beyond that is limited. Please see the detailed analysis about the computational cost in Global Response A1. To overcome this limitation, we need to take a completely different approach that does not require the WDT algorithm, as it is the main computational bottleneck. This is one of our future research directions.

---

Q2. Testing on real-world and texture-rich datasets

A2. As pointed out, our current implementation does not support texture reconstruction, which is why our method is not yet applicable to real-world datasets. This is also a future research direction we intend to explore.

---

Q3. Ablation on the influence of $\Lambda_{wdt}$ and $\Lambda_{real}$

A3. We believe there might be a misunderstanding or typo in this question, as $\Lambda_{wdt}$ and $\Lambda_{real}$ are multiplied to get the final face existence probability (Eq. 2). They are not hyperparameters that we can arbitrarily choose. Therefore, there is no specific ablation study we can conduct on them. If you meant $\lambda_{weight}$, $\lambda_{real}$, and $\lambda_{qual}$, which are the coefficients for regularization (Section 4.3.2), please see Figure 7 and Appendix E.3 for the ablation study of these coefficients. Please let us know if we misunderstood your question.

We appreciate your detailed comments and positive evaluation about our work. Please let us know if we addressed your questions correctly.

---

Q1. Non-manifoldness

A1. Please see Global Response A2.

---

Q2. Computational cost

A2. Thank you for the suggestion. Please see Global Response A1 for detailed statistics about the computational cost. We will include this information in the final version of the main paper.

---

Q3. Technical novelty in the estimation step of 4.1

A3. There were two main motivations for developing the lower bound estimation step in 4.1 instead of using the prior framework: precision and efficiency, as suggested at the end of 3.2. As shown in the paper, our estimations were much more precise than the previous method. Regarding efficiency, our approach significantly reduces the computational burden, even when the dual simplex is a single point (Table 4, d=2, k=2 case). We argue that our approach effectively relaxes the computational challenges of the previous method, despite being based on simple principles. Additionally, its implementation was not as straightforward as the prior framework, which is why we used CUDA instead of PyTorch. For instance, while the prior framework only requires a point projection operation, we needed other operations, such as line-line distance computation. Therefore, we believe the technical novelty is not trivial.

---

Q4. Including an overview of the technical pipeline in the main body

A4. Thank you for the suggestion. We will modify the paper to incorporate more details about the experiments in the main body if we are granted an additional page in the camera-ready version.

---

Q5. Technical mistake about description above equation (3)

A5. Thank you for pointing this out. You are correct that the description was wrong. Specifically, Eq. 3 is correct, but in line 154, we should have written, "sign is positive when inside." We will correct this in the revised version of the paper.

---

Q6. Sharpness of the lower bound in Eq. (5)

A6. Thank you for the insightful question. Let's consider an example under the same setting as Figure 6. In Figure 6(d), we rendered the case when $\Delta^{1} = \{ p_1, p_7 \}$. As you pointed out, the reduced power cell of $p_1$ remains unchanged when we delete $p_7$. However, consider the case where $\Delta^{2} = \{ p_1, p_3, p_7 \}$, which also does not exist in the WDT. In this scenario, the reduced power cell of $p_1$ changes when we delete $p_3$ and $p_7$ because $p_1$ and $p_3$ are connected in the power diagram. This observation also applies to our (d=3, k=2) case.

---

Q7. Approximating the reduced power cell

A7. We briefly mentioned approximating the reduced power cell with pre-computed PD but did not provide details. To elaborate, as we compute PD at every optimization step, we collect the points that comprise the power cell of a point $P$ at each step. By keeping and accumulating these points, we can effectively find possible half-planes that could comprise the reduced power cell of $P$. Therefore, we use the half-planes between $P$ and the (accumulated list of) points to approximate the (boundary of) reduced power cell of $P$. Although this approximation is not very accurate in the initial optimization steps, it does not undermine the optimization process. This is because, as optimization progresses, most possible half-planes to define the reduced power cell are collected for each point. We will include this detail in the revised version of the paper.

---

Q8. Non-differentiability of regularization terms

A8. For min/max functions, we used a smoothed version because they are used to compute face probabilities, a main component of our approach. We found that differentiable min/max functions stabilize the optimization process. However, we used absolute values and lengths without modification because they are almost differentiable and used mainly for regularizers, which did not harm the optimization process.

---

Q9. Handling unseen parts of the model in multi-view reconstruction

A9. In multi-view reconstruction tasks, we use real regularization as described in Appendix C.5. This regularization automatically fills unseen parts of the object (e.g., inner parts). In the post-processing step (Appendix D.2.7), we discard most of these automatically generated, invisible faces. Therefore, when a particular side of the model is not depicted, that part would be reconstructed as the maximum volume that does not violate observations from other views. We will include an example of this case in the revised version of the paper.

---

Q10. Vertex-based reality values

A10. We did not consider edge or face-based reality values because we aimed to store all mesh information in the points. This design will simplify its adoption in other ML frameworks, such as generative models. It is much easier for neural networks to generate point-wise features than generate unfixed edge or face combinations. By storing all connectivity information in points, we avoid generating face combinations. It is also more efficient since the number of edges or faces becomes excessive as the number of points increases. Storing real values for such edges or faces is overwhelming. However, edge or face-based reality values could be useful in the final optimization step to fine-tune geometry and remove non-manifoldness, as discussed in Global Response A2.

We appreciate your detailed comments and positive evaluation about our work. Please let us know if we fully address your questions.

---

Q1. Description about running time of “Differentiable Surface Triangulation”

A1. Thank you for pointing out the error. We acknowledge that our description was incorrect. We will change the description to "quadratic" instead of "exponential" in the revised version.

---

Q2. Interpolations between two DMeshes, especially with topology/connectivity changes

A2. Please see Global Response A3. Additionally, please refer to Q8 below to understand how we prevent our formulation from becoming too binary or non-differentiable.

---

Q3. Examples of non-manifold meshes, estimation of frequency, and reasons for occurrence

A3. Please see Global Response A2.

---

Q4. Point cloud reconstruction results with varying densities

A4. Thank you for the suggestion. We will include experimental results in the final version of the paper to demonstrate how the quality of mesh reconstruction depends on the number of sampled points. We expect the results to degrade as the number of sampled points decreases since we are optimizing only Chamfer distance loss without any topological assumptions. Developing a more robust reconstruction algorithm based on our method would be an exciting future research direction.

---

Q5. Evaluation on a larger test set

A5. Thank you for the suggestion. We will include more examples from the Objaverse dataset in the final version of our paper.

---

Q6. Minor typos

A6. Thank you for identifying these. We will fix them in the revised version.

---

Q7. Freezing surface topology

A7. As we understand, surface topology corresponds to geometric topology (e.g., genus of the shape) rather than mesh topology (individual edge connectivity) (Appendix A). If this is the case, we can freeze the surface topology by freezing all mesh connectivity, as you suggested. However, if you refer to changing mesh connectivity while freezing surface topology, our method does not support this functionality yet, because our method does not make any specific assumptions about surface topology during optimization. This would be an interesting research direction to explore. Please let us know if we understood this question correctly.

---

Q8. Preventing optimization from converging to binary values of probability

A8. To answer this question, let us consider two cases where the existence probability of a face converges to either 1 or 0.

First, if the probability has converged to 1, but we want to decrease it, we can simply adjust the real values stored in the 3 vertices of the face to reduce its probability. Since $\Lambda_{real}$ does not include any nonlinear function like sigmoid, we do not worry about falling into a local minimum due to the vanishing gradient problem.

Next, let us assume the probability has converged to 0, but we want to increase it. There are two possibilities for the probability converging to 0: $\Lambda_{wdt}$ becoming 0 or $\Lambda_{real}$ becoming 0. If $\Lambda_{wdt}$ is almost 0, we did not use any measures because there are many possible combinations with very small $\Lambda_{wdt}$. Enumerating all possible face combinations (up to $n \choose 3$) and computing $\Lambda_{wdt}$ for them requires significant computational cost, making it impractical to amend this problem efficiently. However, using WDT is beneficial because the face existence probability usually increases as its vertices become closer. Thus, we observed that reconstruction loss moves points to minimize the loss, usually increasing (formerly very small) $\Lambda_{wdt}$ of the desirable face.

If the probability of a face converges to 0 because $\Lambda_{real}$ is 0, we need a measure to prevent it because we ignore faces with very small probability to reduce the computational burden (Appendix C.3, C.5). These faces cannot get gradients to escape zero probability without any measures. Hence, we introduced the real regularizer in Appendix C.5. The regularizer increases the real value of points connected to points with high real values, helping escape local minima effectively.

We will include this detailed discussion about local minima in the revised version of the paper.

---

Q9. Existence and uniqueness of $\Lambda_{wdt}$ and $\Lambda_{real}$.

A9. Thank you for the question. We believe $\Lambda_{wdt}$ is related to the tessellation of the spatial domain, and $\Lambda_{real}$ is about selecting desired faces from the tessellation. Thus, $\Lambda_{real}$ depends on $\Lambda_{wdt}$: if tessellation changes, selected faces may change accordingly. When $\Lambda_{wdt}$ is fixed (and thus tessellation is fixed), we could argue that $\Lambda_{real}$ that globally minimizes the reconstruction loss would exist and be unique.

Regarding $\Lambda_{wdt}$, we refer to literature on WDT [1]. Certain constraints, like those related to Local Feature Size (LFS), guarantee we can recover a topologically equivalent object to the ground truth mesh by selecting faces from (W)DT. We believe this applies to our case and guarantees the existence of a solution theoretically. However, for uniqueness, we do not think only one $\Lambda_{wdt}$ reconstructs the given geometry. For instance, we can subdivide faces on the current DMesh by inserting additional points or end up with different $\Lambda_{wdt}$ due to different initialization. Thus, we believe we can theoretically guarantee the existence of $\Lambda_{wdt}$ but not its uniqueness.

[1] Cheng, Siu-Wing, et al. Delaunay mesh generation. Boca Raton: CRC Press, 2013.

Thank you for the great questions, it inspires us a lot. To answer your additional questions,

---

1. We have not embedded any specific behavior to prefer $\Lambda_{real}$ over $\Lambda_{wdt}$, or manually optimize $\Lambda_{real}$ instead of $\Lambda_{wdt}$ in the optimization process. Technically, $\Lambda_{real} = 1$ and $\Lambda_{wdt} = 1$ would get the same amount of gradient, because they are just multiplied together to get the final face existence probability (Eq. 2). However, note that $\Lambda_{wdt}$ includes sigmoid function in its formulation (Eq. 4). Therefore, the parameters that are fed into the sigmoid function could get very small gradient when $\Lambda_{wdt}$ is almost binary. In contrast, $\Lambda_{real}$ does not have sigmoid function in its formulation, which means that we do not have to worry about vanishing gradient problem for $\Lambda_{real}$. In this case, the optimization process would "naturally" optimize $\Lambda_{real}$ instead of $\Lambda_{wdt}$, because $\Lambda_{wdt}$ would not be very optimizable. So we intended to say that even when $\Lambda_{wdt}$ is binary, optimizer would be able to adjust $\Lambda_{real}$ to escape the binary state.

---

2. As described above, if $\Lambda_{wdt}$ is almost binary, optimizer would automatically adjust $\Lambda_{real}$ instead of $\Lambda_{wdt}$. And as far as we understand, your concern is about only manipulating $\Lambda_{real}$ instead of $\Lambda_{wdt}$, even when we could get better optimization result when we change $\Lambda_{wdt}$ instead to get better tessellation. Correct us if we are wrong, but if our understanding is correct, our current approach does not care about this possibility yet. Our current algorithm minimizes the loss function in blind manner right now, regardless of specific $\Lambda_{real}$ and $\Lambda_{wdt}$. Therefore, we would have to say it finds the local minimum near current tessellation, instead of finding the global minimum near the optimal tessellation. However, we believe that the coarse-to-fine strategy in Appendix D.2 is a very good method to re-initialize $\Lambda_{wdt}$, thus tessellation, near the global optimal tessellation. Therefore, we'd like to suggest to use that method to find the global minimum.

---

3. Yes, as you mentioned, the optimization loss was defined with multi-view inverse rendering. The reason about the triangle size is related to the answers above. Because $\Lambda_{real}$ is usually easier to optimize than $\Lambda_{wdt}$, it is optimized faster. That is, while the entire tessellation does not change a lot, only the faces that we select change fast.

To elaborate, at the end of the first optimization to fit DMesh to a single torus, most of the points are gathered around the torus surface. Since there are not so many points located at the places far away from the torus surface (as we remove unnecessary points using regularization), the faces in that region become larger than those located near the torus surface. Then, we start from that DMesh to fit different shapes like double torus, and in that process, rather than moving (single) torus surface toward the target shape, the optimizer just removes the undesirable parts on the torus surface by adjusting $\Lambda_{real}$. And then, it makes the probability of desirable faces larger, and they usually have larger faces, because they were located far away from the original single torus. After that, $\Lambda_{wdt}$ is optimized slowly to fine-tune the shape.

After this process, if we re-initialize DMesh using the coarse-to-fine strategy, then it would get much better initial tessellation to start from. Then, it would end up in much smaller, fine-grained faces in the end.

We believe it would be very interesting to explore if we can impose any kind of guidance to select (or prefer) between $\Lambda_{wdt}$ and $\Lambda_{real}$ during the optimization. Thank you again for the inspiration.

Thank you for the detailed response.

For Q8:

1. when you say "we can simply adjust the real values stored in the 3 vertices", do you mean that the optimization process naturally has gradients that minimize $\Lambda_\text{real}$ or that you need to embed this behavior separately?
2. In the case of the probability of face existing = 1, are there any cases when we would prefer the optimization process to decrease $\Lambda_\text{wdt}$ instead? I am curious to understand if the over-parametrization caused by having two parameters ($\Lambda_\text{real}$ and $\Lambda_\text{wdt}$) might cause any kind of dead-locks in the optimization process. As you mentioned in Q9, for a fixed tesselation there is a global minimizer in terms of $\Lambda_\text{real}$, but I believe that to efficiently optimize from one surface to another we will need to change tessellation first and only then optimize $\Lambda_\text{wdt}$.
3. Thank you for providing the mesh-to-mesh optimization experiment. This is very interesting. Do I understand correctly that the optimization loss was defined through multi-view inverse rendering? I noticed that the tessellation has changed and that in both cases the meshes seem to get bigger triangle size. Do you have any intuition why this happens?

Thank you for the fast reply. This is very interesting. It will be interesting to look if stopgradient or just rescaling the gradients will help to define the balance between $\Lambda_\text{wdt}$ and $\Lambda_\text{real}$.

I think the mesh-to-mesh example will be a good addition to support the claims of the paper, so please consider including this in the final version.

At this point, I don't have any more questions and will gladly adjust my recommendation.

We appreciate your comments, but we believe there may have been some major misunderstandings about our work. We have tried to address your comments as thoroughly as possible. If we have not correctly addressed your concerns, please let us know.

---

Q1. Weakness about “depth testing”

A1. We believe there was a misunderstanding regarding the term "depth test" in our manuscript. We used this term to describe the differences between the two renderers we employed in the multi-view reconstruction task (Appendix C.3.1, C.3.2).

In estimating the relative ordering between faces, the first renderer $F_A$ (Appendix C.3.1) can produce incorrect orderings because we compute the depth value of each face globally, which might not align with local orderings. This could result in inner faces being perceived as “seen” from the outside, leading to false inner structures (Figure 11(a)). This renderer is used in the first phase of optimization.

In contrast, the second renderer $F_B$ (Appendix C.3.2) produces precise orderings between faces, thereby removing these false inner structures (Figure 11(b)). This renderer is used in the second phase of optimization, not during post-processing.

In the post-processing step, we remove unseen inner faces based on the observation inherent to our approach (Appendix D.2.7), which tessellates the entire domain. Since other methods do not use this kind of tessellation, this approach is not applicable to them.

We acknowledge that our use of the term could have been misleading. However, as described above, it differs from conventional "depth test" or "visibility test" techniques that can be applied to any reconstruction method. We used the term to describe a situation inherent to our approach, not a general case. We did not perform any conventional "depth test" or "visibility test" to remove inner structures. We will clarify this point in the revised version.

---

Q2. During the post-processing step, the inner structures are removed via depth testing. How does the method handle the boundaries caused by the previous operation?

A2. As described above, the post-processing step in our method is based on principles inherent to our approach, not a general depth testing. Therefore, it does not produce additional boundaries on its own or get affected by boundaries produced in previous optimization phases.

---

Q3. In line 91, "... and it is not." should be completed for clarity. Maybe the authors missed some words at the end.

A3. Thank you for pointing this out. We intended to write: "... and is only confined to handle point clouds.” We will correct this in the revised version of the paper.

---

Q4. The DMesh is constructed from the normal mesh. However, in real applications, we often do not have access to the mesh. How does the DMesh help in reconstructing the result?

A4. When we reconstruct a mesh from point clouds or multi-view images, we do not use the normal (ground truth) mesh as input. In our experiments, we sample point clouds or capture multi-view images of the ground truth mesh, which are then fed into the reconstruction algorithm. The ground truth mesh is used only to evaluate the reconstruction quality. Thus, our reconstruction algorithm can be used even when there is no normal mesh available. Please also refer to the videos about point cloud and multi-view reconstructions in the index.html file in our supplementary material. These videos demonstrate that our method does not require the normal mesh as input.

It seems to me that your global answer is already visible to reviewers.

AC