TopoGaussian: Inferring Internal Topology Structures from Visual Clues

We appreciate the valuable comments from the reviewer.

1. (W1 & W2 & Q1) Reconstruction error with respect to ground truth structure

Thank you for pointing this out. We perform more quantitative analysis on the accuracy of reconstructed topology compared with GT with 3 metrics, and achieve 3.21x, 7.51x, and 3.47x smaller error than the original baselines on average (Fig.11-13). Since this is a common question among all reviewers, we explain it with more details in the general response.

2. (W3) The proposed method involves multiple steps, such as ..., but lacks thorough validation for these steps.

Thank you for bringing this issue. The reviewer points out two important point to be tested in our pipeline:

• Filling quality
• Input motion quality

We have the following ablation studies to test the robustness of these parts in our original paper:

• In Fig.19, we test the influence of different filling resolutions, where we find that the results are not affected by different resolutions and output similar results
• In Fig.22, we test the impact of different video quality, including light condition and view point, where the results also remain similar, proving the robustness of our pipeline.

Moreover, we have performed another ablation study in Fig.20 to test the robustness with different material properties and object size, which also strengthen the robustness of our pipeline

3. (Q2) The article mentions that the volumetric shape generation stage is faster than mesh-based methods. Is this stage entirely consistent with the Phy(s)Gaussian?

Thank you for asking this valuable question. We do not find ''PhyGaussian'' on the Internet and guess the reviewer means ''PhysGaussian''. The point cloud filling process (which transfers the surface point cloud from GS to a volume point cloud) in our pipeline is similar to PhysGaussian. On the other hand, since PhysGaussian does not contain topology optimization, the topology representation part, including the point and surface representation (neural implicit surface and quadratic surface) is not relevant to it.

4. (Q3) What is the relationship between the metrics used in the article—reconstruction quality and optimization loss?

Thank you for bringing this issue. We would like to discuss these two metrics together with the three new metrics (density loss, test loss, and CoM loss) we provide in the general response. There are many metrics we need to consider to measure the result:

1. Motion loss of input measures the difference between simulated motions of results and input.
2. Motion loss of test set (unseen motions) measures the difference of simulated motions between results and test set.
3. Density loss compared with GT measures the difference of output topology (density distribution).
4. CoM loss compared with GT measures the deviation of the center of mass.
5. Smoothness loss of the result (3D printing concerns) measures the average laplacian of the internal surface in output topology.

We use the second to fourth metrics for model evaluation by comparing them with the ground truth (GT) in the testing set. These metrics are strictly reserved for evaluation and should not be utilized during optimization. On the other hand, the differential of the fifth loss is difficult to solve, which makes it hard to be added in the gradient-based optimization process. Therefore, we choose our optimization objective to minimize the first loss function, and use the other four metrics to validate and measure the accuracy and quality of the results. It may be a promising task to add the fifth loss as a regularization term in the optimization process and enhance the quality of the results in the future.

We are grateful for the reviewer’s insightful comments.

1. (W1) Reconstruction error with respect to ground truth structure

Thank you for pointing this out. We perform more quantitative analysis on the accuracy of reconstructed topology compared with GT with 3 metrics, and achieve 3.21x, 7.51x, and 3.47x smaller error than the original baselines on average (Fig.11-13). Since this is a common question among all reviewers, we explain it with more details in the general response.

2. (W2) Multi-video training

This is a brilliant experiment proposal. We have performed new experiments on the synthetic wobble doll and horse examples to leverage this idea. The experiments share similar settings with the original ones, while we generate several different videos (with different view point, light conditions, vibration amplitude, etc.) to perform optimization, with optimization loss based on all the input motions. We can observe that our model successfully outputs a smooth, accurate result with multiple motions as input, which matches the physical characteristics properly, with almost identical final optimization loss. For visual demonstration, please refer to Fig.24 in appendix.

3. (W3) Limitation to simple material compositions

Thank you for bringing up this issue. We would like to clarify the ability of our pipeline in the two mentioned parts:

• Continuously varying material: Our method choose to focus on a dual-material setting due to the application concerns. In practice, it is difficult to manufacture a continuously varying topology structure by traditional methods or 3D printing, which means that a continuous result will be unrealistic to implement in industry. Therefore, we add a sharp sigmoid function in our pipeline to restrict our output to an industrial-friendly dual material result. That said, if we choose a smoother sigmoid function, our pipeline can also optimize a continuous topology structure. We use an experiment on the synthetic wobble doll and horse to demonstrate this, with similar settings to the original ones. The final optimization loss is 2.2e-5, which is similar to the binary material (5.6% difference), while the visualization is shown in Fig.23.
• Multi-object setting: Our pipeline can handle the interaction between objects through the collision-handling system mentioned in Sec 6.3, and has been exhibited in the collision experiment in Fig.21. The limitation here is that although we allow interaction, we can only optimize the topology structure of a single object, which is an intriguing problem for future work.

Thank you for proposing the advice to strengthen our work. We have revised the manuscript based on the advice and will briefly explain the revision.

1. Comparison Against GT

Thank you for suggesting the qualitative comparison with the ground truth. We have added an extra column in Figs. 14 & 15 to demonstrate the ground truth structure. We observe that our method outputs the closest result, which is consistent with the quantitative results reflected in Fig.11.

2. Unseen video evaluation

Thank you for suggesting the additional statistics and qualitative comparison. We have added Fig.26 to visualize one example in our unseen test set, where we observe that the motion of our inferred structure is almost identical to that of the ground truth. We have calculated the PSNR value of our method and the baselines with respect to the ground truth and reported the statistics in the following table:

where we can observe that our method achieves the highest PNSR, leading to the most accurate results. The full statistics are exhibited in Fig.25.

3. Multi-Video Training

Thank you for your advice on the further comparisons. We have updated Fig.24 to add comparison of the multi-video result against the single-video result and the ground truth, where we observe that the multi-video and single video results are generally similar. To analyze the new results quantitatively, we have also calculated their density loss compared with the ground truth and reported the statistics in the following table:

Here we can observe that the multi-video loss is slightly smaller than the single-video loss.

Thank you for your constructive review on the additional experiments! Your feedback has helped us a lot to improve our manuscript. We are also encouraged by your comment on the interior structure reconstruction problem we study, and we hope our work can attract more researchers to investigate this less explored problem and inspire interesting follow-up applications.

If you have any other comments or suggestions on improving our work, please feel free to let us know. Thank you again for your time and review!

We appreciate the reviewer's effort in reading and evaluating our work carefully!

1. (W1&W4) Optimization loss and comparison between the ground truth topology and inferred topology

Thank you for pointing this out. We would like to clarify that the optimization loss in Fig.3 is the residual during the optimization process (the L2 loss between the optimized motion and input motion). It is mainly used to show the two failure cases, mainly demonstrating the loss in the specific optimization case with only qualitative information.

For more general and quantitative metrics to exhibit the effects of optimization, we provide 3 new metrics and perform experiments with our synthetic dataset, achieving 3.21x, 7.51x, and 3.47x smaller error than the original baselines on average (Fig.11-13). Since this is a common question among all reviewers, we explain it with more details in the general response.

2. (W3) It is unclear whether the time/smoothness improvement comes from the GS representation itself or from the authors' method.

We thank the insightful question from the reviewer. We would like to clarify that all methods, including all baselines and our pipeline, is based on GS. All these methods require GS to extract the surface information from the input. The main difference between our method and the baselines lies in the further processing and representation of the information extracted by GS. In more details, the baselines in our paper build the simulator and topology optimizer based on mesh. On the other hand, our method builds a particle-based pipeline by introducing a particle-based differentiable simulator and topology representation (point, neural implicit surface, and quadratic surface), providing a more flexible and various topology representation. Therefore, the time/smoothness improvement comes from this part without relation to GS itself since GS is also used in baselines.

3. (W2) How can the mesh-based representation be made compatible with the rest of the system that is particle-based?

Thank you for bringing this issue. Mesh and particles are only two different geometry representations for the discretization in simulator and optimizer. We only need to perform a traditional particle resampling on the mesh to make it compatible with our pipeline. For example, in rigid cases, we perform a resampling on Eqn.1 and calculate the center of mass by $\mathbf c=\frac{1}{m}\sum \rho_iV_i\mathbf x_{ci}$.

We thank the reviewer for the constructive questions.

1. (W2) Reconstruction error with respect to ground truth structure

Thank you for pointing this out. We perform more quantitative analysis on the accuracy of reconstructed topology compared with GT with 3 metrics, and achieve 3.21x, 7.51x, and 3.47x smaller error than the original baselines on average (Fig.11-13). Since this is a common question among all reviewers, we explain it with more details in the general response.

2. (Q4) Would it be possible again to summarize for me what actually is the main goal and main application of this work?

Thank you for bringing this issue. Our pipeline reads the photos and videos of an object as input, and outputs an interior structure which can explain the motion in the input. In many cases, simply guessing a fully solid structure from surface information will restrict the possibility of physical parameters. For example, if a wobble doll is fully solid, its center of mass will be restricted at a high position and it will be impossible to remain stable. Therefore, many practical applications require us to depict the internal topology in order to satisfy the physical characteristics. The application of our work covers many areas including computer vision, robotics and manufacturing. One example is to build a physical artifact from an online video of an unknown object, and use 3D printing to reconstruct it in the real world, even if the video is synthetic (AI generated). Another potential application is to validate the authenticity given a period of video by analyzing the interior structure of the objects in the video using our pipeline

3. (Q1) Why do we require the internal topology to be known for 3D printing?

Thank you for asking this question. In 3D printing, the printer prints the objects layer by layer, and needs to know the detailed structure of each layer. More concretely, we must tell the printer which part of the layer should be printed (solid part) and which should not (hollow part). This requires us to detailedly describe the interior topology of the object in our printing file.

4. (W1 & Q2 & Q3 & Q5) Writing Problems including typos, writing styles and lack of information

Thank you for providing corrections and suggestions on our writing, and we have fixed those misses in our revised version. Besides, we have added more detailed information on our volumetric representation in appendix A.3, and revised the abstract based on the suggestions in Q5.