InfiniteMesh: View Interpolation using Multi-view Diffusion for 3D Mesh Reconstruction

Thank you for finding our paper's ideas easy to follow and recognizing the improvements in accuracy, computational efficiency, and the enhanced continuity and quality of the generated 3D content. Your feedback is greatly appreciated and encourages us to continue our work in this area.

Please kindly note that we update more experiment results and analysis in our updated paper, please re-download our main paper and supplementary materials

Q1: Combination of existing methods, lacking novelty

A1: We respectfully disagree. The proposed IVI module is definitly different with previous SOTA approaches.

First, as mentioned in lines 54-77 and 216-219 of our main paper, the main motivation of our paper is that simply adding dense views does not ensure better reconstruction results, but requires inter-frame consistency. We propose to solve this issue by introducing view interpolation (IVI module) instead of traditional video-based generation to obtain consistent dense views for 3D reconstruction, which is definitely different with previous SOTA approaches.

As shown in Table. 2, Figure. 3 of our main paper, Table. 7, Figure. 6 and Figure. 7 of the Appendix in our updated paper, where our IVI module shows strong multi-view consistency in the generated video compared with video-based dense view generation methods.

Q2: Evaluation of OOD (out-of-distribution) datasets or real-world datasets.

A2: As mentioned in lines 264-269 and 285-286, we train our model on the synthetic Objaverse dataset and and evaluate on the real-world GSO dataset, so the setting is both OOD and real-world.

Meanwhile, we have included complex results with different input types in Figure. 6 and Figure. 7 in the Appendix of our updated paper, such as from Objaverse image, Artistic image, Photographic image and Real-world captured image. As shown in Figure. 6, as highlighted in red areas, compared with SV3D and V3D, better geometry consistency can be obtained by our method.

As shown in the results in Figure. 6 of the Appendix, better multi-view consistency images can be obtained by our approach, compared with other video-based methods, and differences in the mesh results are highlighted in red areas. For example, our method outperforms other video-based methods with more accurate geometry details in the forklift and cat, while SV3D and V3D show flattened results, treating three-dimensional objects as nearly two-dimensional objects. In the milk case, our approach effectively converts 2D artistic images into consistent multi-view images and intact meshes, maintaining shape consistency that others fail to achieve. Additionally, our method reconstructs more consistent details in the doll's arm, as highlighted in red areas, while other video-based methods result in texture blurring issue.

Q3: Experimental settings in the comparative experiments, and comparisons with LGM.

A3: As mentioned in lines 293-303, we followed the commonly accepted settings and baselines. LGM's performance is compared in both V3D and InstantMesh.

On the other hand, LGM generates multi-view images with text or single image as input for 3D generation, though with different technical approaches, making the comparison reasonable. We add more explanations in the appendix-section. G in our updated paper.

[1] Chen Z, Wang Y, Wang F, et al. V3d: Video diffusion models are effective 3d generators[J]. arXiv preprint arXiv:2403.06738, 2024. [2] Xu J, Cheng W, Gao Y, et al. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models[J]. arXiv preprint arXiv:2404.07191, 2024.

Q4: When training the multi-view enhanced InstantMesh, are the input multiple views derived from the previous interpolation step or directly using ground truth？

A4: Thank you for bringing this up. During the training of the LRM component, we use the ground truths, and we use the predicted images and dense multi-view images obtained from IVI module for mesh reconstruction and texture generation. We will include this clarification in our revised paper.

Q5: Performances of secondary interpolation on the already interpolated views

A5: We do consider this setting, however, as mentioned in lines 469-472 and Table. 5, we present a quantitative analysis on the interpolation number n, proving that n=2 yields the better results. Since further increasing n leads to a limited performance increasing in reconstruction results, which prove that adding too many views cannot improve the model's performance.

Thank you for finding our paper well-written, our IVI module efficient in enhancing multi-view consistency, and that our method's effectiveness was evident through our experiments. Your acknowledgment of our work's strengths is greatly appreciated.

Please kindly note that we update more experiment results and analysis in our updated paper, please re-download our paper for more details.

Q1: More dense multi-views results

A1: Thank you for this valuable suggestion. We provide more dense multi-view images results in Figure. 6 in Appendix of our updated paper with different types of input. We also provide more 360° reconstruction dense images in Figure. 7 of the Appendix to better show the details of our reconstructed mesh.

According to Figure. 6 and Figure. 7, our method outperforms SV3D and V3D with better mesh and texture reconstruction on different types of input, such as such as images from real-world, etc. which prove the generalization abalities of our method. Please kindly note that our model is only trained with objaverse dataset. Please see Figure.6 and Figure. 7 of the Appendix for more details.

Q2: Performance on more complex and diverse scenarios

A2: Thank you for this feedback. In Figure. 6 of the Appendix in our updated paper, we provide results with more complex and diverse scenarios, and differences in the mesh results are highlighted in red areas. For example, our method outperforms other video-based methods with more accurate geometry details in the forklift and cat, while SV3D and V3D show flattened results, treating three-dimensional objects as nearly two-dimensional objects. In the milk case, our approach effectively converts 2D artistic images into consistent multi-view images and intact meshes, maintaining shape consistency that others fail to achieve. Additionally, our method reconstructs more consistent details in the doll's arm, as highlighted in red areas, while other video-based methods result in texture blurring issue.

As illustrated in Figure. 6, taking complex and diverse scenario images as input, our method can generate better mesh and textures.

Q3: Geometry consistency comparisons with natural images as input.

A3: As mentioned in lines 264-269, the GSO dataset we used for evaluation is sampled from real-world objects, and results in our experiment are generated from natural images.

Meanwhile, we have included complex results with different input types in Figure. 6 and Figure. 7 in the Appendix of our updated paper, such as from Objaverse image, Artistic image, Photographic image and Real-world captured image. As shown in Figure. 6, as highlighted in red areas, compared with SV3D and V3D, better geometry consistency can be obtained by our method.

Q4: More ablation studies on the IVI module.

A4: As mentioned in lines 453-470 of our main paper, we conducted both quantitative and qualitative ablation studies on the IVI module, including view interpolation for LRM (Figure. 5), camera trajectories (Figure. 4 and Table. 4), and interpolation number n (Table. 5).

Besides, to fully evaluate the quantitative results, we also add the baseline (Wonder3D) results in Table. 4 of appendix in the updated version of our paper. The baseline results for Chamfer Dist/Vol. IoU/PSNR/LPIPS are 0.0186, 0.4398, 13.31 and 0.2554, while results with our IVI module are 0.0101, 0.6353, 18.27 and 0.1397 which outperform the baseline results with large margins on all evaluation metrics, which proves that our IVI module works positively for dense image generation. Please see Table. 4 for more details. Please kindly note that the baseline results are obtained with wonder3D without IVI module. Meanwhile, our IVI module with elevation can further improve the results.

Furthermore, in Figure. 7 of the Appendix, we also provide more visual ablation study result with and without elevation. As shown in Figure. 7, with elevation in camera trajectories, our IVI module provides more details for video results, showing better quality in the reconstructed mesh, which is highlighted in red areas. For example, the fork of the forklift and the eyes of the dragon are completer and more refined.

Thank you for your rebuttal. I am wondering why are you not providing video results?

Thank you for appreciating the contributions of our IVI module in improving multi-view consistency and mesh quality. Your acknowledgment of our work outperforming current state-of-the-art models is greatly appreciated.

Please kindly note that we update more experiment results and analysis in our updated paper, please re-download our main paper and supplementary materials for more details.

Q1: Inference time comparisons of mesh reconstruction and view interpolation

A1: (1). Mesh reconstruction: Our 3D mesh reconstruction LRM part takes an average time of 1.464 seconds for inference, which is similar with InstantMesh that constructs meshes in an average time of 1.270 seconds.

As shown in Figure 2 (c) and Equation 4 of our main paper, all image tokens are concatenated for subsequent operations. We have a position embedding p∈R^{V,P,D} and a concatenated tensor X∈R^{V,P,D}, where V represents the number of view images. p serves as the query and X acts as the key in the cross-modal attention operation. The matrix multiplication is mainly performed along P and D dimensions. In our experiment, P is 401 and D is 768, which are significantly larger than V. Therefore, increasing V has minimal impact on computational time.

(2). View Interpolation: Our IVI module takes 3.5s for a single view interpolation process. In our experiment, four interpolations are required, the total video generation time is approximately 14s.

The quantitative comparison results with SOTA video generation methods are as follows:

SV3D: 85.198 seconds

V3D: 31.893 seconds

IVI (Ours): 14.324 seconds

Compared with SV3D and V3D, our View Interpolation module (IVI) is with less time consumption. Please kindly note that all results are obtained with a A40 GPU.

Q2: Ablation study of IVI module's camera trajectories

A2: Thank you for this observation. As shown in Figure. 4 of our main paper, taking a rabbit image as input, compared with IVI module without elevation, results of our IVI module with elevation in camera trajectories can reconstruct better mesh with better back and hand (red areas in Figure. 4 of our main paper).

Meanwhile, we add more qualitative results on camera trajectories with more distinctive examples in Figure. 7 of Appendix of our updated main paper. With elevation in camera trajectories, results with our IVI module show better quality in the reconstructed mesh, which is highlighted in red areas. For example, the fork of the forklift and the eyes of the dragon are completer and more refined.

Besides, In Table. 4 of our main paper, we provide quantitative results of our method with different elevation angles (camera trajectories). We also add the baseline (Wonder3D) results in Table. 4 of the updated version of our paper. The baseline results are obtained with wonder3D since we use it as baseline without IVI module. As shown in Table. 4, results with our designed camera trajectories with and without elevation all outperform baseline with large margins, which proves that all our designed camera trajectories work positively for dense image generation, and trajectories with elevation can further improve the performances on all evaluation metrics. This is because that most of the areas can be observed in the trajectory without elevation, and the rarely observed areas can be observed by trajectories with elevation.

Q3: Explanation of loss functions in Equation 7

A3: Thank you for highlighting this. In Equation 7, Lrgb, Ldepth, Lnormal, and Lmask refer to the loss of RGB images, depth, normal, and mask maps of the reconstructed mesh, and Llpips and Lreg refer to LPIPS and regression loss, respectively. We have added this in E of Appendix of our updated paper.

Q4: 360-degree reconstruction videos

A4: Thank you for this valuable suggestion. We provide more visual results in Figure. 6 in Appendix of our updated paper with different types of input. We also provide more 360° reconstruction dense images in Figure. 7 of the Appendix.

As shown in the results in Figure 6, better multi-view consistency images can be obtained by our approach, compared with other video-based methods. For example, our method outperforms other video-based methods with more accurate geometry details in the forklift, cat, milk bottle and doll's arm.

Our method outperforms SV3D and V3D with better mesh and texture reconstruction on different types of input, such as such as images from real-world, etc. Please kindly note that our model is only trained with objaverse dataset.

Q5: Interpolated views for mesh reconstruction

A5: We generate n=2 interpolated views between each pair of adjacent main views. As shown in Table 5 of our main paper, better results on most of the evaluation evaluation metrics can be obtained when set n=2, therefore, we set n=2 in our experiment.

Thank you for appreciating the clean writing, clear problem definition, and our comparison with recent state-of-the-art models. Your feedback is invaluable, and we'll work on addressing the concerns you've raised to improve the clarity and impact of our work.

Please kindly note that we update more experiment results and analysis in our updated paper, please re-download our main paper and supplementary materials for more details.

Q1: Adding dense views for LRM models will help the reconstruction.

A1: In this paper, we argue that dense views with better inter-frame consistencies help the reconstruction, not only dense views. As mentioned in lines 42-43 and 54-70 of our main paper, simply adding dense views does not ensure better reconstruction results. Due to lacking inter-frame consistency, the results of video-based approaches, such as SV3D and V3D, in Table. 2, Figure. 1 and Figure. 3 of our main paper can obtain limited performances.

As mentioned in lines 72-77 and 216-219 of our main paper, our IVI module utilizes two main views as condition and reference for view interpolation, which achieves better multi-view consistency, compared to video-based methods like SV3D and V3D, leading to superior reconstruction results.

We provide more results in Appendix of our updated paper, which proves the effectiveness of our approach. As shown in the results in Figure 6, better multi-view consistency images can be obtained by our approach, compared with other video-based methods. For example, our method outperforms other video-based methods with more accurate geometry details in the forklift, cat, milk bottle and doll's arm.

Please kindly note that we provide results with different types of images as input, such as images from real-world, etc., and our model is only trained with objaverse dataset, which proves the generalization ability of our approach.

Q2: More video results and dense image results to show the quality of this “infinite” multi-view diffusion model.

A2: Thank you for pointing this out. We have added video results in Figure. 6 and Section. B in the Appendix of our updated paper.

As shown in the results in Figure. 6 of the Appendix, better multi-view consistency images can be obtained by our approach, compared with other video-based methods, and differences in the mesh results are highlighted in red areas. For example, our method outperforms other video-based methods with more accurate geometry details in the forklift and cat, while SV3D and V3D show flattened results, treating three-dimensional objects as nearly two-dimensional objects. In the milk case, our approach effectively converts 2D artistic images into consistent multi-view images and intact meshes, maintaining shape consistency that others fail to achieve. Additionally, our method reconstructs more consistent details in the doll's arm, as highlighted in red areas, while other video-based methods result in texture blurring issue.

Q3: Quantitative ablation study on the view-interpolation design

A3: Thank you for pointing this out. As mentioned in lines 462-465 and 467-470 of our main paper, we conducted quantitative analysis on camera trajectories in Table. 4 and interpolation number n in Table. 5.

Besides, we also add the baseline (Wonder3D) results in Table. 4 in the appendix of the updated version of our paper. The baseline results are obtained with wonder3D without IVI module. As shown in Table. 4, results with our IVI module outperforms the baseline without IVI with large margins on all evaluation metrics, especially for "Chamfer Dist", "Vol. IoU", "PSNR" and "LPIPS", which proves that our IVI module works positively for dense image generation.

Q4. The definition of n in equation 3.

A4: Thank you for pointing this out. n represents the number of interpolated images during a single view interpolation process. We update the description in line 214 of our revised paper.

Q5. Training and inference time for different number of input images for LRM model.

A5: Training time and Inference time for mesh reconstruction: Our 3D mesh reconstruction LRM part takes about 1 day for training on 8 Nvidia A100 GPUs and an average time of 1.464 seconds for inference, which is similar with InstantMesh that constructs meshes in an average time of 1.270 seconds.

As shown in Figure 2 (c) and Equation 4 of our main paper, all image tokens are concatenated for subsequent operations. We have a position embedding p∈R^{V,P,D} and a concatenated tensor X∈R^{V,P,D}, where V represents the number of view images. p serves as the query and X acts as the key in the cross-modal attention operation. The matrix multiplication is mainly performed along P and D dimensions. In our experiment, P is 401 and D is 768 which are significantly larger than V. Therefore, increasing V has minimal impact on computational time.

Thank you authors for the detailed response. I still have questions about your last answer. Isn't the training/inferencing time proportional to $V^2$ as you performed cross attention between $p \in \mathbb{R}^{V,P,D}$ and$x \in \mathbb{R}^{V,P,D}$?