SC3D: Self-conditioned Generative Gaussian Model with 3D-aware Feedback

1. Claim about key contributions: the 3D-feedback idea appears already in VideoMV. Since the VideoMV is already available on Arxiv in March, authors need to justify more clearly about the difference and contribution w.r.t. VideoMV.

We will more clearly justify the difference with VideoMV in our revised paper.

• VideoMV aims to generate multi-view consistent images through 3D-aware denoising sampling. However, on the overall image-to-3D pipeline, its (a) lacking joint training and (b) inability to use geometric information hinder its capacity to fully leverage 3D-aware knowledge and unify the two stages. Moreover, this method fails to address the "data bias" between multi-view generation and 3D reconstruction. While its 3D-aware sampling method, employed in the inference phase, can improve multi-view consistency, it also introduces biased information from reconstructed 3D models, resulting in outputs that are misaligned with the input image (see Fig.3 and Fig.4 in our attached PDF).
• Our approach begins with a two-stage image-to-3D generation pipeline, aiming to address the "data bias" problem in the two stages, and thus propose a unified self-conditioning framework to achieve high-quality 3D generation. We utilize both geometry and appearance information from the reconstructed results, incorporating joint training of the two stages. The feedback loops in both the training and inference stage help reduce data bias and generate 3d assets with high-quality geometry and textures adhere to the input image.

2. Lacks generalization results: the method is evaluated on google scan objects, which is standard. However, i am curious to see if the approach generalizes to real world images.

We provide the generalization results in Fig.2 of our attached PDF. Our SC3D can also generate high-quality 3D models from out-of-distribution images and real world images.

3. lacks one ablation: SVD+RGBs feedback, which is missing in Tab. 2 and Tab. 3.

We acknowledge the importance of ablating coordinate map feedback. As demonstrated in Fig. 6 and Tab. 1 in our attached PDF, the reconstructed results in the 4th column show that relying solely on the color map leads to poor geometric quality in fine details. The combination of RGB and coordinates map feedback provides the best results by enhancing both geometry and texture quality, showcasing superior performance.

4. The readability of the Alg.1 and Alg.2 can be improved. Currently it is too specific and looks like python program. A more abstract algorithm is expected in a scientific paper.

We revise the two algorithms and the updated algorithm is shown in the common response.

5. A typo: in line 213, after comma, "we" instead of "We" (wrong capitalized "W")

Thanks for pointing out this typographical error. We will correct it in the revised version.

6. Why there is no SVD+RGB feedback in the ablation study? I think that is important to see how does the coordinates map contribute compared to multi-view RGB images as feedback.

We apologize for our oversight. We have included the qualitative and quantitative results of this setting in Fig.6 and Tab.1 of our attached PDF. The experimental results confirm the necessity of our two feedback types, demonstrating that CCM feedback enhances detailed geometry, while RGB feedback contributes to high-quality texture. The combination of these two feedback types brings greater benefits.

Thank you for your feedback and for bringing these concerns. Our previous response to your questions may have led to your misunderstanding of the core contribution of our approach. We would like to clarify the distinct contributions and the unique value of our method.

• Comparison with SyncDreamer. In terms of specific practices, our 3D-aware feedback differs from the volume features used in SyncDreamer[1]. While many multi-view generation methods, such as MVDream[2], EpiDiff[3], and SPAD[4], employ "3D-aware" features to enforce multi-view consistency, the 3D information they use does not derive from explicit and accurate 3D models. Instead, these methods typically rely on implicit multi-view interactions—like full multi-view self-attention in MVDream, epipolar-constraint attention in EpiDiff and SPAD, or SyncDreamer’s use of latent volumes and projection features. However, these approaches do not accurately reflect the true 3D geometric structure. Our SC3D framework focuses on explicitly modeling 3D geometric structures with more accurate 3D-aware information derived from the reconstruction module. Moreover, the explicit geometric structure provided by our SC3D framework allows it to be adaptable and generalizable across various network designs other than the implicit geometric features that should be trained or fine-tuned in each applied scenario.
• Discussion about LGM. The SC3D framework is designed to seamlessly integrate multi-view generation with reconstruction, demonstrating robust adaptability across different models. While the LGM[5] model is employed in our reconstructions, SC3D's inherent versatility allows it to also incorporate alternative models such as GRM[6], GS-LRM[7], among others. This flexibility enhances the framework's ability to adapt to a wide range of reconstruction models, distinguishing it from more rigid systems that are tightly coupled to specific models.
• Our key contributions. Our SC3D framework integrates multi-view generation with 3D reconstruction, aiming to reduce the "data bias" between these two stages. To address these challenges, we implement a self-conditioning mechanism and employ a joint training approach. Our multi-view generator in our framework models explicit 3D geometric features by incorporating accurate 3D-aware information from the reconstruction model. Our framework is highly extensible and can accommodate various multi-view generation networks and reconstruction networks.

We sincerely hope that you will reconsider the contributions and value of our approach. Looking forward to your feedback again.

[1] SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (ICLR 2024)

[2] MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024)

[3] EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion (CVPR 2024)

[4] SPAD : Spatially Aware Multiview Diffusers (CVPR 2024)

[5] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (ECCV 2024)

[6] GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation

[7] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting

1. Some related works lack citation and discussion: DMV3D and Carve3D.

Thanks for your suggestion, and we will cite DMV3D and Carve3D with more discussions in our revised paper:

• DMV3D employs a 3D reconstruction model as the 2D multi-view denoiser in a multiview diffusion framework, to achieve generic end-to-end 3D generation. However, it does not leverage the powerful capabilities of pretrained image or video diffusion models, and training from scratch on 3D data limits its generalization. In contrast, our unified framework can leverage the visual prior from large image and video datasets, offering greater potential and capacity to tackle complex 3D modeling challenges.
• Carve3D employs a RLFT algorithm with a multi-view consistency metric to enhance the consistency of multi-view diffusion models. This metric is computed by comparing the generated multi-view images with those rendered from the reconstructed NeRF. However, Carve3D does not address the challenge of poor reconstruction quality resulting from limited and differently distributed training data for reconstruction model, which is the second argument for "data bias" we put forward in line 32 of the original paper.

2. I encourage the authors to provide more visual results to help readers understand and appreciate the diffusion/reconstruction process. For example, could the authors provide some visual results of x~0 at different denoising steps?

Thanks for the helpful advice. We visualize reconstruction results at different denoising steps in Fig.5 of the attached PDF. Results show that floaters and distorted geometries are generated in the early stages due to multi-view inconsistency. The quality of geometry and appearance tends to be higher in denoising process.

3. In the comparisons, although quantitative results are provided, could the authors include some qualitative (visual) comparisons to the baseline methods?

We add more visual comparisons to the baseline method and results in Fig.3 and Fig.4 of our attached PDF. We compare our SC3D with SOTA image-to-multiview generation methods and image-to-3D generation methods. The results show that our SC3D generates more consistent and higher-quality multi-view images, and produces 3D assets with superior geometry and textures. We provide our detailed analysis in the common response.

4. Other minor issues about typos.

Sorry for our carelessness. We will fix it in the revised version.

1. The presentation of the paper is not clear.

• Confusion about Fig.1 and Fig.2 in the original paper. We slightly adjust these two figures to ensure that both outputs are 3D representations (The updated figure sees Fig.1 in the attached PDF.). The two decoders are the same VAE decoder, and the two reconstruction models are the same too. We also include symbols $\mathcal{G}$ and $F$ in the updated figure.
• Move the lines 169-172 to where we actually describe the training strategy. We agree with the suggestion and the mentioned lines are moved to the "Training Strategy" section in the revised version.
• Explanation of c_skip in Equ.1. $c_\text{skip}$ is a parameter that controls how much of the original x_0 is retained. Its form is similar to the skip connection and is defined as $c_\text{skip}(\sigma)=\frac{1}{1+\sigma^2}$. The definition can be found in Appendix A.1 of our original paper.
• Typos. We committ to fixing these errors, and have conducted a thorough proofreading of the manuscript.
• Contradictory statement in lines 227-228. Sorry that our description causes some misunderstanding. The statement aims to show that the low speed of volumetric rendering makes it challenging to replace our current Gaussian Splatting reconstruction model with a NeRF-based reconstruction model. We will make it clearer.
• Replacing the algorithm code with more concise pseudo code. We provide the pseudo code in the common response, and will update the manuscript.

2. Comparisons are not very comprehensive.

• None of the methods in Figure 1 are qualitatively compared. We include more comprehensive qualitative comparisons in our attached PDF. As shown in Fig.4, the two-stage baseline methods like LGM and InstantMesh, often generate incorrect or incomplete geometry due to the inconsistency of the generated multi-view images. We also conduct ablation studies on our feedback machinism in Fig.6. The reconstructed results show that our SC3D with full 3D-aware feedback achieves superior performance in both geometry and texture. Please check the common response for detailed analysis and visualization.
• Comparison of different views in Figure 3. We apologize for the confusion and fix it in Fig.4 of the attached PDF.
• Add visual cues in Figure 9. We appreciate the suggestion and will add visual cues as we did in our attached PDF.

3. Some claims are not justified.

• Is the section on augmenting the diffusion model with camera control in 3.1 a claimed contribution of the paper? The usage of Plücker embedding is not one of our paper’s contributions. It is a common positional encoding utilized in several recent works [1~4]. Our approach simply adopts this design.

[1] DMV3D: Denoising Multi-View Diffusion Using 3D Large Reconstruction Model

[2] SPAD: Spatially Aware Multi-View Diffusers

[3] EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion

[4] Free3D: Consistent Novel View Synthesis without 3D Representation

4. My questions are described in the weaknesses section, but the most important is fleshing out the comparisons to understand what sort of improvement this method makes. Having qualitative comparisons to existing methods would go a long way in answering my question.

We've provided our detailed responses in the above reply and the common response. In summary, we conduct detailed visual comparisons in Fig.3, Fig.4 and Fig.6 of our attached PDF. Qualitative comparisons for image-to-multiview (Fig.3) and image-to-3D (Fig.4) generation show that our SC3D generates more consistent and higher-quality 3D assets than other methods. The ablation study on feedback (Fig.6) shows the importance of our feedback loop in improving texture quality and geometric details. We provide the detailed analysis in the common response.

5. Overall the paper does describe some limitations of the method, but it’s not clear if they are relevant. For example, is using the gaussian splatting method really a limitation in this case? I’d be interested to know how long this method takes (is it much slower than LGM baseline), how computationally intensive it is, or how sensitive it is to the initial generation by the multi-view diffusion model.

• Is using the GS method a limitation? We consider that 3D mesh is more commonly used in downstream applications, and there are still challenges in converting Gaussian Splatting to high-quality meshes.

Table B

• Computational cost. We acknowledge that our SC3D has limitations in computation cost, and will include it in the revised version. In Tab.B, we compare the inference time of baseline methods under the same setting. The LGM baseline employs ImageDream[5] to generate 4 views of 256x256 resolution, and reconstruct them into 3DGS. Our SC3D approach uses SVD to generate 8 views of 512x512 resolution. For a fair comparison, we report the inference time of "SV3D + LGM", where SV3D[6] is a multi-view generator fine-tuned from SVD. Compared to "SV3D + LGM", our additional overhead mainly arises from the feedback mechanism at each step, involving VAE decoding, 3D reconstruction and rendering, and conditioning injection. As shown in Tab.B, the inference time remains within acceptable bounds.
• Sensitivity to initial multi-view generation. SC3D is not sensitive to the initial multiviews. Our reconstruction model may produce poor results in the early steps due to the inconsistency of the initial multi-view images. But the quality of the reconstructed results rapidly increases in the denoising process (see Fig.5 in the attached PDF).

[5] ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation

[6] SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion

1. Lack detailed visual comparisons with baseline methods.

We conduct comparisons with more baseline methods, and show the visualization results in Fig.3 and Fig.4 of the attached PDF. We compare SC3D with the SOTA image-to-multiview and image-to-3D generation methods.

• Image-to-multiview generation. We compare SC3D with SyncDreamer[1], SV3D[2] and VideoMV[3], and present the qualitative results in Fig.3 of our attached PDF. SyncDreamer and SV3D fine-tune image or video diffusion models on 3D datasets but do not use explicit 3D information, often resulting in blurry textures or inconsistent details. VideoMV aggregates rendered views from reconstructed 3D models at the inference stage, but it fails to take into account the "data bias" between these two stages. Although VideoMV improves the multi-view consistency, it introduces biased information from reconstructed 3D models, leading to results that are unaligned with the input image. Our SC3D framework involves the joint training of the two stages and uses geometry and appearance feedback for multi-view generation, enhancing the consistency and quality of the generated multi-view images.
• Image-to-3D generation. We compare SC3D with TripoSR[4], VideoMV[3], LGM[5] and InstantMesh[6], with visualization results in Fig. 4 of the attached PDF. TripoSR reconstructs 3D model from a single image without using generative models, resulting in low-quality geometry/appearance and limited generalizablity. VideoMV reconstructs 3DGS from its generated multi-view images. Due to its biased multiview generation (as indicated in Fig.3 in the attached PDF), it may generate inconsistent texture against the input image and somewhat distorted geometry. Moreover, the two-stage baseline methods like LGM and InstantMesh (i.e., an off-the-shelf image-to-multiview generation method plus LGM or InstantMesh to fulfil the image-to-3D generation) produce incomplete or inconsistent geometry due to the gap between the two stages. Our SC3D bridges the multiview generation and 3D reconstruction where the benefits of each module can be transferred to the other one, therefore generating high-quality 3D assets.

[1] SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (ICLR 2024)

[2] SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (Arxiv 2403.12008)

[3] VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model (ECCV 2024)

[4] TripoSR: Fast 3D Object Reconstruction from a Single Image (Arxiv 2403.02151)

[5] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (ECCV 2024)

[6] InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (Arxiv 2404.07191)

2. The paper suffers from poor organization.

We acknowledge the issues with our paper's organization and typesetting, and will seriously polish the manuscript.

• Figures 4 and 5 are currently not referenced in the text. In the revised version, we will ensure that these figures, which illustrate the comparison with baseline methods and out-of-distribution (OOD) testing results, are properly integrated into Section 4 to strengthen our analysis and conclusions.
• The purpose of Figure 6 is confusing, as its caption suggests it shows results from another work. We included Figure 6 in the original version to show that VideoMV's fusion of reconstruction results at the inference stage leads to a deviation in appearance from the input. To avoid any confusion, we will remove Figure 6 and provide a clearer and more comprehensive comparison like Fig.3 and Fig.4 in the attached PDF.
• Typesetting and Formatting. We will carefully re-evaluate the typesetting to eliminate blank spaces and enhance the overall layout.

3. Will jointly training the multi-view generation network and 3D reconstruction network make the training unstable? Will the jointly training mechanism increase the training time? More details about potential disadvantages of jointly training should be discussed in the paper.

• Training stability. Our joint training method is stable. This benefits from the following two aspects. (We will include additional training details in the revised version.)
  1. The pretrained video diffusion model effectively utilizes its powerful visual generation capabilities to produce high-quality initial images.
  2. We initialize the output layers of the condition encoders to zero, ensuring that even suboptimal initial reconstruction results do not adversely affect the network significantly.
• Training time. Jointly training the multi-view generation network and the 3D reconstruction network increases training time and requires more GPU memory, as it involves training two models simultaneously with the feedback mechanism. We've measured the time required for 1,000 training steps on a single A100 GPU with the setting listed in Appendix B of our paper, as detailed in Tab.A. We will further discuss the computation requirements in Section 4.3 (Limitations) of the revised paper. It is worth noting that SC3D has minimal impact on inference speed, as the 3D feedback mechanism incurs only a slight overhead.

Table A

4. As mentioned in Line 211-218, the setting of two ablated experiments seems the same, what is the difference between them?

Sorry for the confusion of the notations in the table. Actually, in the "Variant" column, "SVD" means the metrics for multi-view generation results, while "GS" shows the metrics for the reconstructed 3D results. In the revised version, we will merge Tab.2 and Tab.3 from the original paper to provide a clearer comparison. The updated table refers to Tab.1 in our attached PDF.