GaussianAnything: Interactive Point Cloud Flow Matching for 3D Generation

Qualitatively image-to-3D performance. We have included the quantitative metrics for image-to-3D in Tab.2 as requested, which extensively evaluates both the rendering and geometry generation quality. Specifically, we report the CLIP-I, FID, KID, and MUSIQ metrics for visual quality evaluation, and Point Cloud FID (P-FID), Point Cloud KID (P-KID), Coverage Score (COV), and Minimum Matching Distance (MMD) for 3D quality metrics. As demonstrated, our method achieves state-of-the-art 3D generation performance over all the benchmarks, except for FID/KID metrics. Note that PSNR/LPIPS is not reported since different 3D generation methods have different output scale and canonical poses, which make it hard to align the predictions accurately. CLIP-I serves the same functionality here to evaluate the alignment between input and generated 3D.

Clarifications on the conditions. GaussianAnything does not support multi-view images directly as input for now, and we have revised the claim in the abstract accordingly. For the point cloud condition, since our stage-2 diffusion model adopts both point cloud and image/text as the inputs, the point cloud condition is intrinsically supported by the GaussianAnything framework. Besides accepting generated/edited point cloud in stage-1, we demonstrate in Fig. 11 that our method also adopts point cloud generated from other pre-trained point cloud generative models, such as Point-E. This verifies the generalizability and flexibility of our proposed method.

Clarification on the flow matching objective. “Velocity-prediction” is the default prediction setting in flow matching, as opposed to noise prediction in the DDPM training. In this paper, we also adopt the velocity prediction and follow the unified notations from SD3 in Eq. 6 and Eq. 7. Briefly speaking, the objective transforms the velocity-relevant terms into the weight part, so the conventional eps-pred diffusion loss is still used here. The detailed derivation is included in the appendix A.3, from equations (13)-(27).

Insights of 3D VAE training. Regarding the 3D VAE training, some key designs help.

1. To maximize the benefits of the transformer's flexibility for 3D-aware operations, we incorporate it extensively into our design. Specifically, we utilize transformer-based architectures for multi-view attention in the VAE encoder and for 2D Gaussian Splatting (2DGS) up-sampling in the decoder. Additionally, adopting a pre-norm architecture effectively stabilizes training, ensuring more reliable convergence.

2. We also prioritize incorporating as many 3D attributes as possible into the input. As shown in Table 3, using a rich set of attributes, including RGB-D-N-Plücker coordinates, significantly enhances the quality of 3D encoding by providing comprehensive geometric and photometric information.

3. For 3D latent prediction, we use ground-truth sampled point clouds as anchor points. During our experiments, we found it challenging to regress z_0​ directly from multi-view inputs, likely due to the difficulty of spatial compression for point clouds, which demands specialized operations. To address this, we use the XYZ coordinates sampled directly from the 3D object surface as z_0​, augmented with learnable z_x​, as illustrated in Fig. 1. This approach facilitates accurate latent prediction while leveraging the spatial structure of the point cloud.

4. Finally, in the 2DGS up-sampling stage, we predict residuals instead of directly generating final outputs. This residual-based approach improves the quality of predictions by focusing on refining the details of the generated results.

Evaluations on the 3D generation. As explained in R1, we adopt the same text evaluation pipeline of 3DTopia. For text-to-3D evaluation, we adopt the CLIP score as the common metric and include the CLIP-I, Render-FID, and Point-FID for the image-to-3D evaluation.

More T23D results. We have included more text-to-3D results in Fig. 7, Fig. 9, Fig. 10, and Fig. 11. In Fig. 9 and Fig. 10, we especially verify our method’s capability over complex long captions selected over the DF-415 dataset. To further boost the generation performance, using larger and more diverse 3D datasets like Objaverse-XL and MVImageNet will be helpful. We will explore these designs in the future.

Support of PBR. Our method can directly support PBR by changing the final 2DGS rendering techniques to PBR-based Gaussian rendering techniques, such as [3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes, TOG 2024]. This can be achieved by fine-tuning the pre-trained 3D VAE decoder only with the PBR rendering pipeline.

LoD visualization. We have included the K LoD VAE reconstruction results in Fig. 8 in the appendix. Specifically, 3 LoDs are shown here and reflect the trade-off between rendering speed and quality. Each upsampling is achieved by a two-layer transformer block and requires 4.921, 9.518, and 9.068 milliseconds for the three stages up-sampling correspondingly. The speed is profiled on the A100 GPU with batch size 1.

Discussions on the limitations:

1. Enhancing the 3D VAE Quality: The performance of the 3D VAE could be improved by increasing the number of latent points and incorporating a pixel-aligned 3D reconstruction paradigm, such as PiFU (ICCV 2019), to achieve finer-grained geometry and texture alignment.

2. Incorporating Additional Losses in Diffusion Training: Currently, the diffusion training relies solely on latent-space flow matching. Prior work, such as DMV3D (ICLR 2024), demonstrates that incorporating a rendering loss can significantly enhance the synthesis of high-quality 3D textures. Adding reconstruction supervision during diffusion training is another promising avenue to improve output fidelity. 3. Leveraging 2D Pre-training Priors: At present, the models are trained exclusively on 3D datasets and do not utilize 2D pre-training priors as effectively as multi-view (MV)-based 3D generative models. A potential improvement is to incorporate 2D priors more effectively, for instance, by using multi-view synthesized images as conditioning during training instead of single-view images.

4. Expanding Dataset Diversity: Utilizing more diverse and extensive 3D datasets, such as Objaverse-XL and MVImageNet, could further enhance the quality and generalizability of 3D generation.

By addressing these aspects, the proposed method could achieve significant advancements in both the quality and versatility of its 3D generation capabilities.

1. Qualitative evaluation issue

Evaluation image issue. As explained in A.2, we adopt the GSO renderings from the official Free3D repo for qualitative evaluation. After careful checking, we observe that their rendering adopts a single point lighting and some viewpoints of the object are constantly dim. We replaced those images in the updated Fig. 3 with bright and clear in-the-wild images for visual comparisons. More visual results are also included in Fig. 7.

Viewpoint issue. The baselines have different poses since they have different canonical poses. Also, for LRM-like work, their canonical pose is dynamically determined by the input view. Therefore, objects generated from different baselines have different results even when rendered from the same given pose. We alleviate this issue by manually aligning the poses in the updated Fig. 3, with the video comparison in the supplementary (371-supplementary/rebuttal/371-i23d-video.mp4).

2. Editing issue. We mean to disjoint the two parts here to demonstrate that editing on the latent code-based 3D editing yields smoother and more reasonable results, while 3D-space direct editing shows tearing artifacts. Off-the-shelf 2D editing techniques like self-guidance (Epstein et al, NeuIPS 2023) can be applied to achieve more versatile 3D editing (e.g., 3D drag). We leave these explorations as the future work.

3. More visual results. More visual comparisons on image-to-3D are included in Fig. 3 and Fig. 7, and generations with more complex captions are included in Fig. 9, Fig. 10, and Fig. 11.

4. Writing issues We have revised the parenthesis missing issue in the updated version. In Ln 249-251, we use the term set to denote un-ordered 3D latent space with no explicit 3D structure. This terminology is aligned with “3DShape2VecSet”, and we will revise the writing accordingly. In Ln 252-254, we have revised the writing of “manifold” to “the sparse 3D point cloud” for accuracy.

Statement of diffusion model training We have updated the original statement to ensure greater accuracy and clarity. While we acknowledge that the original statement was inaccurate, the advantages of our proposed latent space remain valid due to the following reasons:

1) Native 3D Diffusion: Multi-view image diffusion models are not true 3D diffusion models and often face view inconsistency. In contrast, our point cloud-structured 3D latent space enables training native 3D diffusion models, addressing this limitation directly.

2) Scalability: Compressed latent spaces do not hinder scalability. Models like Stable Diffusion and DiT demonstrate excellent scalability by training in compressed spaces. Similarly, our representation scales by increasing point clouds, enabling higher-dimensional latents with greater capacity, benefiting from larger models and datasets.

3) Improved Geometry: Tab. 2 shows our method consistently outperforms multi-view reconstruction approaches like One-2-3-45. This highlights our approach’s suitability for 3D diffusion training, achieving superior geometry reconstruction at lower training costs.

Experiments comparison

Experiment setup. For the quantitative text-to-3D evaluation, we follow 3DTopia and use the prompt list proposed by “Gpt-4V is a human-aligned evaluator for text-to-3d generation” (CVPR 2024), which comprises 100 prompts generated by GPT-4.

Evaluation metrics. We include CLIP-I, FID, KID, and MUSIQ for rendering metrics and Point Cloud FID (P-FID), Point Cloud KID (P-KID), Coverage Score (COV), and Minimum Matching Distance (MMD) for 3D quality metrics.

Image-to-3D. We include extensive evaluation on the image-conditioned 3D generation performance based on the above metrics in Tab. 2. As can be seen, our proposed method achieves the best performance except for FID (second-best). The evaluation details are also updated in the paper.

More text-to-3D examples with complex captions. We present additional text-to-3D examples in Figures 7, 9, 10, and 11, using captions of varying complexity to showcase our method’s generalization and robustness. For fair comparison, most captions follow 3DTopia's style, with Figure 10 featuring more complex DreamFusion-inspired captions. Unlike optimization-based methods like DreamFusion, which leverage 2D diffusion priors trained on complex captions, our native 3D diffusion model is trained from scratch on Cap3D. Despite this, our method delivers comparable or superior results. While improved 3D caption annotations could enhance performance, this is a data-related issue beyond the scope of this paper.

Qualitatively image-to-3D performance. We compare against strong competitors like Lara, which uses One2345++ for stage-1 multi-view generation, and demonstrate superior 3D reconstruction. Additional qualitative comparisons on challenging in-the-wild data are provided in Figures 3 and 7, with video renderings in the supplementary. Unlike Lara, which struggles with view consistency, our native 3D diffusion model enables stable, end-to-end 3D generation with intact and faithful reconstructions.

DINO features and conditioning mechanism

DINO features. Leveraging DINO features for image conditioning is standard practice in 3D generation, as demonstrated by recent works like Instant3D (ICLR '24), LRM (ICLR '24), CLAY (TOG '24), and LGM (ECCV '24). Besides, we use the DINO ViT-L network with a 518×518 resolution input and 14×14 patch size, and encode the input image into 1369 tokens. This ensures efficient processing without creating a bottleneck in the 3D generation pipeline.

Image-3D alignment. As shown in Fig. 3, our method surpasses pixel-aligned approaches in novel views, thanks to our native 3D diffusion model design. Supplementary video renderings (371-supplementary/rebuttal/371-i23d-video.mp4) also highlight our method’s superior faithfulness. While methods like Splatter Image perform well over input views, they produce blurry or collapsed novel views. The CLIP-I metric in Table 2 further confirms our method’s superior consistency.

Conditioning mechanism. We argue that we did not claim the CrossAttention-based conditioning as a novelty or a contribution in our paper. We just adopt this modern network design and present it in the paper for clarity.

LRM. The LRM is a deterministic regression model which is designed to take image(s) as the input. To support text-conditioned 3D generation, a high-quality text-to-multiview model like Instant3D is required as preprocessing. However, in this cascaded framework, the 3D generation is intrinsically handled by the Instant3D side, not the LRM side. We have clarified the writing accordingly.

Questions regarding Flow Matching formulation We follow the velocity parameterization of flow matching and directly predict v. For detailed derivation of our formulation in Eq. (6) and Eq. (7), please refer to Sec. A.3 in the appendix, and Eqs. (13)-(27).

Thanks for your timely response and insightful suggestions to our work.

Regarding the evaluation metrics for Text-to-3D, originally we follow the previous methods (3DTopia, LN3Diff) and only reported CLIP score. As suggested, we have added two Aesthetic score metrics (MUSIQ-AVA [A] and Q-Align [B]) as mentioned in L:414 and updated Tab. 1 accordingly. As shown here, our method achieves better performance over the Aesthetic scores by a clear margin. Note that Q-Align [B] leveraging LLMs is the SOTA method for image aesthetic assessment.

Regarding Fig. 10, we mean to include more visual results of our method over long captions and visualize the results over diverse viewpoints and random seeds. We have revised the captions as suggested.

As also mentioned in our "rebuttal reply to Reviewer 21hH", the text-to-3D performance can be further improved by incorporating more diverse 3D datasets and scaling up the model. Currently all 3D-native generative models still fall behind optimization-based (e.g., SDS-based) 3D generation by a clear margin, but support speedy 3D generation within seconds. We believe our method has made a step forward to bridge the performance gap and can inspire future methods to further improve in this direction.

[A] MUSIQ: Multi-scale Image Quality Transformer, Ke et al, ICCV 2021

[B] Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels, Wu et al, ICML 2024

We thank Reviewer cogY for the prompt response.

1. First, we need to emphasize that our method focuses on a native 3D generative model with a point cloud-structured latent space that enables direct 3D synthesis and editing, a capability not achieved with multi-view-based 3D reconstruction methods. While both approaches have their merits, they represent distinct and valuable lines of research.

2. Although our primary goal is not to surpass multi-view methods in visual quality, our geometry evaluation (shown in Tab. 2) demonstrates that our method outperforms them regarding geometric consistency and robustness. Besides, our method achieves comparable in appearance metrics, excelling CLIP-I and MUSIQ scores.

3. We want to point out that multi-view methods often exhibit unstable performance and failure cases in real-world scenarios. This may not be observed in cherry-picked successful cases, but we do observe this in practice and shown in Fig.3. This is because the image-to-multiview stage often generates 3D inconsistent images and thus causes distortion. In contrast, our native 3D methods demonstrate more stable performance across various cases. While GRM's code has not been released, it has almost the same architecture as LGM, which we have already compared. We will include additional comparisons and analysis with stronger multiview baselines in the final version.

Thank you for including the metrics in time!

I will raise my score accordingly; however, I still find the overall generation quality somewhat lacking, as noted by other reviewers. While it is understandable that multi-view diffusion models perform better in this regard due to leveraging foundation image models, this paper does not adequately acknowledge this fact. For instance, the abstract claims that the method "outperforms existing methods in both text- and image-conditioned 3D generation," which is inaccurate. This is also not discussed in the limitations.

Could you consider incorporating a stronger multi-view baseline, such as InstantMesh or GRM (GRM doesn't appear to have source code but there is an official demo, which provides api calls for inference)? It would also be helpful to explicitly clarify that there remains a performance gap between 3D-native models and fine-tuned multi-view models.

Dear Reviewer cogY,

Thanks for your clarifications regarding the concerns.

First, we have updated our manuscript and revised the claim of "outperforming all other methods" and have also included discussion of the limitations compared to multi-view based methods. Specifically, our method outperforms native 3D diffusion model and shows comparative texture quality compared to multi-view based 3D reconstruction methods with better geometry consistency.

Second, though GRM outperforms LGM, it remains within the multi-view generation + pixel-aligned GS prediction paradigm. Since no open-source code is available, we have used InstantMesh as the proxy for state-of-the-art multi-view methods. As requested, we have provided a quantitative evaluation of InstantMesh below. As can be seen, while InstantMesh performs SoTA on the texture metric CLIP-I and FID, it still lags behind our method in the geometry performance.

Moreover, we have included the prior model and GPU resources used in the comparisons. As can be seen, our native point cloud-structured 3D diffusion model does not leverage any existing priors and use the minimum computation resources. Therefore, it is reasonable that our design is currently inferior to the baseline methods regarding the texture fidelity. In the future, we will scale up our model and incorporate strong pre-trained priors to further improve the quality.