Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials

We thank the reviewer for the review and the feeback!

1. Lack of support for material components such as index of reflection, scattering, coat, sheen etc.:

Our model indeed only tackles essential PBR parameters (albedo, roughness and metallness) as these are the most important ones, and also the focus of 3D artists when they work on real-time applications like video games. These real-time applications are also the most likely to benefit from AI-generated content as they require a large quantity of 3D assets and, depending on the game, the quality of these assets need not to be as high as, say, a CG for a Hollywood production. Additionally, simple PBR textured assets can serve as a starting point for artists to extend the material map, making it a good place to start from.

Note that in the field of generative 3D asset creation, to our knowledge, there are no methods that go beyond the simple PBR model (albedo, metallness, roughness). Our approach can in principle be extended to predict additional parameters (e.g., say the full Disney BRDF), given sufficiently representative training data.

We’ll add a note in the final version to clarify these limitations.

2. Performance on anisotropic and glass like materials.

We also do not tackle transparent materials yet. Our approach could also be extended to those in principle, again given sufficient training data, but less trivially as it would require modifying the deferred shading part to support transparency.

Thank you for the review and the helpful comments!

1. Technical Novelty:

To our knowledge, Emu3D is one of the first academic work that performs text-to-3D with PBR in a feed-forward way. We introduce several novel aspects that come together to achieve high quality and efficient text-to-3D reconstruction with PBR materials. These include:

(i) A novel albedo + shaded RGB grid generation paradigm, with the intuition that the combination of these two modalities can help the reconstruction stage in inferring materials (ablated in Table 1, Fig. 3 and Video).

(ii) A novel deferred shading loss that regularizes the predicted materials in an efficient manner (ablated in Table 1, A.6.3, Fig. 12 and Video).

(iii) A Volumetric SDF renderer that extends Lightplane with Triton kernels for supporting large batch sizes, image resolution, and denser point sampling on rays. Additionally, we enable a direct loss on the SDF values (A.6.4), also implemented as Triton kernels for scalability (ablated in Table 3, Fig. 4 Row 4 vs 5 and Video).

(iv) A novel texture refinement module that uses cross-view attention layers, which greatly improves the fidelity of reconstructed textures by operating in UV space (ablated in A.7 and Fig. 14, Fig. 4 Row 5 vs Row 6, Table 1 and Table 3, and Video).

2. Facilitating Comparisons:

This is a great point and we are in discussion to open the data as much as possible. We also plan to release the PBR assets generated on the dreamfusion prompts, which are a typical benchmark for comparing text to 3D models. Note, however, that we do use publicly available GSO dataset for sparse-view image to 3D reconstruction. The reconstructed assets using our method on GSO will also be released.

3. PBR resolution:

The resolution of our 2x2 grid is not 512x512, it is 512x512 per image in the grid, making the grid resolution to be 1024x1024 (albedo and shaded, each 1024 x 1024 x 3). These images are input to EmuLRM, which produces UV textures with resolution 512x512 each for albedo, metalness and roughness. These are further upscaled to 1024x1024 resolution textures by the texture refiner.

4. Comparison to Unique3D:

Unique3D upscales the 4 view grid to a resolution of 2048x2048, compared to our 512x512 per image resolution, which can potentially impart higher detailed textures. However, Unique3D only predicts RGB colors, not texture maps, and lacks material outputs as well. This prohibits asset use in novel environments due to baked in lighting in the albedo.

Further, we show a qualitative comparison to more recent methods like Unique3D as well as Stable 3D which came out last week in the common response PDF. The quality of assets produced by our method is visually more appealing than both.

We will add this discussion and the comparisons to the final draft.

We thank the reviewer for the review and the feedback on our work!

1. Use of SDF instead of opacity fields:

We do not claim that using SDF in 3D reconstruction or generation is novel per se (e.g., StyleSDF, SDFusion, GET3D, Latte3D, etc. uses it).

We do note that we are among the first to use SDF in LRMs specifically. Other works that do the same include MeshLRM, InstantMesh, Large Tensorial Model (LDM) and Direct3D, but these are concurrent (released either after or just before the submission deadline).

What makes our approach stand out, even among these more recent papers, is the fact that we use the SDF formulation directly, in a single “phase”. Other papers usually require a first stage training with opacity fields, followed by a second stage where the opacity field is replaced with SDFs / meshes. In other words, we show that two stages are unnecessary.

Furthermore, we do not simply exchange NeRF opacity fields with a VolSDF formulation. Instead, we implement the SDF renderer by extending the open source Lightplane Triton kernels supporting large batch sizes, image resolution, and denser point sampling on rays. Additionally, we enable a direct loss on the SDF values (A.6.4), also implemented as Triton kernels for scalability (note that a naive pytorch implementation of such a loss only allows a batch size of 1 image on an a100 GPU at our base resolution due to increase memory requirements). While these are “engineering” contributions, they are key to making this a practical method.

Note that we compare against the concurrent works MeshLRM and InstantMesh, which were released before our submission.

2. Texture refinement evaluation, before and after refinement outputs:

Our texture refiner is evaluated extensively both quantitatively and qualitatively:

(i) Table 1: Qualitative evaluation of 4-view PBR reconstruction on the internal dataset. The effectiveness of the texture refiner is evident from improvements in LPIPS and PBR PSNR from config F to config G, where the only change is the addition of the texture refiner. Further improvement from config H to I, again solely due to the texture refiner, underscores its efficacy.

(ii) Table 3: Qualitative evaluation of 4-view non-PBR reconstruction on the GSO dataset. The significant improvement in texture quality, particularly in PSNR and LPIPS, between config C and D, demonstrates the impact of adding the texture refiner.

(iii) Fig. 4, row 5 vs. row 6 (ours without tex-refiner and ours with tex-refiner): Qualitative evaluation shows the effectiveness of the texture refiner, with clear improvements in texture fidelity.

(iv) Fig. 14: Qualitative evaluation further demonstrating the impact of texture refinement.

(v) Supplementary Video: 360 degree view texture refinement ablation for select assets.

For before and after refinement visualizations of uv texture maps, please check the common PDF doc. We will further add UV space improvements to the final draft for better understanding as suggested.

3. Blurriness in Image-to-3D and why not optimization:

We suspect that the degradation in textures is due to the volumetric 3D representation of colors. Since texture is inherently a surface property, extending it into 3D space beyond the surface is inefficient, leading to blurriness.

For instance, this degradation occurs in all volumetric LRM-based methods like Instant3D, Instant Mesh, TripoSR, and NeRF stage of MeshLRM, SparseNeuS, etc. In contrast, approaches like GRM or the mesh stage of MeshLRM, which use pixel-aligned Gaussians and rasterization-based rendering respectively, capture textures much better due to their near-surface or surface representation. Similarly, our texture refinement uses UV representation with rasterization for rendering textures, ensuring textures exist only on the surface.

While optimizing with respect to input images is a viable approach, it has issues: (a) Since the entire shape might not be visible in the input views, seams can appear between optimized visible and unoptimized non-visible areas, leading to incoherence and contrast mismatch. (b) Reconstructed objects are not perfect, leading to projection mismatches between the rendered images of the reconstructed object and the input images. This mismatch can provide incorrect supervision to the optimization objective, as the projections of imperfect 3D reconstructions may not align with the input images. (c) Time cost: Optimization takes a longer time.

A learned approach like our texture refiner can combat both (a) and (b) since the training process learns to overcome these issues in a data-driven manner and can do so with a single feedforward pass, addressing the time cost (c).

Thank you for detailed review and the helpful comments!

1. Degraded PBR quality with only RGB shaded inputs (weakness #1):

Yes, indeed the PBR quality suffers when only RGB shaded inputs are provided (upper half Table 1) since the material decomposition is ambiguous. For this reason, the complete proposed approach for text-to-3D partially mitigates this problem by generating both the shaded and the albedo grid, which enhances the PBR quality (Fig. 4).

2. Evaluation of variance in albedo and shaded RGB prediction (weakness #2):

Yes, for a given text prompt, the albedo and shaded RGB produced by the Emu model can vary due to the stochastic nature of diffusion. However, evaluating the effects quantitatively is challenging in the absence of the absolute ground truth. For evaluating the quality of outputs in the text-to-3D setup, we conducted a user study for user preferences using the fixed model seed, but evaluating the quality of material might be a bit more challenging for a non-professional user study participant.

3. Confusion matrix for metalness (weakness #3):

It is true that, physically, materials are either metallic or dielectric, but computer graphics artists often use non-binary values of metalness, for example to represent materials that are a mixture of dielectrics and metals (e.g., certain rock conglomerates, transitions between oxidized/rusted and polished metal); in our own dataset, a non-trivial proportion of metallic assets have non-binary metalness. This justifies using PSNR or similar regression metric for metalness too.

4. Non uniform lighting (weakness #4):

If the input views are captured with non-uniform lighting, it will indeed hamper PBR reconstruction performance. However, in our application, i.e., text-to-3D, lighting is controlled by fine-tuning the Emu grid generator on 3D models rendered with consistent lighting.This makes it biased toward generating images with consistent across views and with uniform lighting.

5. Failure cases (weakness #5):

We will add a discussion and visualizations corresponding to failure cases in the final draft.

---

Answers to questions

1. Filtering standard for the 10K objects used for fine tuning Emu:

To filter the dataset to obtain the top 10K objects, we first remove those with too low aesthetic score. We then select the top 10K objects with the highest CLIP score (dot product between CLIP of the object render and CLIP of the caption). This number was empirically determined, as including more assets, i.e. including assets with a lower CLIP score, results in decreased text-image alignment. To control material quality, we only select assets with valid metallicness and roughness maps. Compositional quality is generally low, as a vast majority of assets in the dataset are single objects rather than scenes.

2. Details about the dataset:

We use an internal proprietary dataset which is similar to Objaverse. The usage has been scrutinized by the professional legal counsel, which raised no concerns about copyrights. Copyright issues are why we did not use Objaverse.

3. Adapting 3-channel Emu with 6-channel outputs:

We employ a text-to-image diffusion model pre-trained on billions of images annotated with text and expand its input and output channels by a factor of 2 to support simultaneous generation of the shaded appearance and albedo (Section A.6.1). This requires only the input and output layers to be adapted.

4. Figure 7 results:

Yes, correct, the results are deterministic. Here, the task is sparse view reconstruction from 4 posed RGB images (without the albedo input), resulting in 3D meshes and their PBR materials. The results correspond to config G in Table 1, which shows the mean PBR PSNR on the whole test set. For the 6 images shown in the figure, the mean PSNR is 23.49 for albedo, 24.91 for metallicity, and 21.02 for roughness, with the standard deviation of (2.72, 3.86, 4.17) respectively. The object selection for the figure was intended to target objects with a non-uniform metalness or roughness (not just a single value) to make for an interesting visualization and highlight the model’s effectiveness.

5. Comparison to Richdreamer:

Please check the PDF in the common response.

6. How are albedo predictions used by the LRM:

In the vanilla Instant3D LRM, an image encoder encodes patchwise feature tokens of the image using a pre-trained DINO network (Caron et al. 2021). LightplaneLRM further extends this by adding Splatter layers on the DINO outputs in the subsequent image-to-triplane transformer. For accommodating additional albedo predictions, we add the encoded albedo patchwise features to the DINO network. This enables access to albedo as well as shaded image information to DiNO, and therefore subsequently to the image-to-triplane transformer and the splatting layers.

We will add these details to the final draft.

7. Qualitative Luma Genie 2.0 results:

Please check the PDF in the common response.

8. Runtime breakdown:

Emu image generation is 20s, LRM reconstruction is 2s, meshification is 8s, and texture refinement < 1s.