MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR

Clarification

It's worth noting that our main target in this paper is to introduce a material prior to text-to-3D to generate 3D objects with proper materials, rather than studying the optimal representation in text-to-3D. As recognized by reviewer Kg7h, our work is the first work to introduce such a prior in this line of research.

Q1: This paper could benefit from a better motivation.

A: Thanks for your suggestion, and we will refine our manuscript based on your comments. It's worth noting that the mentioned IJCV paper is more related to inverse rendering (a reconstruction task), which is conceptually different from text-to-3D (a generation task). In text-to-3D, only a text input provides loose semantic constraints about the shading and the reflectance, leading to a lot of ambiguities and uncertainties that simply applying inverse rendering techniques can not resolve.

Q2: Why was Mip-NeRF chosen? More related work should be included.

A: As discussed in the Clarification, our focus of this paper is on the material side rather than the representation side, thus discussing works on 3D representations in the related work and comparing different 3D representations may deviate from our focus. As mentioned in Section 3.4, we followed previous studies and used DMTet, not Mip-NeRF, as our 3D representation. Finally, we will revise our manuscript to include possible related work on material estimation, BRDF compression, and illumination representations.

Q3: Why was the TwoShotBRDF dataset used? Why was the TwoShotBRDF dataset used? How to handle fibrous objects?

A: As mentioned, our work is the first one to introduce material prior to the task of text-to-3D. Being the first attempt, we choose Cook-Torrance as the material model and TwoShotBRDF as the source of our material prior, as they cover a large portion of possible materials, and we believe they are sufficient to verify the effectiveness and importance of involving such a material prior to text-to-3D. Our experiments also demonstrated this. Specifically, the Cook-Torrance model is a widely used material model in many commercial softwares such as UE4 and academic works. TwoShotBRDF is a large-scale BRDF dataset containing most of the commonly observed materials. In terms of fibrous objects, both its geometry and material are too challenging for current text-to-3D. Involving other material models, as well as adding more datasets to enhance the material prior is worth exploring, which is left as future work. Same as the handlement fibrous objects. We will clarify this in the revision.

Q4: Details about hyperparameters, computational cost, and initial geometry.

A: We will clarify this in the revision. Also, code and model to reproduce our results will be released. For hyperparameters of different losses, we borrow some part of the default setting of Neural-PIL and fine-tune them to ensure four losses decrease simultaneously. For the computational cost, the optimization of each object will take around 40 minutes (20 minutes for geometry and another 20 minutes for appearance) on 4 A100 GPUs, and about 30G memory is used in each GPU. Finally, we follow standard practice in text-to-3D (e.g. Fantasia3D) to select initial geometry, which is ruled by the configuration file of each 3D object to be optimized. This part is not investigated in-depth as it is not our main focus.

Q5: Details of user study.

A: For the dimension of "disentanglement", we will present the shaded images of generated 3D objects and the corresponding diffuse map to the participants, and ask them whether the diffuse map contains weird and unexpected bright lights or shadows.

Q6: How well does this method generate implausible objects?

A: Thanks for your suggestion. Our method is capable of generating implausible objects like "a wool goblet" and "a golden ice cream". We update our manuscript and include these results in Appendix Figure 8 (supplementary). More cases will be included later.

Q7: The illumination used in every render should be included.

A: Thanks for your suggestion, and we will include the illumination in the revision.

Q8: Could the material MLP be used to generate different sets of BRDF parameters from different material models from the same latent space?

A: It is an interesting direction to explore in the future. It's possible to learn different latent spaces or a shared latent space for different material models, and subsequently use a set of MLPs to generate multiple sets of BRDF parameters.

Q9: Limitations.

A: The discussion of limitations is included in the supplementary.

Q1: Difference between Text-to-3D and Inverse Rendering

Thanks for recognizing that our work is the first to introduce the material prior to 3D AIGC, particularly text-to-3D. We argue that inverse rendering and text-to-3d are two conceptually different tasks. They face different challenges as theirs are respectively reconstruction and generation tasks. Specifically, inverse rendering aims at reconstructing the correct materials given single or multi-view images, where these multi-view images provide strong constraints to eliminate any ambiguity. On the contrary, our text-to-3D method aims at generating proper materials when generating a 3D object given only a text input. Such a text input only provides loose semantic constraints on the possible materials, leading to lots of ambiguity and a much larger set of valid materials.

Q2: The semantic-aware material regularization is not well evaluated.

A: In the last sentence of the method section, we state that our proposed regularization $\mathcal{L}_\text{mat}$ is more effective than the material smoothness regularization proposed in prior works (Zhang et al 2021, Munkberg et al, 2022). As requested, we include a comparison between different regularizations in Appendix Figure 7 (supplementary). More cases will be updated in a few days. From the case we can see, that text-to-3D requires not only two geometrically close points to have similar materials but also two semantically close points to have similar materials. Therefore previous material smoothness regularization is not sufficient. This is also an important challenge of text-to-3D that differentiates it from inverse rendering.

Q3: Under multi-light setup, why is the albedo still engtangled with the environment map?

A: This is exactly a specific example of challenges mentioned in Q1 that differentiate text-to-3D from inverse rendering. The task of text-to-3D only has a text input that provides loose semantic constraints to regularize generated materials via the SDS loss. The SDS essentially maximizes the similarity between the text input and images rendered from the target 3D content, and we empirically found some tiny bright lights and shadows in the rendered images can easily fool the SDS loss for a higher similarity. To summarize, although our material prior provides more information to help generate proper materials, and experiments demonstrate its effectiveness, the task of text-to-3D is very challenging and some ambiguities need further investigation to resolve.

Firstly I want to thank the authors for the reply. According to the reply, I raise a further question.

Can the author comment on the expected comparison results of the following methods:

1. generate material and geometry jointly, like the paper's method and the Fantasia3D work.
2. a 2-stage method that combines DreamFusion (or recent SOTA 3D generation work, e.g. Magic3D, DreamCraft3D, DreamGaussian) to some inverse rendering method (e.g. Nerfactor, TensoIR). In this method, in the first stage, we generate 3D objects aligned with the text, and in the second stage, we disentangle the material from the baked texture.

I agree that text-to-3d is a generation task. But I wonder if material estimation in the paper's task is also a generation task. If yes, what is the benefit of treating it as a generation task instead of a reconstruction task?

Clarification of the Task

It's worth noting that our task is conceptually different from material estimation in image generation. As stated in the introduction, our task is text-to-3D, which generates 3D objects given a text input. And, the goal of this paper is to introduce a material prior so that generated 3D objects have more realistic and coherent materials, which, as recognized by reviewer Kg7h, is the first work in this line of research.

Q1: Why not consider absorption or sub-surface scattering, other than reflection?

A: Being the first work that explores a material prior to text-to-3D, we prioritize tackling reflections first as it covers a large portion of possible object materials, and currently is more tractable. We thus believe starting from BRDF is a good choice and sufficient for us to verify the effectiveness of such a material prior. For absorption, sub-surface scattering, and other cases, we leave them as future work where we can explore various potential solutions to enhance the material prior. For example, we could leverage hybrid material MLPs to model absorption or sub-surface scattering as well.

Q2: The diffuse component of gold in Figure 1 is weird.

A: For the diffuse component of gold in Figure 1, we follow the conventions in UE4 (Karis et al 2013), where the diffuse term $\mathbf{k}_d$ can’t be applied to the rendering equation directly. As mentioned in Section 3.2, we compute the specular term for rendering with $\mathbf{k}_s = (1-m) \cdot 0.04 + m \cdot \mathbf{k}_d$, and the diffuse term for rendering with $(1-m) \mathbf{k}_d$. Hence, for metal material whose metalness $m$ is close to 1, the yellow components for gold materials are indeed from the specular reflection. We will clarify it in the revision.

Q3: The user study without any statistical tests.

A: We followed previous works in text-to-3D to present our user study. Thanks for the suggestion, we will try to provide the statistical tests in a few days.

Q4: Compare with traditional material editing methods in the CG community.

A: As discussed, we are the first work that explores material prior to text-to-3D. Currently, it is non-trivial to apply traditional CG methods to text-to-3D for material editing, which contains only a text input without any UV mapping. Nevertheless, integrating traditional CG methods into text-to-3D is definitely a promising direction we shall explore in the future.

Thank you again for your valuable feedback, which we already provided detailed responses to. We hope our responses address your concerns. As the deadline for discussion is approaching, we'd like to ask if you have further comments. Looking forward to your reply :)