Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials

Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.

---

W1. Importance of Addressed Task for Material Painting

While the derendering task mentioned by the reviewer is important, we believe the task of adding materials to albedo-only 3D objects is also essential for reducing the extensive time 3D artists spend creating realistic texture maps. Although graphics tools like ShaderMap integrate this functionality, they are limited to producing simple lighting effects. Our method fully explores the potential of MLLMs to accurately capture complex materials and provide a solution for realistic, part-level material generation.

Additionally, we believe that derendering and material restoration are not mutually exclusive fields but rather complement each other. Research in the latter area can serve as a tool for automating the generation of 3D material data and help researchers gain a deeper understanding of material properties and their realistic effects in the physical world.

---

W2. Discussion of Material Grouping Method

Thank you for pointing out. We acknowledge that, in theory, the assumption that parts with the same base color have the same material can be limiting. However, in our experiments, we found that in most cases in the dataset, semantic patches obtained through SemanticSAM tend to have similar colors when they represent the same material. Based on this observation, we implemented a merging process, carefully controlling the merging threshold based on experimental results. This approach reduces the number of GPT-4V queries without compromising effectiveness. We will include more details about this in the final version.

---

W3. Unleashing GPT-4V's Ability for Material Painting Beyond VQA

Our core contribution is not in applying multimodal methods for VQA but in effectively leveraging and fully exploring the potential of MLLM to achieve an innovative and practical application: material painting. GPT-4V, even with well-designed VQA tricks, cannot perform material painting on objects because it is confined to the 2D space. By designing a comprehensive pipeline, we aim to unlock MLLM's ability to understand the 3D physical world and generate realistic material representations.

Our pipeline include an advanced segmentation method in the UV space of materials, a rich set of visual and text cues for faster material retrieval, and an innovative algorithm that generates SVBRDF material maps using a region-to-pixel approach. This integration bridges 2D MLLM models and the 3D physical world, empowering large language models with the capability to paint materials to objects at the part level.

---

W4. Key Implementation Details

You are correct that Appendix B.2 contains the key implementation details of our method. In Section 3.2.3, we describe the essential aspects of SVBRDF map generation in text form and refer to figures in the appendix to aid understanding. However, we acknowledge that the main text contains limited illustrations for this section. We will enhance the discussion and include more illustrative figures in the final version to improve clarity.

---

W5. Material Effect of Presented Results

Materials are inherently complex physical properties, and differences can be seen by zooming in on different regions, such as the globe, bathtub, and shield in Figure 5. I would like to clarify that the material effect is not solely due to higher metallicity producing a brighter lighting effect. Instead, it results from the interaction of multiple PBR maps: increased metallicity results in dampening of the base albedo and increased surface shine, increased roughness reduces specular highlights, and displacement and height control fine surface details. These values are assigned based on the albedo, type, and local region of the object, so not all parts have the same higher metallic settings. (More infomation in Figure 16)

Regarding the video demo, we're sorry for the quality of the rendered scene at the end frame. Due to significant memory requirements for rendering scenes, this picture may not fully reflect the clarity and precision of the objects. We recommend referring to the clearer examples in the main part of the video and paper.

---

W6. The Clarification on Related Works

We have cited and discussed several texture generation works in the Material Capture and Generation subsection in Sec 2. Related Work. We introduce recent works on texture generation, such as Paint-it and Collaborative, which generate various material texture maps based on shape. We also reference earlier works like Fantasia3D and Matlaber. We discuss the approaches and limitations of these methods, and we will update the final version with more related works in this area.

---

Q1. The significance of identifying shadows and lighting

Sorry for any confusion. We would like to clarify that besides retrieving materials for albedo-only 3D objects, our approach involves describing rendered thumbnails of real materials in the Material library to construct the dataset annotations. These thumbnails may include lighting and shadows. The task in our whole setting requires two capabilities of recognizing materials: 1. Accurately describing real materials under ambient lighting, and 2. Inferring local materials from distorted albedo. We will add more explanations about this point in the final version.

---

Q2. Method of Key Diffuse and Indexing

The key diffuse refers to the diffuse map of the material retrieved by the MLLM. This can be understood through the multiple arrows stemming from index in Figure 11. After obtaining the nearest neighbor coordinates in step 2), we use the coordinates to index the values of the same position in other material maps paired with the current diffuse. We will draw more illustration(like serial number) on our figure to make it more clear.

---

Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.

---

W1. A clarification of novelty of Make-it-Real

Thank you for the comments. Our key innovation lies not in the MLLM model technology itself, but in how we effectively leverage and fully explore the potential of MLLM to achieve innovative and practical applications. MLLM alone cannot perform material painting on objects, so we develop a practical and comprehensive pipeline to unlock this capability.

Our core contributions include an advanced segmentation method in the UV space of materials, a rich set of visual and text cues for faster material retrieval, and an innovative algorithm that generates SVBRDF material maps using a region-to-pixel approach. This integration bridges 2D MLLM models and the 3D physical world, empowering large language models with the capability to accurately assign materials to objects. The generated materials are compatible with various rendering engines, enhancing the versatility and utility of our approach in creating realistic 3D assets.

---

W2. About quantitative evaluation of painting materials

Thank you for your understanding regarding this novel application. Previous work primarily focused on 3D object generation without explicitly separating base color from different PBR properties. However, there is now a growing interest in enhancing the physical material properties of 3D objects, yet evaluation methods remain limited.

Obtaining ground truth for material properties is challenging mainly due to the ambiguity in material ground truth for certain objects; for example, a white albedo mug could be made of metal or ceramic, leading to different material maps for the same albedo input. Given this ambiguity, we use the evaluation methods described in GPT4Eval[1] and conduct user studies to directly assess improvements in visual effects. This approach allows for a clear and direct comparison of visual quality. We hope our exploration of material properties will inspire further interest and lead to the development of more comprehensive evaluation benchmarks.

[1] GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation

---

W3. Addressing Baked-in Shadows and Lighting Effects

This issue is less pronounced in artist-created models, where albedo maps have less shadows and lighting effects. Our tests show that the albedo obtained from inverse rendering methods[1] closely matches the original artist-created albedo maps from the Objaverse dataset, as illustrated in the first row of Figure C3. For generative models, baked-in shading effects are more pronounced. Our method addresses this by integrating mature derendering algorithms, specifically the Intrinsic Anything[1], into our pipeline with minimal complexity. This integration derenders from four viewpoints and back-projects to obtain a better albedo map, then applies our Make-it-Real method for material painting. As shown in Figure C3's last row, this approach reduces lighting noise and supports a wider range of inputs. We will include more details about the derendering of inputs in the final version.

[1] IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination.

Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.

---

W1. Realism and Application of Generated PBR Maps

Thank you for pointing this out. We acknowledge that the generated PBR maps are not true representations of physical maps, and we will clarify this in the final version paper. Nevertheless, our algorithm generates PBR maps that closely approximate the visual characteristics of true PBR properties. As demonstrated in Figure 5, our results significantly enhance the realism of 3D objects. Additionally, the analysis in the appendix (Figure 16) further examines the effects of each individual map, which mimic real textures, such as metallic properties (e.g., the dampening of the base albedo and increased surface shine on parts with high metallicity like scales and axes), roughness (with low roughness objects showing specular highlights), and displacement (fine bump effects on stone horses and bathtubs). In all cases, we produce visually comparable effects to true PBR maps.

---

W2. Impact of Albedo Map Quality on Performance

We agree that the final results can be affected by the quality and detail of the initial albedo maps. However, our method is designed to enhance the physical appearance even when the input albedo map varies in detail.

On one hand, for objects that require meticulous craftsmanship and significant attention to detail, such as the truck and shield in Figure 5, our method precisely segments and assigns appropriate materials, significantly enhancing realism and producing high-quality objects with highly refined materials.

On the other hand, for objects with relatively simple albedo maps that can be created quickly, like the saxophone shown in Figure C1, our method substantially improves the object's appearance under environmental lighting conditions, increasing realism and achieving a qualitative improvement.

---

Q1. The clarification on PBR texture maps

We would like to provide some clarification about this. The PBR texture maps in the material library are sourced from MatSynth. This dataset compiles high-quality PBR materials from publicly available resources like AmbientCG, which are widely used by artists for creative work. The materials are typically created using methods such as photogrammetry, scanning, and high-precision software, ensuring both visual fidelity and accurate physical properties.

---

Q2. Performance with Fragmented UV Mapping

Thanks for your insightful questions. Overly fragmented UV mapping degrades the performance of our method. As shown in the second column of Figure C2, UV mappings with excessive fragmentation and color entanglement can cause the 2D segmented images to mix with other regions when reprojected into the UV space, leading to material blending issues. However, our method shows good results with original artist-created UV mappings and Blender's built-in mapping methods in Figure C2, such as smart, sphere, and unwrap. This indicates that our method still demonstrates good robustness with many mapping techniques.

In practice, we observe that most objects in Objaverse have UV mappings with good properties, meaning they are not excessively fragmented into small pieces. Additionally, we can control the UV mapping process: For some low-quality UV maps and generative objects that originally lack UV maps, we can re-unwrap them to achieve higher quality.

---

L1. Advantages of the Proposed Method Over Offline Tools

Firstly, I would like to acknowledge that ShaderMap is a valuable tool for material creation, and it's great to see a thoughtful comparison of different approaches.

Our method offers several distinct advantages:

1. Focus on 3D Models: Our approach excels in directly generating complete and fine-grained PBR maps for 3D models, offering more comprehensive integration with 3D assets compared to traditional tools. Tools like ShaderMap primarily focus on estimating higher-quality PBR texture maps from low-quality material images, such as photographs or instrument-captured material images, which may include lighting or shadows. These inputs are inherently material images and often do not focus on rendered views of 3D objects. In contrast, our method directly enhances 3D models, providing a more robust solution for integrating high-quality materials.
2. Complexity and Usability: Our method stands out for its ease of use and ability to produce semantically meaningful material assignments without requiring professional knowledge of 3D software. While ShaderMap can generate basic PBR maps (such as normal and specular maps) from albedo maps, this functionality currently yields relatively simple material effects. The material effects may appear to have lighting effects but do not necessarily match true semantic meanings. Since the entire albedo map is used as input, the software lacks fine-tuned regional specificity. Further manual adjustments are typically needed for local map values, requiring user interaction and expertise in 3D software, as well as additional time to achieve realistic results.

Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.

---

W1. Differences between Make-it-Real and MaPa and the superiority of our method.

• MaPa: Introduces a text-driven segment-based procedural material graph representation. It uses a pre-trained 2D diffusion model as an intermediary bridge to connect text and material graphs. The method includes segment-controlled image generation and material graph optimization. It simply use CLIP for material retrieval.
• Make-it-Real: Utilizes Multimodal Large Language Models (MLLMs), specifically GPT-4V, to recognize and apply real-world materials to 3D objects. The method involves texture segmentation, material matching, and SVBRDF map generation. It creates a comprehensive material library with detailed descriptions, allowing MLLMs to search and assign materials automatically.

The core difference lies in how to perform the material retrieval. MaPa [1] still uses CLIP for the material retrieval process, while we rely directly on MLLM. We believe this core difference significantly improves the retrieval effectiveness for the following reasons:

1. CLIP can only focus on one area at a time. When generating images that need retrieval, other parts need to be masked while retaining the local part. This causes the retrieval process to lose the global semantic information of the object. MLLM, on the other hand, can retain the global semantic information by only circling the specified area in the pixels [2].
2. We creatively propose registering a hierarchical dictionary in MLLM by providing descriptions and images of material spheres, allowing MLLM to master a richer material library (The library is much larger and more practical with 1394 materials while MaPa only have 118).
3. With the rapid development of MLLM, transitioning this process from CLIP retrieval to a multi-step hierarchical and logical inference decision process relying on LLM is more promising with better explainability.

To further validate the above points, we supplement the experiment detailed in W2 testing using MLLM for retrieval compared to using CLIP.

There are also other differences that improving the performance compared to MaPa:

1. We propose a new method to achieve 3D segmentation leveraging Semantic-SAM and MLLM, which is more general than MaPa that adopt an out-of-date mesh segmentation method [3].

---

W2. Experiments on the superiority of using MLLM for retrieval.

we conducted additional experiments comparing GPT-4V (our method) and CLIP (MaPa method) to verify the superiority of using MLLM for retrieval.

Specifically, we use the CLIP model as a baseline for comparison, which is the method adopted in PSDR-Scene [4] and MaPa [5]. We selected 100 objects from Objaverse with high object quality and annotated each object with different materials following the material classification of [4], resulting in 13 coarse types (Objaverse-C) and 80 subtypes (Objaverse-F). The retrieval results are reported in the table below. The results fully demonstrate the effectiveness of using MLLM for retrieval. We discover CLIP often fails in cases with metallic materials, largely due to the difficulty of extracting material regions without distortion. Our pipeline, using GPT-4V, queries four viewpoints for global shape and object semantics, can leveraging its strong open world prior knowledge to achieve promising results. Additionally, we observed that CLIP’s accuracy significantly drops when dealing with fine-grained categories compared to coarse categories, while GPT-4V maintains a high retrieval accuracy. This result also indicates that MLLM is more practical, as such systems often need to handle thousands of materials in a much finer granularity in real-world applications.

---

[1] Zhang S, Peng S, Xu T, et al. MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

[2] Yang J, Zhang H, Li F, et al. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

[3] Sagi Katz and Ayell, et al. Hierarchical mesh decomposition using fuzzy clustering and cuts

[4] Yan K, Luan F, Hašan M, et al. Psdr-room: Single photo to scene using differentiable rendering

[5] Vecchio G, Deschaintre V. MatSynth: A Modern PBR Materials Dataset

Thanks for your detailed review and insightful comments! We will address your questions below and in the revised paper.

---

W1: More Comparisons with Artist-Created Materials:

Thank you for the insightful suggestion! We conducted an additional user study comparing materials generated by our Make-it-Real method to those created by artists. We filtered the Objaverse dataset for objects with material maps and randomly sampled 200 objects for the experiment.

Our results are shown in the following table. The second row highlights our method's superiority over albedo-only objects, with 73.1% of users recognizing enhanced material effects, reinforcing the paper's conclusions. The first row indicates that users rated the material effects generated by our method as superior to or on par with artist-created materials for 61.6% of the objects. This is a highly promising result.

We examined these cases closely, as visualized in Figure C1. For objects like pistol, wooden chair, speakers, and saxophone, Make-it-Real showed strong similarity to artist-made materials, maintaining consistent metallicity, roughness, highlights, and coloration. In cases such as shield, boots, landline phone, and oil drum, our method produced materials that, while different, are realistic and sometimes visually superior. Our pipeline approaches the level of refinement seen in some artist-created materials, which often require significant time and effort, and clearly outperforms more basic and crude artist-generated materials. This highlights its effectiveness and potential for practical applications.

---

W2: Addressing Baked-in Shadows and Lighting Effects

This issue is less pronounced in artist-created models, where albedo maps have less shadows and lighting effects. Our tests show that the albedo obtained from inverse rendering methods[1] closely matches the original artist-created albedo maps from the Objaverse dataset, as illustrated in the first row of Figure C3. For generative models, baked-in shading effects are more pronounced. Our method addresses this by integrating mature derendering algorithms, specifically the Intrinsic Anything[1], into our pipeline with minimal complexity. This integration derenders from four viewpoints and back-projects to obtain a better albedo map, then applies our Make-it-Real method for material painting. As shown in Figure C3's last row, this approach reduces lighting noise and supports a wider range of inputs. We will include more details about the derendering of inputs in the final version.

[1] IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination.

---

W3: Results of Performance Metrics

Thanks for your valuable feedback. We have added tests for average performance metrics, as shown in the table below.

Additionally, we conducted further tests to measure the time required for each of the three stages drawn in our pipeline: (a) Rendering and Segmentation, (b) Material Retrieval, and (c) Material Generation.

---

Q1. The clarification of viewpoint selection

We explain some of the detalis in Sec 3.2.1, and here we provide further details. Initially, we perform rasterization for rendering, and the selection of viewpoints follows the approach used in TEXTure. We render a total of ten viewpoints: eight at 45-degree intervals around the horizontal axis and two additional views, one top-down and one bottom-up, forming our initial set of rendered images.

Among these rendered images, we select the viewpoint with the largest foreground area (the projected area in rasterization) as our primary viewpoint because it is more likely to contain more material information. This viewpoint is then used for subsequent material retrieval and map generation. We further refine the material generation using our proposed material inpainting method in Sec 3.2.3.

---

Q2. Discussion of UV mapping of input mesh

We conducted experiments to evaluate the impact of UV mapping on our method. As shown in Figure C2, except for the second column where the lighting UV mapping method results in fragmented and interwoven colors, other unwrapping methods such as smart, sphere, and unwrap produce better results. Our method does not perform well with overly fragmented UV maps but demonstrates good robustness with other mapping methods.

In practice, we have observed that most objects in Objaverse have UV mappings with good properties, meaning they are not overly fragmented into small pieces. Additionally, for some low-quality UV maps and generative objects that originally lack UV maps, we can control UV mapping, (re-)unwrapping them to achieve higher quality.

---

Q3. Categories construction in material library

These materials originally come from AmbientCG, an open online repository, where each material has a fine-grained category name, such as "Planks". We follow the classification of material categories in MatSynth, resulting in 13 major categories and 80 subcategories. This forms our two-level hierarchical tree structure, which corresponds to our hierarchical prompt retrieval.