Looks Great, Functions Better: Physics Compliance Text-to-3D Shape Generation

The paper proposes a method to generate 3D shapes with increased physical plausibility. The pipeline is first initialized with the output of a traditional 3D generative model. The resulting shape is then optimized to have improved physical plausibility. In the examples, the method demonstrates how holes can be filled and weak support structures can made to be stronger.

The paper introduces Fun3D, a method for generating 3D shapes from text that ensures visual appeal and physical functionality. It features a data-driven differentiable physics layer for optimizing 3D shapes according to solid mechanics principles, providing the practicality of generated shapes for real-world applications.

The paper proposes Fun3D, a physics-informed text-to-3D generation method aimed at producing physically plausible 3D shapes based on text prompts. Existing text-to-3D models primarily focus on visual and geometric accuracy but lack physical realism, which limits practical applications. Fun3D addresses this gap by integrating physics, specifically solid mechanics, into the generative process. It uses a two-stage framework: an initial 3D shape is generated via 3D diffusion models and then optimized through a differentiable physics layer. This layer utilizes a mix of geometry and physics constraints, leveraging finite element method (FEM) data to improve stability and load-bearing capacity. Experiments demonstrate that Fun3D yields more physically robust shapes compared to baseline models like Diffusion-SDF, making it suitable for engineering and other real-world applications.

The paper introduces a novel text-to-3d approach Fun3D that integrates the laws of physics into 3D shapes. By analyzing solid mechanics, it addresses the limitations of existing methods that produce impractical shapes. Fun3D utilizes 3D diffusion models and a differentiable physics layer to optimize shapes according to physical laws. The paper also demonstrates its high-quality generation results with physical functionality.

We greatly appreciate the suggestions and comments provided by the reviewers for this paper. However, the reviewers seem to have misunderstood the core motivation of our work, particularly regarding the integration of physical information within the generative model. They appear to view the incorporation of physics as a trivial or inconsequential task. To address these concerns, we believe it is essential to provide a thorough response that clarifies the underlying motivations and the significant contributions of our approach.

Our primary objective is to explore the application of generative methods in more practical and engineering-oriented domains. While generative modeling has made remarkable strides, most advancements remain within computer vision, emphasizing visually appealing 3D models. However, in engineering, aesthetics alone are insufficient; models must meet rigorous safety and durability standards, which cannot be verified visually. This motivated our integration of physical information into the 3D generation process. Our approach, focused on solid mechanics, is pioneering and distinct from contemporaneous methods like PhysGaussian and Atlas-3D, offering a unique engineering perspective.

Additionally, we have observed that many reviewers are interested in validating the generalization ability of our method and comparing it with numerous generative models that do not incorporate physical considerations. However, we believe that such endless comparisons are not particularly meaningful, as our work primarily addresses two key points: 1) how to embed physics into the 3D generation process, and 2) the differences between the 3D models generated with and without the incorporation of physics, and the resulting impact of this integration. Regarding the first point, we have provided sufficient physics-related formulas to explain how to implement this process. For the second point, we have already demonstrated results and discussed the impact of incorporating physics in three models—Diffusion SDF, Shap-E, and Zero123. Therefore, we do not believe that continuing to add generative models and comparing them is a productive direction for our work. Moreover, based on our review of the original text recommended by the reviewer, it seems they expect us to further compare SOTA methods, which still fall under data-driven approaches and do not incorporate physical information embedding. The key distinction between physics-based generation and traditional data-driven generation lies in their interaction with the environment, as discussed in Section 3.5. When incorporating physical information, it is essential to consider the physical interactions between the 3D model and the real-world environment. A significant challenge posed by physical considerations is that, depending on varying physical conditions, even the same shape can lead to entirely different optimal results. While geometric data can be effectively captured in the dataset, external physical environments and their corresponding variations cannot be fully represented in the data. Consequently, even SOTA models that lack physical considerations may struggle to adapt to real-world physical environments.

Finally, we have observed that the reviewer suggests incorporating physics beyond solid mechanics to demonstrate the generality of our approach. However, this presents significant challenges. The physical models used in different fields, particularly those governed by differential equations, have been developed over decades or even centuries. As a result, each set of differential equations requires targeted consideration based on the specific characteristics of the field, making it difficult—if not impossible—to design a 'one-size-fits-all' approach that can handle all physics-related tasks. Furthermore, we do not believe such a universal approach is necessary. For any specialized engineering domain, embedding the relevant physical equations into the generative model is both meaningful and valuable, as it ensures that the model addresses the unique requirements and complexities of that field. It is also important to note that even current SOTA methods, such as PhysGaussian and Atlas-3D, primarily focus on a limited range of general elastic-plastic or hyperelastic models to produce visually appealing animations and striking effects, often overlooking engineering considerations. Nevertheless, although the aspect of generation was not fully addressed, we believe that both these SOTA physics-based approaches and our work contribute significantly to the advancement of future research in physics-aware generative modeling.

In this paper, the authors have proposed a physics-aware 3D shape generation method named Fun3D. The presentation of the paper is good, and the differentiable physics layer makes the pipeline end-to-end. However, this paper still have some significant limitations. As pointed out by the reviewers, the physical considerations are still narrow, the experiments are not sufficient enough, and the contributions of the paper should be better summarized. Due to the weaknesses above, the AC recommends a decision of rejection of the paper.

Reject