HierT2S: Enhancing Part-Level Text-to-Shape Generation via Hierarchical Structure Modeling

1. The model sometimes produces minor artifacts, particularly in transitions between different parts of the generated shapes, which can slightly detract from overall visual coherence. While the hierarchical tree model effectively segments text into part-level components, there is limited discussion on the constraints and challenges associated with this representation. For instance, it is unclear how well this approach scales when dealing with more complex or ambiguous prompts, such as “a table with intricate detailing on the legs and an engraved top,” where elements might have multiple substructures or unclear hierarchical relationships.

2. The evaluation section lacks a detailed comparison with other methods specifically designed for part-level or fine-grained 3D shape generation. Although the model outperforms existing baselines on metrics like IoU and EMD, it is unclear how HierT2S compares with recent models that incorporate part-based structures or modular designs. Such comparisons would be particularly valuable given the paper’s focus on part-level control.

3. The paper currently lacks a comparison of training time with other state-of-the-art methods, making it difficult to evaluate the computational efficiency of this approach. Without insights into the time and resources required for training, readers may find it challenging to assess whether this method is feasible for deployment in environments with limited computational capacity.

Although the problem this paper focuses on is valuable, this paper only proposes a simple method from the technical view and the innovation is limited. Here I highlight my major concerns.

1. The novelty of the proposed method is limited, that is the combination of existing methods. Firstly, the proposed method is similar to SDFusion except for the part of text feature learning. Second, when learning text features, the proposed segmentation cluster matrix does not substantially improve the Tree-transformer and the proposed relation graph module is just the use of the attention mechanism.

2. The evaluation of the effectiveness of hierarchical feature learning is insufficient. Though this paper compared with existing text-to-shape methods, such as SDFusion and 3DQD, which do not utilize hierarchical information, this paper didn’t cite and discuss existing hierarchical-based methods and didn’t provide a comparison with them. Comparing these methods can justify whether the proposed hierarchical feature learning is effective. These methods include [1, 2, 3 ....]. [1] ShapeScaffolder: Structure-Aware 3D Shape Generation from Text [2] ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model [3] HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generation

3. About long text descriptions. In the abstract, this paper points out that ignoring hierarchical modeling affects long text processing, but the experiment results on long text are missing in the experimental section.

4. About the visualization of the hierarchical tree. I recommend giving visual examples of hierarchical trees built, such as for a text, drawing its hierarchical tree. Otherwise, it is now difficult to know what the hierarchical structure of the learned text tree is.

5. About the definition of metrics. On page 8, the metrics, such as TMD and EMD, are missing definitions or references.

• While combining existing methods can sometimes lead to innovative solutions, the paper lacks a clear rationale for why these specific methods were chosen and how their integration results in a synergistic effect that advances the field. For instance, the text structures in ShapeNet dataset are typically simple, which can be regarded as the format of “{Adj.} {Noun.} … {Adj.} {Noun.}”. In this case, will the extracted text structures have big variances? Authors need to provide quantitative and qualitative analysis on the extracted trees. Moreover, lots of pretrained language models already can understand the language structures. Authors need to evaluate the performance comparison between using the latest pretrained language models and the proposed framework.

• In Relation Graph Module, why do you use BERT instead of other advanced pretrained langugae models as prompt features?

• It would be beneficial to assess whether the integration of the text parsing method, graph relation extraction, and 3D U-Net results in any novel architectural or methodological advancements. If the combination does not introduce new mechanisms or processes, it could be perceived as a straightforward aggregation of existing methods rather than a meaningful contribution. I suggest the authors explore any novel interactions or dependencies between the components that could lead to unique insights or improved performance metrics, and highlight these in the results.

• Authors only conduct experiments on Chairs and Tables of ShapeNet dataset. They generate 3D geometry only, without texture. This is too weak, given there already have works that can generate textured 3D objects in ShapeNet now, such as TAPS3D. In Objaverse dataset, there are lots of methods, such as Wonder3D, Unique3D and etc., that can generate much more complex 3D meshes. Authors need to provide comparison with these methods. Here are the suggestions:

1. Broaden the Dataset Scope: Encourage the authors to evaluate their method on the Objaverse dataset, which includes a diverse range of 3D objects.
2. Texture Generation: Suggest that the authors conduct experiments focusing on texture generation.
3. Computational Efficiency: Time taken for generation and resource utilization.

• The results shown in Figure 1 & 4 are low-quality as the geometry details are fuzzy. Even consider geometry solely without considering many other frameworks do produce colors, the results are not good enough.
• Some of the design choice is not good enough. For example, the sampling point clouds only have 2,048 points which could be the reason the results cannot recover the back strip for the chair. Can the author try higher point samplings, like 8,192?
• I am confused about how the proposed Hierarchical Structure works. Does the learned full node connection really make sense? Can the author provide more in-depth analysis here instead of just final results along with some inputs. I failed to understand how the parts affect the final generation performance. Can we visualize the learned node connections?
• The current paper only performed on very old data Text2Shape as well as ShapeNet. However, we have much more 3D assets such as ABO and Objaverse. Results demonstrate on those newer datasets could be appealing. The current tested text-prompt is really simple.

1. Although I said I like the idea, much of the claimed novelty in HierT2S, specifically the hierarchical modeling of parts in 3D objects, is conceptually similar to prior works like StructureNet, PartNet, and Hierarchical Surface Prediction. This reduces the originality of HierT2S, positioning it as an incremental advance that mainly adds diffusion-based modeling to a well-explored framework. I expect to see different designs for the diffusion process that fit more into the idea of the hierarchical graph instead of just treating those decomposed information as conditions of different scales.

2. The model's hierarchical parsing approach assumes well-organized, descriptive text inputs, which may limit its robustness with less structured or ambiguous text. This makes it potentially less adaptable to a variety of real-world input scenarios compared to more flexible models that handle less structured data.

3. Evaluated solely on the Text2Shape dataset, HierT2S may lack robustness across diverse or complex categories, especially as hierarchical part relationships become more intricate. However, I understand that the paired text-3D dataset with detailed description is very limited. In that case, I suggest the authors incorporate the conditions of different models in this framework (e.g., combining shape completion, image-to-3D, and text-to-3D) to show more capability of this work. Current experiments and demonstrations are too narrow.

The method does not seem to produce part-disentangled edits -- For instance, in Figure 6, adding armrest to the chair changes the thickness of the legs, the width and the shape of the back as well.

Some prior work in this area have shown that part-disentanglement can naturally emerge from training with natural language (https://arxiv.org/abs/2112.06390 and https://arxiv.org/abs/2212.05011). Are any of the regularizations/tricks applied in these papers applicable to improving the part-disentanglement of the edits in your method?