MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers

We thank reviewer 3Tqd for the insightful comments and constructive feedback. We will add real-world application examples in the main text to highlight the significance of our research. Our responses to the specific questions are as follows:

1. Related Works

Thank you for the recommendation. We have added these related works to our updated paper in Page 5, line 234.

2. Exposition

We appreciate the suggestions and have made the recommended edits in the main text.

To clarify how "poorer topology quality affects geometric characteristics," this statement is intended to convey that topology quality also impacts shape quality. Reconstruction methods, such as Marching Cubes, inherently lack the ability to represent precise geometry (such as sharp angles at specific degrees), often relying on dense faces for approximation, which limits their ability to produce sharp geometric details.

3. Data and Code Availability (Responding to R4Q1)

Of course, we will release all our code and data upon paper acceptance.

4. Limitations and Future Work (Responding to R4Q2)

Thank you for the suggestion. We have expanded the discussion on limitations and future work in our updated paper (from Page 20, line 1068 to line 1106). Generating highly complex meshes with many surfaces remains challenging due to the high face count required. However, we are pleased to see recent advances that successfully scale up our method [1,2,3], enabling effective generation of such complex meshes.

Ensuring watertightness during generation can be achieved by incorporating a simple prompt into the training process. As approximately 50% of meshes in Objaverse are watertight, training a model capable of generating watertight meshes upon request is feasible.

5. Stability Clarification (Responding to R4Q3)

In this context, "stability" refers to the success rate during inference. After decades of development, reconstruction methods achieve near 100% success rates with reasonable inputs. However, as a generative approach, our method inevitably produces occasional failure cases.

6. Performance on Open Surfaces

For an example of our method’s performance on open surfaces, please see the second mesh in the first row of Figure 1, which depicts a mesh with open surfaces. Conceptually, our approach can handle both open and closed surfaces equally well, and open surfaces may be simpler for our method to generate due to the typically lower face count required.

References

1. Edgerunner: Auto-regressive Auto-encoder for Artistic Mesh Generation. Tang et al., 2024. https://arxiv.org/abs/2409.18114
2. Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale. Anonymous, 2024. https://openreview.net/forum?id=mhzDv7UAMu https://meshtron.github.io/index.html
3. Scaling Mesh Generation via Compressive Tokenization. Weng et al., 2024. https://arxiv.org/abs/2411.07025

We thank reviewer qFPE for the insightful comments and constructive feedback. Please see our response to the feedback below.

1. Concerns about the User Study (Responding to R3W1)

The majority of participants in our user study were researchers from the 3D field, with a smaller portion being industry practitioners. We appreciate the reviewer for raising this point and have added this clarification to our updated paper (Page 10, Line 503).

2. Concerns about Comparison with Mesh Generation Pipeline (Responding to R3W2)

Many thanks for your suggestion! Your understanding of our experimental procedure is completely accurate. We have also been exploring ways to make the comparison experiment fairer, and your suggestion has greatly helped to resolve this issue. The table below presents results from an additional experiment on ShapeNet with a fixed category, as you suggested. In this experiment, both MeshAnything and MeshGPT were trained and tested only on the "chair" category in ShapeNet.

It is worth noting that MeshGPT does not currently provide open-source code or data, so our experiments used a reproduced version of MeshGPT. To ensure a fair comparison, we reproduced MeshGPT with the exact same architecture and data as MeshAnything, differing only in the use of unconditional generation versus shape-conditioning.

We observed that the gap between MeshGPT and MeshAnything narrowed under this setting, though a noticeable difference remains. This result supports our argument in Section 3 that shape-conditioned mesh generation is considerably less challenging to learn than unconditional mesh generation.

The above experiment have been added to our new version (Page 19, Line 1015).

3. Questions about the Generation Procedure (Responding to R3W3)

All of our qualitative and quantitative experiments employed multinomial sampling, equivalent to a beam search with a beam size of 1, using top_k and top_p values set to 50 and 0.95, respectively. Please see the Figure 8 of our newly updated (Page 20, line 1026), we provide generation variants obtained using this sampling approach.

It is worth noting that generation using max logits tends to yield slightly higher success rates, making it preferable in certain cases.

4. Insufficient Qualitative Results (Responding to R3W4)

We thank the reviewer for this suggestion and have added qualitative comparisons with baselines, such as MeshGPT and PolyGen, in our new version (Page 20, Line 1062).

5. Citation Format (Responding to R3W5)

We apologize for the formatting oversight and thank the reviewer for bringing it to our attention. We have corrected the citation format in the main text to ensure consistency with the template instructions.

We hope our reply could address your questions. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper. We thank you again for your effort in reviewing our paper.

Best regards,

MeshAnything Authors

Thanks for asking. Those papers do not cite your MeshAnything work in an anonymous way. Since you mentioned that they are your follow-up works, you essentially reveal your identities. Namely, anyone who searches the word "MeshAnything" in those papers will know your identities. I hope this makes things clear.

We thank Reviewer qFPE for the valuable feedback.

1. The purpose of citing these papers is solely to demonstrate the scalability of our method. Apart from citing these papers as evidence, we have no better way to substantiate this point. In the rebuttal, we simply provided links to the arXiv papers and did not direct reviewers to read any author-identifying information. ICLR explicitly allows the citation of arXiv papers, which may naturally contain information about the authors of the submission. After thorough review, we found no ICLR guideline prohibiting the citation of papers that reference the submitted work. Therefore, we believe we have not violated the anonymity policy.

2. The ICLR 2019 Reviewer Guidelines (link) state (We failed to find more recent detailed guidelines regarding anonymity and hope the AC can provide clarification.):

   Anonymity. ICLR is double-blind, which means that authors are not aware of reviewer identities and reviewers are not aware of author identities. If you believe a paper contains an anonymity violation, contact your AC immediately.
   Anonymity violations are not considered as part of reviewing criteria, they are requirements for submission. Unless your AC decides that the paper does indeed violate anonymity, proceed to review it as normal.

   Based on this guideline, we kindly believe that whether we violated the anonymity policy should be determined by the AC, and a possible violation should not serve as a reason for Reviewer qFPE to lower our score. We have contacted the AC regarding this matter and respectfully request that Reviewer qFPE consider restoring the previous score of 6.

After another round of carefully reading of other reviews, I think the author's response to Reviewer tUZi violated the anonymity policy. Therefore, I cannot maintain my positive score.

Specifically, the following comments under 3. Scalability of the Method (Responding to R2W3) violate the anonymity policy as they effectively reveal the identities of the authors:

In fact, recent works [1,2,3] following our method have successfully scaled up.

1. Edgerunner: Auto-regressive Auto-encoder for Artistic Mesh Generation. Tang et al., 2024. https://arxiv.org/abs/2409.18114
2. Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale. Anonymous, 2024. https://openreview.net/forum?id=mhzDv7UAMu https://meshtron.github.io/index.html
3. Scaling Mesh Generation via Compressive Tokenization. Weng et al., 2024. https://arxiv.org/abs/2411.07025

We thank reviewer tUZi for the insightful comments and constructive feedback. Please see our response to the feedback below.

1. Whether This Work is a Generation Model (Responding to R2W1)

We appreciate the reviewer’s suggestion regarding the classification of our method as a generation model. While we acknowledge that our method shares some similarities with reconstruction approaches, we maintain that it fundamentally belongs in the generative model category.

The generative nature of our approach is primarily evident in its flexibility with topology. Please see the Figure 8 of our newly updated (Page 20, line 1026), by varying the random seed under the same point cloud condition, our model generates multiple meshes with diverse topologies—something reconstruction methods cannot achieve.

2. It is unfair to use meshes generated by Rodin as the shape conditioning for comparison with MeshGPT. (Responding to R2W2)

MeshGPT does not currently provide open-source code or data. To ensure a fair comparison, we reproduced MeshGPT with the exact same architecture and data as MeshAnything, differing only in the use of unconditional generation versus shape-conditioning. The purpose of this comparison is to contrast these two settings to support our claim in Section 3: shape-conditioned mesh generation significantly reduces the complexity of mesh generation by avoiding the need to learn a complex 3D shape distribution.

Our method’s substantial advantage over MeshGPT in this comparison comes from its avoidance of learning a complex 3D shape distribution, instead relying on shape information provided by methods such as Rodin. This distinction is a core contribution of our work and the motivation behind our comparison with MeshGPT. . Besides, we agree that MeshAnything can be viewed as a remeshing model, with its generative ability reflected in topology rather than shape.

3. Scalability of the Method (Responding to R2W3)

Our model generates meshes with a maximum of 800 faces due to computational constraints. However, as our method utilizes autoregressive transformer, it holds potential for scaling similar to large language models. Numerous techniques from the LLM domain help mitigate memory issues, avoiding the quadratic scaling of memory with mesh face numbers. In fact, recent works [1,2,3] following our method have successfully scaled up; for instance, Meshtron[2] now supports up to 64K faces.

4. Stringent Data Requirements (Responding to R2W4)

Our filtering process resulted in a dataset of 56K samples primarily due to our face-count limitation. In the Objaverse and ShapeNet datasets, approximately 64K triangle meshes have fewer than 800 faces, and we filtered out only about 10% of these to remove redundant or overly simplistic data. With recent advances, such as the extension of the maximum face count to 64K in Meshtron[2], over 80% of the data (about 70K meshes) in Objaverse and ShapeNet would become usable for our approach.

References

1. Edgerunner: Auto-regressive Auto-encoder for Artistic Mesh Generation. Tang et al., 2024. https://arxiv.org/abs/2409.18114
2. Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale. Anonymous, 2024. https://openreview.net/forum?id=mhzDv7UAMu https://meshtron.github.io/index.html
3. Scaling Mesh Generation via Compressive Tokenization. Weng et al., 2024. https://arxiv.org/abs/2411.07025

We hope our reply could address your questions. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper. We thank you again for your effort in reviewing our paper.

Best regards,

MeshAnything Authors

We thank reviewer 7tHF for the insightful comments and constructive feedback. Please see our response to the feedback below.

1. Scalability of the Method (Responding to R1W1, R1W3, and R1Q1)

Our model generates meshes with a maximum of 800 faces due to computational constraints. However, as our method utilizes autoregressive transformer, it holds potential for scaling similar to large language models. Numerous techniques from the LLM domain help mitigate memory issues, avoiding the quadratic scaling of memory with mesh face numbers. In fact, recent works [1,2,3] following our method have successfully scaled up; for instance, Meshtron[2] now supports up to 64K faces.

2. Comparison to Point Cloud Generation Methods (Responding to R1W4 and R1Q3)

We appreciate reviewer 7tHF’s suggestion and have added a comparison experiment. Since Point-E primarily generates point clouds from text, while our method generates Artist-Created Meshes aligned with the input point cloud, a direct comparison is challenging. When applying methods like Point-E in industry, generated point clouds often need to be converted to meshes, making these point cloud-to-mesh conversion methods more comparable to our approach.

Therefore, we selected Poisson surface reconstruction, one of the most widely used point cloud-to-mesh methods, to conduct the following experiment: we sampled a dense point cloud of 100K points on each ground truth mesh surface, reconstructed it using Poisson surface reconstruction, and then optimized the face count through Blender Remesh. The results are presented below.

From the above results, point cloud to mesh methods yield outcomes similar to those produced by Marching Cubes, making it challenging for them to achieve both efficiency and quality in mesh generation. We have included this experiment in the main text.

3. Importance of Artist-Created Meshes (Responding to R1Q2)

Compared to reconstruction-based meshes, Artist-Created Meshes (AMs) offer significant advantages in terms of efficiency and aesthetics.

First, converted meshes often contain orders of magnitude more faces than AMs, leading to inefficiencies in storage, rendering, and simulation. Additionally, their complex and chaotic topologies make post-processing and downstream tasks in the 3D pipeline more challenging, requiring extensive optimization by human artists. Finally, reconstruction-based methods struggle to capture sharp edges and flat surfaces, resulting in oversmoothing and bumpy artifacts, as shown in Fig. 2 of our main paper. These factors make reconstruction-based meshes less suitable for applications, such as 3D games, where efficient rendering and high visual quality are essential.

We kindly direct reviewer 7tHF to line 88 on page 2 of the main text for further details.

References

1. Edgerunner: Auto-regressive Auto-encoder for Artistic Mesh Generation. Tang et al., 2024. https://arxiv.org/abs/2409.18114
2. Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale. Anonymous, 2024. https://openreview.net/forum?id=mhzDv7UAMu https://meshtron.github.io/index.html
3. Scaling Mesh Generation via Compressive Tokenization. Weng et al., 2024. https://arxiv.org/abs/2411.07025

We hope our reply could address your questions. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper. We thank you again for your effort in reviewing our paper.

Best regards,

MeshAnything Authors

I thank authors for their clear response!

1. scalability of the method It does sound like 800 faces is probably insufficient for a lot of realistic applications. I think this is particularly making paper's claims about efficiency weaker. I am not sure if we can judge the scalability of this work based on scalability of derived works.

2. comparison to point cloud agreed, direct comparison is challenging.

3. importance of artist-created meshes Some of the arguments are valid - in particular about chaotic topologies. Some are very arguable - e.g. reconstruction-based method being unable to capture sharp details - this is really setup dependent. I guess the question is more about method being in any way specific to artist meshes - as paper writing suggests so, but in my humble opinion those are completely independent.

In summary, I still think the scalability of the method is a major issue. Additionally, the main message of the paper remains unclear to me - in particular limiting application of a seemingly more generic method to artist meshes only. Thus I am inclined to keep my original rating.

Dear AC,

Reviewer qFPE raised a concern about a potential anonymity violation in our rebuttal and lowered the score from 6 to 3 based on this. We respectfully request the AC to make a determination on this matter and suggest that ICLR consider providing more detailed guidelines regarding anonymity policies, specifically clarifying whether it is permissible to cite papers that reference the paper under review.

Thank you,

MeshAnything Authors