Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale

Dear reviewer QYFu,

First of all, we would like to thank you very much for your review. We sincerely appreciate the time you spent in evaluating our work, and very much appreciate your comments.

Here we will answer to your questions, comments and concerns:

The meaning of the intermediate representation is not well presented. It can be inferred from Figure 4, but I the authors should to discuss it in the text for the sake of the reader - please expand Section 3.1.

We acknowledge that the presentation of the Hourglass architecture could be improved. To address this limitation, we have improved the presentation of the Hourglass architecture in both the Introduction, Sec. 3.1 and Fig. 4 to better highlight its properties and the role of the intermediate representation. In addition, we have included an animation on our project website, which illustrates its inner workings in greater detail. This website will remain active and will be deanonymized upon acceptance.

it is not clear how the sliding window is used during inference. For example, how is the sliding window decided? And given that the sequence is truncated during training, how is a larger mesh generated? Please, provide more details on the sliding window inference process in Section 3.2.

We have enhanced Sec. 3.2 and Sec. 4.2, adding details to clarify the sliding window inference process. In summary, larger meshes are generated by employing a sliding window (or rolling KV-cache) with a size equal to the training chunk. This ensures consistency between training and inference, eliminating discrepancies caused by sequence truncation. Fig. 7, Sec. 4.2, and Fig. 13 provide further explanation and visualizations to illustrate the importance of this approach. In addition, Appx. A provides additional details regarding how similar approaches have been used by other methods to address similar problems.

the authors claim the length of the mesh sequence can be 64K in faces, L158-159. However, I cannot find such examples (even in the supplementary material). Is this a theoretical limit or do authors have examples and did not include them in the material?

We have now provided access to several examples on our project website. Note that 64k refers to the upper bound on generations, which is determined by the maximum sequence length used during training. On the project website, you can find examples with over 56K faces, showcasing the model's capability near this limit.

the authors point out "MESHTRON does not work from time to time", would you please elaborate? This is not an appropriate sentence for a scientific paper.

We apologize for the unscientific phrasing in the original text. We have now replaced it with a precise description of the observed failure modes. Specifically, occasional inference failures manifest as missing parts, holes, distorted (exploding) meshes, or failure to properly terminate the sequence. These issues are rare but highlight the need for further robustness improvements.

how is the global conditioning obtained? Is the input point cloud feed to the model and then the sequence used for cross attention?

That is correct. The input point cloud is fed to the model and used to generate global embeddings, which are then incorporated through cross-attention layers in the sequence. We have updated Fig. 4 to provide a more comprehensive view of Meshtron's architecture and detailed how the conditioning is achieved both at the beggining of Sec. 4, and in the Appendix.

would it be possible to condition from an image latent representation (say encoded by DinoViT) and project it onto the latent space of MESHTRON? (it is fine if it is future work, but worth mentioning in the paper)

This is an excellent point. We believe it is indeed possible, given the flexibility of Meshtron's pipeline. The cross-attention architecture, widely used in multimodal LLM research, allows Meshtron to be conditioned on any input modality that can be encoded into one or more embedding vectors. For instance, image latents encoded by DinoViT could be added as additional conditioning during generation. We have highlighted this possibility in the limitations and future work section (Sec. 5).

a comparison with MeshGPT would be appreciated even if qualitative.

We have included a visual comparison in Fig. 14. The generations observed here are obtained from the official inference code hosted here: https://github.com/audi/MeshGPT . First of all, it is important to highlight that MeshGPT only offers fine-tuned models only able to generate either chairs or tables. Additionally, the quality of the generations is lower than MeshAnything, MeshAnythingV2 and Meshtron.

We hope that these responses clarify your questions and concerns. Please let us know if you have any follow-up / additional questions.

Best regards,

The Authors

Dear reviewer TBNL,

First of all, we would like to thank you very much for your review. We sincerely appreciate the time you spent in evaluating our work, and very much appreciate your comments.

Here we will answer to your questions, comments and concerns:

While the Hourglass Transformer is an effective structure, linking it directly to the perplexity of the last vertex tokens feels somewhat tenuous.

We acknowledge that we did adequately explain this aspect in the paper. Hourglass Transformers have two key properties relevant to Meshtron. The first property is that they process outputs in a hierarchical manner. This hierarchy allows the model to effectively process the input at vertex and face levels of abstraction.

The second property, which contributes to their efficiency but was not properly outlined in the paper, is that Hourglass Transformers dynamically allocate compute to tokens based on their position within the sequence. As a result, some tokens receive more compute than others. This is illustrated in the animation provided in our project website. Specifically, based on the selected shortening --and upsampling-- factors and depending on their position, some tokens will skip the inner Transformer stacks of the Hourglass architecture, whereas others will be processed by the whole architecture. This targeted allocation of compute helps the model focus resources on tokens that require greater attention.

To make this clearer in the paper, we have: (i) complemented Fig. 5 with an additional per token compute graph that illustrates the per-token compute used for different tokens, and (ii) improved the explanations provided in the Introduction as well as Sec. 3.1. In addition, we will keep the animation on the project website, which we will de-anonymize upon acceptance.

We hope this clarifies this connection and are happy to complement this with any other observations the reviewer might have.

In the experiments, the authors used high-quality point clouds as input. However, in practical applications, point clouds often contain noise. This paper does not discuss or experiment on this issue, which inevitably raises concerns about its practicality.

This is a good observation. We originally provide some brief information regarding this in the Appendix, but it was not mentioned in the main text. During training, we augment the point cloud positions and normals with gaussian noise to simulate noisy real-world conditions. In addition, we make sure that point clouds are never sampled from internal structures --which may be present in artist meshes, but are unlikely to appear in scanned or automatically generated meshes-- to aid generalizability to non-artist like meshes. This treatment is crucial, and it is in fact, thanks to this that Meshtron is also able to receive point-clouds from scans as well as iso-surfacing methods (which are often noise and imperfect) and successfully map them to artist-quality meshes.

To address your concern, we have updated Sec. 4 to include details regarding this treatment and their role in enhancing robustness. Additionally, we have added Fig. 11, which illustrates how adjusting input noise allows Meshtron to balance creativity and faithfulness, showcasing its flexibility and applicability in scenarios with varying levels of input quality.

Dear reviewer rUX9,

First of all, we would like to thank you very much for your review. We sincerely appreciate the time you spent in evaluating our work, and very much appreciate your comments.

Here we will answer to your questions, comments and concerns:

The main weakness is the claim that previous work "recent attempts at generating artist-like meshes are limited to 1.6K faces and heavy discretization of vertex coordinates". I want to refer the authors to works such as [1, 2] that show mesh reconstruction with more that 1.6K faces. Both of the mentioned works [1, 2] show high-quality reconstructions and are not compared by the authors.

We acknowledge that we did not sufficiently clarify the problem we are addressing and how other methods, despite generating meshes with large face counts, fall short of solving it. To address this, we have revised the introduction and included comparisons with the iso-surfacing methods DMTet [2] and FlexiCubes [1] in Fig. 2.

In a nutshell, the key limitation of iso-surfacing methods is that they often exhibit poor topology, characterized by overly dense tesselation, over-smoothing, and bumpy artifacts, which results in a significant quality gap between AI-generated meshes and those crafted by artists (see Fig. 2). Consequently, the topology of these methods is insufficient for practical downstream applications.

To overcome this limitations, methods such as MeshGPT and MeshAnythingV1/V2 attempt to model 3D assets directly as meshes. While the topology of these methods is much closer to those created by artists, these methods are heavily constrained by the computational cost of processing the (very) long sequences required for high face counts. Meshtron provides an scalable alternative to generate artist-like meshes composed of many faces --40x more than existing works--, comparable to the face count of iso-surfacing methods (DMTet, FlexiCubes), but providing high-quality topology that resembles that of meshs crafted by artsits. Comparisons and additional generations have been added both to the paper as well as our project website.

How much does the number of points in the point cloud impact the final mesh generation process

We have complemented our manuscript with an additional analysis of this aspect. This effect is shown in Fig 15 and Appx. C5. In a nutshell, we observe that Meshtron high-quality generations even when the number of points deviates from densities observed during training. As one could expect, higher point densities provide more detailed geometric information, allowing Meshtron to generate meshes with finer detail, particularly for complex geometries, even when the specific point cloud density used exceeds those observed during training. Conversely, lower point densities may limit the detail level but still result in coherent and well-structured outputs.

How does the model handle partial point clouds?

We did not consider this application in this project. However, we envision that Meshtron could either be used in tandem with point cloud completion methods to progressively generate high-quality meshes from incomplete point clouds, or be trained with point cloud corruptions to enable the model itself to handle partial point clouds. Both approaches could be promising directions for future work.

Above all, I would like the point regarding the comparision with [1, 2] to be addressed and would change my rating if it was a misunderstanding on my side.

We hope that our previous response, along with the revisions and additional comparisons provided, have addressed this concern. Please do not hesitate to let us know if further clarifications or additional information would be helpful.

We hope that these responses clarify your questions and concerns. Please let us know if you have any follow-up / additional questions.

Best regards,

The Authors

Dear reviewer hX86,

First of all, we would like to thank you very much for your review. We sincerely appreciate the time you spent in evaluating our work, and very much appreciate your comments.

Here we will answer to your questions, comments and concerns:

First, I find it unclear why the Hourglass Transformer is considered the appropriate solution to address the challenge of generating the latter tokens of a triangle.

We acknowledge that we did adequately explain this aspect in the paper. Hourglass Transformers have two key properties relevant to Meshtron. The first property is that they process outputs in a hierarchical manner. This hierarchy allows the model to effectively process the input at vertex and face levels of abstraction.

The second property, which contributes to their efficiency but was not properly outlined in the paper, is that Hourglass Transformers dynamically allocate compute to tokens based on their position within the sequence. As a result, some tokens receive more compute than others. This is illustrated in the animation provided in our project website. Specifically, based on the selected shortening --and upsampling-- factors and depending on their position, some tokens will skip the inner Transformer stacks of the Hourglass architecture, whereas others will be processed by the whole architecture. This targeted allocation of compute helps the model focus resources on tokens that require greater attention.

To make this clearer in the paper, we have: (i) complemented Fig. 5 with an additional per token compute graph that illustrates the per-token compute used for different tokens, and (ii) improved the explanations provided in the Introduction as well as Sec. 3.1. In addition, we will keep the animation on the project website, which we will de-anonymize upon acceptance.

Second, the model’s reliance on curated, high-quality datasets raises concerns about scalability and applicability to domains with limited or noisy data. In real-world scenarios, point clouds are often noisy, yet the paper does not evaluate the robustness of the proposed approach under such conditions.

This is a good observation. We originally provide some brief information regarding this in the Appendix, but it was not mentioned in the main text. During training, we augment the point cloud positions and normals with gaussian noise to simulate noisy real-world conditions. In addition, we make sure that point clouds are never sampled from internal structures --which may be present in artist meshes, but are unlikely to appear in scanned or automatically generated meshes-- to aid generalizability to non-artist like meshes. This treatment is crucial, and it is in fact, thanks to this that Meshtron is also able to receive point-clouds from scans as well as iso-surfacing methods (which are often noise and imperfect) and successfully map them to artist-quality meshes.

To address your concern, we have updated Sec. 4 to include details regarding this treatment and their role in enhancing robustness. Additionally, we have added Fig. 11, which illustrates how adjusting input noise allows Meshtron to balance creativity and faithfulness, showcasing its flexibility and applicability in scenarios with varying levels of input quality.

Finally, although the authors claim significant advantages over state-of-the-art methods, they only compare their approach to one such method. Broader validation and comparisons with a wider range of methods are necessary to substantiate their claims.

We appreciate this feedback and agree that broader validation strengthens our claims. In response, we have added comparisons to iso-surfacing methods FlexiCubes and DMTet, as well as native mesh methods MeshGPT and MeshAnythingV1. These comparisons are detailed in the Introduction and Sec. 4.3. Visual comparisons are provided in Fig 2 and Fig 14, in addition to a very large gallery of results and additional comparisons in our project website.

In summary, in comparison to methods that generate meshes directly (MeshGPT, MeshAnythingV1 / V2), Meshtron is able to generate meshes with many more faces at higher resolution, giving it the ability to create artist-like meshes of complex 3D objects. When compared to iso-surfacing methods (DMTet, FlexiCubes), which can produce meshes with very high face counts, Meshtron is able to do so while obtaining high-quality topology, featuring high-geometric detail and well-structured tessellation, just at it is observed in artist-like meshes. Iso-surfacing methods, on the other hand, often suffer from overly dense tessellation, bumpy artifacts, oversmoothing and insufficient geometric detail, creating a notable quality gap between their meshes and those crafted by artists.

—

We hope that these responses clarify your questions and concerns. Please let us know if you have any follow-up / additional questions.

Best regards,

The Authors