ARMesh: Autoregressive Mesh Generation via Next-Level-of-Detail Prediction

We are profoundly grateful to the reviewer for generously dedicating their time to provide us with such invaluable feedback. In summary, we have decided to reserve additional experiments (e.g., conditional generation on large-scale datasets) for future work. We truly appreciate your understanding of our intentions. Your understanding and support mean a great deal to us, and we would like to elaborate further on this.

We wholeheartedly agree with the reviewer that providing additional experimental results on conditional generation (such as from images, texts, or point clouds) would significantly enhance the persuasiveness of our method’s effectiveness. However, we currently find ourselves constrained by limited computational resources and time, which regrettably prevents us from conducting these additional experiments that require large-scale training, as this process is indeed quite time-consuming and costly. We hope you can kindly understand our limitations.

We also sincerely appreciate the reviewer’s thoughtful suggestion regarding minimal experiments (e.g., fine-tuning on a subset). However, we humbly feel that fine-tuning networks from other methods may present challenges for a fair comparison, as those networks have been trained on significantly more data than a network trained from scratch with a smaller dataset. We are grateful for your understanding of our perspective on this issue.

In our current paper, we have conducted extensive ablation studies to explore the potential properties of our newly proposed representation, along with some unconditional generation results that serve as preliminary experiments demonstrating the efficacy of our progressive simplicial complex representation combined with deep learning for facilitating mesh generation in a coarse-to-fine manner. We humbly believe that scaling up to larger datasets and exploring diverse tasks can be effectively pursued in future studies. We sincerely appreciate your understanding, and we hope our work can still make a positive contribution to our community.

Thank you once again, and we wish you a wonderful day!

We are sincerely grateful to the reviewer for generously dedicating their time to provide us with such constructive feedback. We are committed to addressing your questions in a manner that we hope will effectively resolve your concerns. In line with this year's rebuttal policy, we will provide a textual response only, without any external links. Thank you for your understanding and support.

1. Training and inference speed. This aspect is indeed quite practical, and we appreciate the opportunity to discuss it. Both our method and MeshGPT utilize auto-regressive techniques, but the primary distinction lies in what determines the generation complexity: our approach is based on the number of vertices $V$, while MeshGPT depends on the number of faces $F$. Typically, for a mesh, there is an approximate relationship of $F \approx 2V$, which suggests that our method may require fewer geometric elements for generation. In our experiments, when generating 100 meshes randomly and unconditionally, our method generally takes about 60 seconds per shape, whereas MeshGPT averages around 80 seconds. We would like to humbly emphasize that these speed metrics were obtained using a single H20 GPU. Regarding training speed, it largely depends on the computational resources available; generally, longer training times lead to better generalization. For our model, we typically require about 16 GPU-days, while MeshGPT necessitates a total of 24 GPU-days to complete its training. Lastly, we wish to acknowledge that actual performance may vary slightly across different platforms and machines. We appreciate your understanding in this matter.

2. Extension to conditional generation. Thank you for highlighting the importance of conditional generation tasks! They are indeed invaluable for practical scenarios. We believe that our proposed framework has the potential to support additional tasks, such as conditional refinement, editing, or generation from images or point cloud inputs. To accommodate different modalities, we may consider established conventions that convert images or texts into tokens using ViT or BERT for feature extraction, subsequently fusing these features into our transformer via cross-attention mechanisms. We must humbly acknowledge that achieving these tasks requires very large models and substantial amounts of data. Unfortunately, we are currently limited by our computational resources and time, making it challenging to train large-scale models on extensive datasets during the rebuttal period. We sincerely apologize for this limitation and remain committed to pursuing this direction by extending our methods to perform conditional generation tasks in the future. We sincerely appreciate your understanding and consideration. Thank you for your support.

3. Failure rate. Thank you for highlighting the aspect we previously overlooked. Indeed, there are notable works in the related field aimed at supporting the generation of meshes with a high face count, such as Meshtron, which claims the ability to generate meshes with up to 10k faces, significantly more than what our current study addresses. In our experiments, we found that the failure rate of our method is acceptable, remaining below 5% across 100 randomly generated objects. This metric was measured on the ShapeNet dataset, which includes shapes with fewer than 1,700 vertices and 800 faces each. We sincerely acknowledge that we currently lack the computational resources and time necessary to conduct large-scale experiments focused on generating high face counts. As such, we have positioned our paper to explore the potential advantages of using our progressive simplicial complex representation before scaling up to larger models or datasets. We appreciate your understanding in this matter. Additionally, we have observed that using our trained model for extrapolation, specifically, performing additional refinement steps with training on a low face-count dataset, is not feasible. This limitation arises because our tokenization process includes the vertex index, which restricts our model's capabilities based on the vertex count. As a result, the model has not encountered cases with a high number of vertices, leading to poor generalization when applied to meshes with high face counts. We are grateful for your understanding of this limitation. We will certainly discuss this point in the limitation section of our revised paper. Finally, we would like to share a strategy for managing failure cases. We have observed that unsatisfactory results often become apparent within the first few generation steps. For instance, after generating just 10 steps, we can often determine whether the current generation will yield acceptable results. If the produced outputs show low confidence in generation (as indicated by the classification probability of the generated tokens), we can recognize a potential failure early and stop the generation process. Thank you once again for your valuable feedback.

4. Lack of comparison experiments with recent strong baselines and the absence of experiments on conditional generation. We sincerely apologize for not including comparisons with recent strong baselines (such as publicly available methods like MeshXL, MeshAnything, and DeepMesh) on the tasks of conditional generation from texts, images, or point cloud inputs. We are committed to discussing and citing these methods, as well as presenting our limitations, in our revised paper. We must acknowledge that, due to limited computational resources and time, we are currently unable to perform these tasks, as they require training on a very large dataset and utilizing a large-scale model. Instead, we chose to focus on unconditional generation tasks, which our baseline method, MeshGPT, also addresses. Comparing our results with MeshGPT is more feasible for our experiments, given the lower resource requirements. Additionally, we positioned our paper to explore the benefits of using the newly developed progressively simplicial complex representation before scaling up to such large datasets and models. We genuinely hope for your understanding regarding this limitation and greatly appreciate your support.

We are sincerely grateful to the reviewer for generously dedicating their time to provide us with such constructive feedback. We are committed to addressing your questions in a manner that we hope will effectively resolve your concerns. In line with this year's rebuttal policy, we will provide a textual response only, without any external links. Thank you for your understanding and support.

1. The tokenization process seems too complicated, making the sequence quite long. Thank you for highlighting this limitation! We would like to clarify that our paper does not specifically focus on proposing an effective tokenization method; rather, we present a straightforward solution to make it work (as noted in footnote 4 at the bottom of page 6). In practice, we adapted techniques from LLMs to reduce the number of tokens, employing Byte Pair Encoding (BPE) for compression in our experiments. On average, this approach can reduce token lengths by approximately 2$\sim$3 times. For example, a mesh with 500 vertices and 1,000 faces (where $F \approx 2V$) and an average tokenized length of 10 would require $(500-1) \times 10 = 4,990$ tokens to represent a highly complex mesh. However, with BPE, this number can be significantly reduced to about 2k tokens. In comparison, a baseline method using per-face per-coordinate encoding would need $1,000 \times 9 = 9,000$ tokens, which can also be reduced to around 2k tokens using BPT [1]. It is evident that the compression ratio of our method is comparable to recent strong compression baselines, which underscores a major benefit of using progressive representation for compression, as highlighted in [2]. We truly appreciate your understanding and support in this matter.

2. How can we utilize the intermediate results? We appreciate your feedback on this aspect! One of the key advantages of our representation is the ability to obtain intermediate (coarse) results without needing to generate the mesh with the highest complexity. For instance, consider a simple case where our ground truth is a mesh with 500 vertices and 1,000 faces. Conventional methods may require a total of 1,000 steps for generation; if fewer than 1,000 steps are taken, the resulting mesh may appear incomplete, with some faces missing, making it less representative of the ground truth. In contrast, our method allows us to obtain the ground truth mesh in just 499 steps. Moreover, in many cases, we do not require a mesh of such high complexity. For example, if we only need a simplified version, we can perform even fewer steps (e.g., 100 steps), and the resultant mesh will still closely resemble the original ground truth. By reformulating mesh generation in a coarse-to-fine manner, we can effectively control both geometric complexity and the time and computational budget required to generate a mesh with the desired complexity. Finally, the coarse mesh can be utilized in various scenarios, such as game development and streaming or previewing the generation process. We sincerely appreciate your concerns and support in this discussion.

3. What is the difference between generating the whole mesh and them simplifying it, and the proposed method? We sincerely appreciate your attention to this point! It is indeed a fascinating aspect. The primary advantage of our approach is its ability to save both time and computational resources. When the network is trained with a high-fidelity mesh dataset, conventional methods often require an excessive number of generation steps to produce high-fidelity meshes, making the process quite costly. Additionally, an extra simplification process is needed if a low-poly result is desired. While one could train multiple models with varying mesh complexities to partially address this limitation, we believe this is not the ultimate solution. Our approach allows for greater control over the time and computational budget required for generation, enabling us to achieve results with fewer inference steps. We hope our method can make a positive contribution to our community, and we are truly grateful for your kind support and understanding.

[1] Haohan Weng, et al. Scaling Mesh Generation via Compressive Tokenization. CVPR 2025.

[2] Hugues Hoppe. Progressive meshes. SIGGRAPH 1996.

We are truly grateful to the reviewer for dedicating their valuable time to provide us with such constructive feedback. We sincerely appreciate your insights and will now address your questions, hoping that our responses will adequately address your concerns. In accordance with this year's rebuttal policy, we will provide a textual response only, without any external links. Thank you once again for your understanding and support.

1. We are truly thankful to the reviewer for their kind appreciation. We wholeheartedly agree that introducing topological changes allows any chosen mesh to be simplified to a single point, serving as a universal starting point for generation. Your recognition of this point is deeply appreciated and means a great deal to us.

   (a) How are these virtual edges represented in the later tokenization process? The presence of a (virtual) edge is encoded in the topological list, with the first element indicating whether the edge exists. Specifically, the case V0 in Table 1 signifies that the edge is virtual, while V1 indicates that the edge is real and will be present after the vertex split. The topological list is then tokenized into a sequence of classification tokens, as illustrated in Figure 5. We sincerely recognize that our explanation of these new concepts may not be fully sufficient, and we will provide additional clarifications and examples in the supplementary materials to enhance understanding. We truly appreciate your understanding and support.

   (b) Flexible topology changes. Thank you for your insightful observation! We are grateful for the insights that guide us in this direction. At each vertex split step, we have the flexibility to modify the topology of neighboring simplices as needed. For instance, in the case of a mesh with 100 vertices, where each vertex has an average of 10 neighboring simplices, we estimate an upper bound of $10 \times (100 - 1) = 990$ opportunities to modify the local topologies. It’s important to note that this is an upper bound, as the topological constraints outlined by the four rules will slightly reduce this number. This observation underscores the remarkable flexibility of our approach. However, we have observed some unacceptable changes when utilizing a poorly trained model with limited generalization ability. This has led to generated meshes that appear random and do not reflect typical shapes. To address this challenge, we believe that a straightforward yet effective solution is to ensure that the model is well-trained. Additionally, we have some ideas that may address this problem more comprehensively. For instance, we could allow the model to predict a coarse mesh with 10 vertex splits that introduce all the essential topological changes contributing to the overall topology, while the remaining 90 vertex splits would maintain the overall topology. In this scenario, our model would effectively reduce to a "progressive mesh" method (rather than a "progressive simplicial complex" method), introducing mesh subdivision without topological changes, thus enabling us to better control the complexity of these changes. We view this as an important direction for future work, and we appreciate your guidance in this matter.

2. Are the four rules in Line 180 complete? We are truly grateful for your interest in these four rules! We humbly believe that these four rules are complete, serving as both sufficient and necessary conditions. We have carefully derived them by thoroughly enumerating all possible cases in tables and summarizing the rules that encompass each entry. For those who are interested, we would be glad to provide a rigorous proof in the supplementary material. In a broader sense, our proof can be viewed as somewhat analogous to deriving logical expressions from truth tables.

3. So those values in that example are not corresponding to the example in Fig 4? The topological list $[1, 2, 5, 3, 3, 4, 6, 7]$ presented in line 154 corresponds to Figure 4. As indicated in lines 165~166, this list can be converted to: V1, E0, E3, E1, E1, E2, F0, F1, which describe the topological changes of the eight simplices in the left subfigure of Figure 4. For instance, vertex 0 is classified as V1 because a new edge 5 is created following the vertex split; face 1 is categorized as F1 because it was originally connected to vertex 0, but after the split, it connects to the new vertex. We also observe that the lengths of these two lists are equal, reflecting the number of star simplices, which includes vertex 0 itself along with its adjacent edges and faces. We appreciate your attention to these details!

4. It's a missed opportunity to use the newly defined rules/topo case in Line 167-184 to explain the example in Fig. 4. We have provided explanations for the topological cases in Figure 4, along with examples in lines 172$\sim$174, and we briefly discuss the reasons for the existence of these rules in lines 174$\sim$178. However, we acknowledge that the few examples presented in the current paper may not be sufficient, and we intend to provide further clarifications. For the zero-dimensional simplex, we focus on the selected vertex 0. We observe that after the split, it creates an edge 01 that connects the original vertex 0 to the newly created vertex 1, which corresponds to the topological case V1 in Table 1. Additionally, for edge 3, which originally connects to vertex 0, after the split, it is no longer connected to vertex 0 but instead to vertex 1. This indicates a transition from the source vertex to the target vertex, representing the topological case E1. Similarly, face 1 falls under the topological case F1, as it is now incident to the new vertex 1, as shown in Table 1. Regarding the constraint rules discussed in lines 180~184, we will provide an example and focus on the second rule for further explanation. If there are two edges, e1 and e2, incident to the split vertex and they are subsets of a face with a topological label F0, then the topological labels of e1 and e2 cannot be E1. This is because a face labeled F0 implies that its sub-simplices (e1 and e2) must also be incident to the source vertex $s$, which contradicts the case E1 where the edge is connected to the target vertex $t$. We appreciate your patience and understanding, and we will elaborate further in the supplementary materials, including an additional video to illustrate this point more clearly. Thank you for your interest!

5. The fitting experiment in Fig 11: can the authors provide a more detailed description of how this experiment is done? Let us provide some additional details regarding the experiments presented in Figure 11. DMesh is a differentiable Delaunay triangulation method that learns from a mesh input. We use the target mesh as input and allow DMesh to perform optimization, with its intermediate outputs reflecting its progressive learning outcomes. EdgeRunner, on the other hand, is a large-scale pretrained direct mesh generation network that takes a point cloud as input and produces a mesh in a predefined scanning order (not in a coarse-to-fine manner during the generation process). However, it offers only three levels of complexity conditioning, 1k, 2k, and 4k, without options for other complexities, such as 3k. Consequently, we utilize their method by feeding a point cloud as geometry conditioning and the complexity conditioning token (1k, 2k, or 4k) as input, resulting in a mesh output. We would like to note that although it is claimed that a 4k conditioning should ideally produce a mesh with 4k faces, our observations indicate that the generated mesh contains approximately only 2k faces. This suggests that their method may not effectively control geometric complexity, even when provided with an additional face count (complexity) token as input. Finally, our approach is similar to that of DMesh, where we require the target mesh as input and learn the progressive generation process. The intermediate figures illustrate the progressive results during auto-regressive generation. As shown, our method can accurately control geometric complexity; the number below our subfigure, such as 256, indicates that the mesh has exactly $256 + 1 = 257$ vertices. The face count can be indirectly and less accurately influenced by the number of vertices. We utilize the method described in Section 4.3 to learn the generation process using cross-entropy loss, as we essentially perform token classification after tokenization. The generation process always begins with a single vertex. We sincerely appreciate your interest in our implementation details!

6. Any failure case examples? Yes, we acknowledge that our method can sometimes yield unsatisfactory results, which we believe are primarily due to generalization issues. Unfortunately, due to this year's rebuttal policy, we are unable to include images here; instead, we would like to describe our findings. Each vertex split operation encodes both geometry (e.g., relative offset) and topology (e.g., the topological list), and we have observed that these failure cases reflect both aspects. Geometric artifacts may lead to local shifts; for instance, a chair's leg might be slightly offset by a translation vector of $(0.5, 0, 0)$ due to the discretization of the offset. While the overall topology remains unchanged, the leg is simply misplaced. Topological artifacts present more significant challenges. For example, a mesh may be generated correctly in the first 10 steps, but if an incorrect topology is predicted at the 11th step, this error can cause the result to deviate from the original intent, often resulting in an unrecognizable outcome as the shape falls out of distribution. We are committed to including a detailed analysis of these artifacts, complete with figure examples, in our revised paper. We sincerely appreciate your understanding and feedback as we work to improve our approach.