EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Presentation.

Thank you for the suggestion! We have updated the manuscript as follows:

• Moved the user study table into the main paper for more quantitative comparison.
• Clarified the discussion on image-conditioned generation when introducing face number control to avoid confusion.
• Corrected the typo in "vs."

Q2: Engineering tricks to scale up training.

Thank you for your interest! We employ standard engineering techniques, including flash attention, gradient checkpointing, and bfloat16 mixed-precision training to optimize GPU memory usage. With these optimizations, each batch requires approximately 20 GB of GPU memory, accommodating up to 4,000 faces (roughly 20,000 tokens per sequence).

We promise to release the code soon to facilitate better understanding and reproducibility of our method.

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Lack of details on the point cloud sampler.

Thank you for the reminder! We simply sample 8192 random points from the surface of the input mesh.

Since the primary use case for the point-conditioned model is retopology of an existing dense mesh, we introduce random surface perturbations by adding Gaussian noises. This enhances robustness to noisy or irregular surfaces, which are commonly encountered in meshes generated by the Marching Cubes algorithm.

However, we do not apply augmentations related to point count or missing regions, making the model less effective in such scenarios. Designing augmentations based on the actual data distribution could possibly enable the model to learn to predict with fewer points or handle missing regions.

Q2: Comparison with MeshGPT and VQ-VAE based tokenizer.

It is challenging to make a fair comparison with MeshGPT due to differences in training settings. For example, MeshGPT is trained on single-category data from ShapeNet (e.g., tables, chairs), while our model is trained on general objects from Objaverse. Additionally, MeshGPT quantizes vertices at a resolution of $128^3$, which results in significant information loss, while we use a finer resolution of $512^3$.

Previous studies have also explored the performance of VQ-VAE. For example, MeshAnything continues to use VQ-VAE but reports that avoiding compression (retaining 3 tokens per vertex) yields better results compared to MeshGPT's default setting of 2 tokens per vertex. Both MeshXL and MeshAnythingV2 have abandoned VQ-VAE entirely, as they found compression to negatively affect performance.

Q3: Results on image-to-mesh generation with varied image poses.

Thank you for the suggestion! During training, we align the generated mesh with the pose of the input image, following practices from prior image-to-3D methods.

Most test images, however, are 2D paintings collected from the web, making it challenging to alter the pose of the same object. Nonetheless, our project page includes 52 image-conditioned samples, showcasing more challenging poses, such as side or top views.

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Unfair comparisons due to different training datasets.

We acknowledge the difficulty in ensuring that the datasets used are identical. MeshAnything does not release the specific subset list of Objaverse and additionally incorporates data from ShapeNet. Moreover, our model's capability to support longer sequences enables us to utilize a greater number of samples from Objaverse, which plays a critical role in enhancing performance.

Q2: Is the training sequence unique for each mesh? How did you define the start of the sequence?

Yes, similar to previous works, we sort the faces in an empirical $y\text{-}z\text{-}x$ order and always begin with the first half-edge of the first face. The EdgeBreaker algorithm is deterministic when the starting half-edge is fixed, which ensures that the sequence is unique for each mesh.

Q3: Comparison of inference speed.

Thanks for the advice! Since MeshAnything and MeshAnythingV2 do not support face count control, we report only the average generation speed:

Due to the larger number of parameters in our model, the generation speed is slightly slower compared to MeshAnythingV2. However, our model demonstrates greater robustness and delivers better performance.

Q4: Details on user study.

We have revised the manuscript to include additional details about the user study. Volunteers were asked to evaluate the results based on three criteria: geometry consistency with the input point cloud, visual appearance of the triangle faces, and overall mesh quality. Each volunteer was presented with mixed results from randomly selected methods. Additionally, we have provided more samples—68 point-cloud-conditioned and 52 image-conditioned meshes—generated by our method on our project page for better comparisons.

Q5: Lack of quantitative evaluations like CD or ECD.

Thanks for the advice! We believe that surface metrics may not effectively capture the aesthetic quality of the mesh, which is the primary objective of auto-regressive mesh generation. Additionally, on our more challenging test dataset, baseline methods often fail noticeably and are unable to produce complete mesh surfaces, further diminishing the relevance of surface metrics in evaluating performance.

Q6: Meaning of the dashed line in Figure 2.

The dashed line indicates that our image-conditioned diffusion model is trained separately. Initially, we train the point-conditioned model (ArAE), as depicted above the dashed line, and subsequently use the learned fixed-length latent space to train the diffusion model, as shown below the dashed line.

Q7: Adherence of face count control.

Since our face count control is learned implicitly, the model cannot be strictly constrained to adhere to the specified range. We employ four learnable tokens to control the face count: unconditional, $(0, 1000)$, $[1000, 2000)$, $[2000, 4000)$. Our observations indicate that the control token $[1000, 2000)$ provides the most robust face count control and generation. In contrast, the other tokens exhibit less robustness and may also be influenced by the complexity of the input shape.

Thanks for the reply!

Following your advice, we conducted a new qualitative comparison experiment to evaluate the Chamfer Distance (CD) and Hausdorff Distance (HD) between the generated meshes and the input point clouds. Due to the lack of widely recognized evaluation datasets, we tried to construct two test datasets from different sources:

• AI-Mesh: This dataset comprises 30 meshes generated by other image-to-3D models, extracted using the Marching Cubes algorithm.
• Artist-Mesh: This dataset consists of 100 artistic meshes obtained from basemesh, which were not seen during the training process.

All meshes are normalized into $[-1, 1]^3$. The summarized evaluation scores are presented below:

We observed the following:

• Our model achieves the best performance on surface metrics, even when using only point clouds as input. In contrast, the MeshAnything series further leverage surface normals.
• MeshAnythingV2 performs significantly worse on the more challenging AI-Mesh dataset, often failing to generate complete surfaces in these cases (consistent with the results shown in Figure 5).

As the deadline for editing the PDF is approaching, we will include these experiments in the final version of the paper.

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Difference from MeshGPT and MeshAnything in terms of the core generative approach.

Our core contribution lies in a novel mesh tokenizer designed for improved compression of mesh sequences, which enhances training efficiency and boosts performance. For the core generative approach, auto-regressive mesh generation methods all rely on transformers and next-token prediction.

Q2: The generated geometry may not adhere to the conventions that artists follow.

The goal of auto-regressive mesh generation is to learn and replicate artistic mesh conventions from large datasets. We believe our work advances this objective and can generate meshes that are indistinguishable from artist-created ones.

Q3: Limited number of generated faces.

As discussed in the paper, although 4000 faces may not suffice for complex objects, this count is adequate for many common objects, as demonstrated in both the paper and on the project page. Moreover, previous methods are limited to generating up to 1600 faces with lower robustness.

To further increase the face count, one approach is to develop improved tokenization algorithms that achieve a higher compression rate. Another possible direction is to employ local attention and perform sliding-window inference, based on the observation that triangle topology relies more on local features than on global features.

Q4: Unfair comparison with Unique3D.

We argue that our work is the first to achieve end-to-end image-conditioned artistic mesh generation. Existing image-to-3D methods continue to depend on isosurfacing techniques, such as Marching Cubes, to extract dense meshes, which fundamentally differ from our approach. We selected Unique3D for comparison as it is among the most recent open-source works in the field of image-to-3D, and this comparison aims to highlight the differences between dense meshes and artistic meshes.

In this sense, CLAY also produces dense meshes extracted using Marching Cubes similar to Unique3D, which are far from artistic meshes. We acknowledge that our approach may offer less geometric detail and generalization capability compared to 3D diffusion-based methods; however, this is not our primary focus. As in your own words, we believe "the comparison may not accurately reflect the strengths and weaknesses of each method, as they are optimized for different conditions and face distinct sets of challenges".

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Lack of discussion about training progress such as loss curves.

Thank you for the reminder! Our model is initially trained on approximately 112K data samples from Objaverse and Objaverse-XL. Subsequently, it is fine-tuned using 44K higher-quality samples from Objaverse. During the first training stage, the loss decreases from around 6 to 0.37, while the second stage further reduces the loss to 0.32. Notably, our data augmentation increases the training loss initially; however, the final performance proves to be more robust upon convergence.

Q2: Is it possible to perform unconditional mesh generation?

Yes, we conducted early-stage experiments with unconditional generation. Instead of prepending conditional tokens to the sequence, we trained an auto-regressive model using only mesh tokens. However, we found that unconditional generation has limited practical applications. Unlike language models, which are inherently suited for completion tasks, mesh sequences produced by EdgeBreaker are not easily prepared as the input. Consequently, we chose to focus on conditional generation from point clouds or single-view images.

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Presentation.

Thank you for the feedback! We have revised the manuscript to enhance its comprehensiveness:

• Missing citations in the introduction have been added to appropriately acknowledge EdgeBreaker.
• Due to space limitations in the main paper, we found it challenging to present both the original EdgeBreaker algorithm and our variant. Therefore, we have moved the detailed descriptions to the supplementary material and opted to use an example in the main paper to illustrate our method. Additionally, we have added text in the main paper to direct interested readers to the supplementary material for more details.

Q2: Public availability.

Thank you for your interest! We promise to release the code recently for better understanding and reproduction of our method.

Unfortunately, the models may not be released at the same time due to dataset licensing issues of Objaverse-XL. However, we have made many test samples available on our project page, which should assist future methods in making comparisons.

Thanks for the reply!

We apologize for the missing point in the presentation. To make it easier for readers to find information, we aim to keep all implementation details in a single location. However, due to page limits of the main paper, we are unable to include the full content without omitting critical information. Instead, we will include additional sentences in the final version to direct interested readers to the supplementary materials.

Regarding the dataset issue, this paper is not solely an open-source project. To avoid anonymity violations and to the best of our knowledge, there are very few precedents of major corporations with formal legal departments releasing models trained on datasets with contentious licenses such as Objaverse-XL. Nevertheless, we are actively working to make the checkpoints available as soon as possible. Additionally, we have provided all necessary details, including final training losses, to ensure reproducibility.

Lastly, we agree that a more extensive quantitative experiment is important. We conducted a new qualitative comparison experiment to evaluate the Chamfer Distance (CD) and Hausdorff Distance (HD) between the generated meshes and the input point clouds. Due to the lack of widely recognized evaluation datasets, we tried to construct two test datasets from different sources:

• AI-Mesh: This dataset comprises 30 meshes generated by other image-to-3D models, extracted using the Marching Cubes algorithm.
• Artist-Mesh: This dataset consists of 100 artistic meshes obtained from basemesh, which were not seen during the training process.

All meshes are normalized into $[-1, 1]^3$. The summarized evaluation scores are presented below:

We observed the following:

• Our model achieves the best performance on surface metrics, even when using only point clouds as input. In contrast, the MeshAnything series further leverage surface normals.
• MeshAnythingV2 performs significantly worse on the more challenging AI-Mesh dataset, often failing to generate complete surfaces in these cases (consistent with the results shown in Figure 5).

As the deadline for editing the PDF is approaching, we will include these experiments in the final version of the paper.