MeshXL: Neural Coordinate Field for Generative 3D Foundation Models

📝 W1: The limitation of sequence length.

💡A: The main focus of our paper is to establish end-to-end methods with less inductive bias and pave the way for scaling up learning from large-scale 3D data. We restrict the number of faces to 800 for better alignment with PolyGen and MeshGPT. However, by equipping MeshXL with modern LLM techniques, such as RoPE and ring attention, we are ready to extend our method to more complicate 3D meshes. Additionally, we will also be able to train larger MeshXL networks.

📝 W2: Potential redundancy in the mesh representation and intersecting faces.

💡A: The redundancy in mesh representation paves the way for large-scale pre-training. We acknowledge that the NeurCF representation does generate shared vertices multiple times. However, this enables us to represent each 3D mesh with only one coordinate sequence and design an end-to-end pipeline on large scale 3D data.

Analysis of potential surface artifacts. The potential occurrence of surface artifacts could be the limitation for our MeshXL, and all existing auto-regressive mesh generation methods (PolyGen and MeshGPT) for now. However, after training directly on large-scale 3D meshes, MeshXL can already generate high-quality 3D mesh data under a high success rate. It is also possible to further improve MeshXL by treating traditional methods as reward models or filters to eliminate the surface artifacts.

📝 Q1: The resolution of MeshXL.

💡A: To align with MeshGPT, we choose the resolution of $N=128$ in all our experiments. However, our MeshXL adopts an architecture close to large language models, whereas we can easily extend MeshXL to a larger $N$ with a time and space complexity of $O(N)$ by simply learning a $N$-sized coordinate embeddings and an output classifier with $N$ output channels. Comparing to the $O(N^3)$ complexity in traditional hard coding algorithms, it is easier for us to train models with higher resolution. We will also release the models with larger resolution in the future.

📝 Q2: Inference time analysis

💡A: The inference time of our MeshXL is closely related to the numbers of generated faces and the model sizes. We perform inference time analysis with a batch size of one using BFloat16 on a single RTX 3090. We carry out a an analysis on 1) the average inference time of generating a given number of triangles.

We also show 2) the average inference time of generating 3D meshes.

📝 Q3: Fair comparisons and evaluations on larger datasets.

💡 A: We reproduce MeshGPT with gpt2-medium (355m) and re-implement MeshXL (marked as MeshXL$^{ShapeNet}$) by first training the model on all ShapeNet categories before fine-tuning it to a specified category. 📊 Please refer to the global rebuttal for exact results. With a similar amount of parameters (350m), our method could consistently achieve better generation results with a higher COV score, a lower MMD score, and a closer 1-NNA score to 50%.

Additionally, to study the effectiveness of large-scale pre-training, we conduct evaluations on Objaverse. We show that as the model size grows, MeshXL exhibits better 3D mesh generation quality, with a higher COV score, a lower MMD score, and a closer 1-NNA score to 50%.

Additionally, we have provided additional generation results in the uploaded PDF file. We have also open-sourced our code and pre-trained weights to the community.

📝 Q4: Justification of Neural Coordinate Field.

💡A: We thank the reviewer for pointing out this constructive suggestion.

1. Justification of Neural Coordinate Field. Neural coordinate is a representation that treats vertex coordinates as coordinate embeddings. Meanwhile, field should be a function within continuous domain. As we currently learn coordinate embeddings that uniformly spread along each axis, we will try learning interpolation functions (e.g. linear interpolation, gaussian interpolation, or B-spine basis functions) to extend our embeddings to the continuous domain. We will also try extending our method to the continuous domain by replacing the coordinate embeddings with sinusoid function. We will perform detailed experiments in our revision.
2. Relation to the ordering in PolyGen. We adopt the same vertex representation as PolyGen, but a different mesh representation. Our MeshXL and PolyGen both represent vertices with discrete coordinates. However, MeshXL represents a 3D mesh only with an ordered sequence of coordinate, while PolyGen decouples the vertices generation and polygon generation, and adopts two sequences for each mesh, i.e., the vertex sequence and the face index sequence to connect the generated vertices. Therefore, the potential redundancy in our representation in turn supports end-to-end training and better suits learning from large-scale 3D data.

📝 W1: Reasons why single stage method works, and analysis on previous works.

💡 A: We thank the reviewer for helping us improve our paper.

1. The coordinate sequence representation and auto-regressive generation make our method come true. With a well-defined ordering system, each 3D mesh can be represented by one unique coordinate sequence. Additionally, decoder-only transformers have long been proven to excel in auto-regressive sequence generation. Therefore, by establishing a decoder-only transformer trained on the collection of coordinate sequences (3D meshes), our one-stage pipeline is able to generate high-quality 3D meshes.
2. Analysis of existing works. Though previous methods have achieved initial success in 3D assets generation, they suffer from certain limitations.
   1. To preserve high-frequency information, the point and voxel representations require dense sampling on object surfaces, which inevitably lead to great redundancy when it comes to flat surfaces.
   2. The reconstruction-based methods rely heavily on the quality of the generated multi-vew images.
   3. The VQVAE-based 3D generation methods require a two-stage training strategy, which poses extra challenges in large-scale training.

📝 W2: Fair comparison with previous methods.

💡 A: We reproduce MeshGPT with gpt2-medium (355m) and re-implement MeshXL (marked as MeshXL$^{ShapeNet}$) by first training the model on all ShapeNet categories before fine-tuning it to a specified category. 📊 Please refer to the global rebuttal for exact results. With a similar amount of parameters (350m), our method could achieve better generation results with a higher COV score, a lower MMD score, and a closer 1-NNA score to 50%. After pre-trained on larger datasets, our method can achieve even better generation results.

📝 W3: Comparison with deformation-based and command-based methods.

💡 A: The training of MeshXL enjoys a better flexibility comparing to both deformation-based or command-based methods.

1. MeshXL vs. deformation-based methods. The deformation-based methods build on prior geometry knowledge for great initialization. However, The deformation-based method requires external expert knowledge for good template initialization, which is difficult to generalize to complex 3D meshes. However, our MeshXL direct learns from large-scale collection of diverse 3D mesh data.
2. MeshXL vs. command-based methods. The command-based methods adopt command sequences to represent the creation process of the visual data. However, collecting commands is much harder than collecting the generated results (3D meshes). Additionally, the command space requires careful designs and is often limited (arc, circle, and extrude in DeepCAD) to create only simple objects, while our MeshXL learns directly on large-scale 3D mesh data with great diversity.

📝 Q1: Comparison with MeshGPT

💡 A: See weakness 2.

📝 Q2: Reason why end-to-end training leads to better mesh generation results.

💡 A: The main motivation of our work is to explore an simple and effective mesh representation to support large-scale training. Therefore,

1. We can hardly say whether two-stage methods are better or worse than end-to-end learning. However, training a good vector-quantized tokenizer is challenging [R1]. In latent diffusion, instead of adopting a VQVAE, the denoising diffusion process learns to generate latent codes predicted by a VAE, which learns the feature distribution of the latent codes.
2. Our end-to-end design better supports large-scale training.
   1. By representing a 3D mesh with one unique coordinate sequence, MeshXL can directly learn from large scale 3D data. Meanwhile, MeshGPT requires to train a vector-quantized representation of vertex feature. Additionally, PolyGen requires to generate a vertex sequence and then connect the generated vertices into polygons. Therefore, learning a good alignment between different modules is challenging and requires careful supervision.
   2. Based on the quantitative results on ShapeNet categories in Table 2, MeshXL can produce high-quality 3D meshes with higher COV scores, lower MMD scores, and closer 1-NNA scores to 50% than previous methods.

[R1] Li, Tianhong, et al. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

📝 Q3: Fair comparison with existing methods

💡 A: See weakness 2.

📝 Q4: Scaling law does not apply to the lamp category.

💡 A: The limited training data leads to overfitting. After pre-processing the ShapeNet dataset, the lamp subset contains only 565 samples for training. Therefore, larger models will easily overfit. Instead, we show in Figure 3 of the main paper that when pre-training on extensive 3D data, MeshXL achieves both lower training and validation loss. Additionally, by evaluating on Objaverse, we also notice that as the model size grows, we achieve better generation results.

📝 Q5: Normal vector comparison with GET3D

💡 A: In Figure 7, we show the normals to compare the smoothness of object surfaces. The results show that 3D meshes generated by GET3D have rough surfaces with tens of thousands of triangles, while ours depict the 3D shape with much smoother surfaces and less triangles.

📝 Q6: Open-sourcing.

💡 A: We have released our code and pre-trained weights to the community. We will also keep updating for additional features.

📝 W1: The novelty of MeshXL.

💡 A: The motivation of our MeshXL is to extend the mesh representations, architecture design, and training strategy in existing auto-regressive methods, i.e. PolyGen and MeshGPT, to support efficient large-scale training on extensive 3D mesh data. Specifically, our method improves existing methods from the following aspects:

1. Training strategy. Both MeshGPT and PolyGen adopt a two-stage training pipeline. This, however, leads to a complicated training procedure less favors large-scale generative pre-training. However, MeshXL is a fully end-to-end pipeline, which naturally favors large-scale pre-training.
2. Mesh representation and architecture. By modeling 3D mesh data into one unique coordinate sequence, our MeshXL only consists of a decoder-only transformer for next-coordinate prediction. Meanwhile, PolyGen requires a vertex sequence and a vertex index sequence to turn vertices into polygons. Additionally, MeshGPT adopts a vector-quantized approach by learning a fixed-size codebook to represent the vertex feature before learning to generate token sequences auto-regressively. Though we have introduced a bit more redundancy in our mesh representation, our method enjoys a much simpler architecture design and better supports large-scale training.
3. Data Processing. The data pre-processing in our MeshXL is also much simpler as we are only required to permute 3D faces into a coordinate sequence. Our simplified data pre-processing increases the total throughput to further support large-scale training. Meanwhile, MeshGPT requires to build a face graph with respect to face connectivity within the 3D meshes to extract face and vertices features.

📝 W2: Additional metrics for mesh quality assessment.

💡 A: We will add more metrics for better mesh quality assessment in the revision.

1. How well is the triangulation. Following suggestions from reviewer XQPQ, we evaluate the aspect ratio, face area, and number of faces for a better evaluation in the following table. Though the meshes generated by our MeshXL have a higher average aspect ratio, we achieve a smaller variance with much less 3D faces. This indicates the stability of our generation ability and the efficiency of the direct mesh representation. Since we train our MeshXLs only on triangular meshes, long-thin triangles inevitably exist in our training data. In future works, we will co-train our MeshXLs on triangular meshes, 3D quads, and even hybrid representations to reduce the existence of long thin triangles for better generation quality. | Method | Aspect Ratio | | | Face Area | | | Number of Faces| | |-|-|-|-|-|-|-|-|-| | | mean | std. | | mean | std. | | mean | std. | | GET3D | 6.27 | 116.03 | | 0.000 | 0.000 | | 27251.80 | 11535.135 | | MeshXL - 125m | 10.47 | 16.88 | | 0.031 | 0.096 | | 327.34 | 174.53 | | MeshXL - 350m | 10.25 | 16.09 | | 0.032 | 0.099 | | 342.24 | 193.97 | | MeshXL - 1.3b | 10.23 | 15.91 | | 0.034 | 0.102 | | 320.36 | 195.43 |

2. Watertight meshes. A watertight mesh does not have any boundary edges. Therefore, it is debatabe whether we should generate watertight meshes in 3D assets generation. Currently, watertightness is mainly required to perform physical simulation or turn 3D meshes into implicit field and distant functions. However, in 3D assets generation, many common 3D shapes including cloth, terrain, and leaves are not watertight. Furthermore, it is also challenging and inaccurate to specify the interior of a 3D mesh for an reliable evaluation. Therefore, our method mainly focus on establishing a more direct representation for 3D mesh generation.

📝 Q1: Question about the ordering.

💡 A: We will cite PolyGen in Line 103, and clarify the relation between our mesh representations and PolyGen's in section 3 .

1. We adopt the same ordering system as PolyGen and MeshGPT, which first permutes the vertices within each face cyclically based their coordinates in z-y-x order (from lower to higher), then permutes the faces based on the permuted coordinates from lower to higher. Since there are no identical polygons in a 3D mesh, the ordering strategy will create one unique sequence for each 3D mesh.
2. Relation to the ordering in PolyGen. We adopt the same vertex representation as PolyGen, but a different mesh representation. Our MeshXL and PolyGen both represent vertices with discrete coordinates. However, MeshXL represents a 3D mesh only with an ordered coordinate sequence, while PolyGen decouples the vertices and polygons in 3D meshes, and represents each 3D mesh with two sequences, i.e., the vertex sequence and an index sequence to connect the generated vertices. Therefore, our coordinate sequence representation enables us to train an end-to-end model directly on large-scale 3D data.

📝 Q2: Clarifying the claims.

💡 A: We will clarify our claims and motivations in our revision.

1. We will modify the claim into "comparing to the indirect way of mesh generation, our end-to-end pipeline better supports large-scale pre-training, especially considering the training and data processing efficiency". We will also do further study to explore better auto-regressive pipelines.
2. In our main paper, our intended idea is that the 3D mesh representation has great flexibility that can represent flat surfaces with much less data (i.e. two triangles for a rectangle surface vs lots of points and voxels) and preserve details with more faces in curved surfaces. We will clarify this in our revision.

📝 Q3: The definition of "polynomial face".

💡 A: In our paper, a k-sided polynomial face is a polygon with k vertices and k lines. For example, triangles and quadrilaterals are special cases with three and four sides, respectively. We will clarify this in our revision. We have also open-sourced our code to help readers better understand our method.

We sincerely appreciate your recognition of our work and the valuable feedback to help us improve our paper.

In our revision, we will highlight that our work paves the way for scaling up training on large-scale 3D mesh data. Our mesh representation turns a 3D mesh into one unique coordinate sequence, which enables us to simplify our architecture design into a decoder-only transformer model, facilitating an end-to-end training pipeline that does not require sophisticated data pre-processing, careful model design, or complex training strategy and better suits large-scale 3D mesh data.

To further improve our paper, we will include a detailed discussion between MeshXL's mesh representation and that of PolyGen. Additionally, we will also incorporate objective evaluations on the triangulations for mesh quality assessment.

We acknowledge that the potential generation of certain artifacts is a limitation for now. To alleviate the potential occurrence of artifacts, we will keep exploring methods to integrate domain knowledge as filters or reward models. We will also work on co-training MeshXL on triangle meshes, 3D quads, and even hybrid representations to reduce the occurrence of long thin triangles as also mentioned by reviewer XQPQ.

Once again, we thank you very much for your recognition and valuable suggestions from all reviewers to help us improve our work.

📝 W1: The absence of domain knowledge to prevent potential artifacts.

💡 A: In our work, we put emphasis on exploring a sequential way to model 3D meshes that better suits large-scale generative training on extensive 3D data. Therefore, the potential generation of surface artifacts is currently the limitation for our method, as well as PolyGen and MeshGPT. However, our MeshXL has the potential to incorporate with traditional methods by treating the domain knowledge as filters or even reward models to eliminate certain artifacts.

📝 W2: Accessing meshes with additional objective metrics.

💡 A: We will add more objective metrics in our main paper. We calculate the face area and the aspect ratio with respect to the definition: $\text{Aspect Ratio} = \frac{\text{longest edge}}{\text{shortest altitude}}$. Since PolyGen generates polygon meshes rather than triangle meshes, we could not calculate the aspect ratio for PolyGen. From the below table, though GET3D achieves a lower average aspect ratio, it suffers from a higher variance with tens of thousands of faces. Meanwhile, MeshXL achieves a much stable aspect ratio with a larger average face area, indicating that the our MeshXL has the stability to generate high-quality 3D meshes.

How to alleviate long thin triangles. Currently, we train our MeshXL on 3D triangular mesh data. Therefore, long thin triangles inevitably exist in our ground truth training data, for example, the legs of tables or chairs. To alleviate the generation of long thin triangles, we will train our method on 3D quads in the future, which has been proved to be an efficient and common mesh representation in the 3D design industry, and could avoid the generation of long thin triangles.

📝 Q1: Addressing potential artifacts in the generated meshes.

💡 A: See Weakness 1.

📝 Q2: The sequential mesh representations.

💡 A: We will clarify the sequential mesh representations in the revision.

1. We train only on triangular meshes in our paper. In order to accelerate the data processing and training procedure, we choose an easier setting by turning all 3D meshes into triangular meshes to validate that the next-coordinate generation is capable of high-quality 3D mesh generation. Thus, the output of our method is always represented with $\text{<tri>} (x,y,z),(x,y,z),(x,y,z); ... \text{</tri>}$.
2. We can easily extend MeshXL to hybrid representations. In our paper, we present “<tri> · · · </tri>” and “<quad> · · · </quad>” as a potential that our method can be extended to both triangular meshes, 3D quads, and even the hybrid representation. We will further extend our work on the hybrid representation to alleviate the existence of long thin triangles as mentioned in weakness 2.
3. If there are both triangles and quadrilaterals in a 3D mesh, our method also follows the ordering strategy introduced in line 111 - 114, which first permutes the vertices within each face cyclically based their coordinates (z-y-x order, from lower to higher), and then order the faces based on the permuted coordinates from lower to higher. Therefore, a 3D mesh with both triangles and quadrilaterals can also be represented by one unique sequence. Thus, even with the hybrid representations, our MeshXL can generate these 3D meshes in one sequence.