ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

We thank the reviewers for the constructive reviews. We provide our feedbacks as follows.

Weaknesses1

A1: Based on the pretrained weights, we conducted a small-scale fine-tuning experiment specifically targeting text-to-3D generation. During fine-tuning, we removed redundant prompt tokens. Additionally, we froze the gradients of all text token embeddings except for the mesh tokens. The learning rate was set to 1e-5, with a context length of 1536, batch size of 4, and a gradient accumulation step of 2. The fine-tuning lasted for 5 epochs. After fine-tuning, the model lost general text processing ability but improved significantly on text-to-3D tasks, achieving results comparable to Trellis on our test dataset. This shows the model can gain on a specific task by sacrificing general capabilities. We think Shapellm-4o did not outperform Trellis because diffusion/flow models currently work better than autoregressive models for visual generation. Beating diffusion/flow with autoregressive models is a major challenge. Our work focuses not on maximizing 3D generation performance but on creating a unified model that can both understand and generate 3D content autoregressively. This is a key innovation. As autoregressive visual generation advances, our unified model’s performance will also improve.

Weaknesses2

A2: We conduct additional experiments to evaluate our model on 3D question-answering tasks. As shown in the table below, the best scores for each metric in the table are highlighted. Our model achieves consistently superior performance across all methods, demonstrating its strong capabilities in 3D QA tasks.

Weaknesses3

A3: We used the GPT-Image-1 API for generating and editing the data, selecting the "low" quality option. The average cost per image editing pair was around 0.01, with a total expenditure of approximately 750. After constructing the image editing pairs, we used Trellis for 3D reconstruction of the edited images. On average, the time to generate one asset was 5 seconds, and the total GPU time consumed was about 100 hours, which equals approximately 200. Therefore, the total cost for the entire process amounted to around 950.

Weaknesses4

A4: We apologize for the confusion. As described in Section 3.4 of the paper, we initially constructed 70K 3D editing text-image pairs. Based on these, we generated a total of 420K multi-turn dialogue samples following specific rules. Specifically, for each 3D editing pair, we created three dialogue types: (1) pure 3D editing, (2) image-to-3D followed by 3D editing, and (3) text-to-3D followed by 3D editing—yielding approximately 210K dialogues. To further enrich the diversity and support multi-task interleaving, we inserted additional pure text instructions into these dialogues, ultimately expanding the dataset to 420K dialogue samples. We will revise the text in the ‘Limitations’ section to clearly reflect that 70K is the number of base 3D editing pairs, while 420K refers to the full dialogue dataset derived from them.

We thank the reviewers for the constructive reviews. We provide our feedbacks as follows.

Weaknesses1

A1: While prior works such as Emu3 have demonstrated the effectiveness of using a shared tokenizer for both understanding and generation in the 2D vision domain, extending this paradigm to 3D data is far from straightforward. In general, mapping modality-specific features into a unified language space is a powerful and broadly applicable idea—one that spans across modalities from text to vision. However, each modality presents its own challenges. In the case of 3D, the representation is structurally richer and requires maintaining spatial integrity and multi-view consistency, which makes joint modeling significantly more difficult than in 2D. Our contribution lies in successfully adapting this framework to the 3D object domain by employing a new voxel-based VQ-VAE, and empirically validating its effectiveness on both generation and understanding tasks. We appreciate the reviewer’s insight and will incorporate this discussion into the related work section to better contextualize the novelty of our approach.

Weaknesses2

A2: We sincerely thank the reviewer for the careful observation. The inconsistency arises primarily form differencies in data preprocessing: our original setting does not normalize the object occupancy within the input image, whereas Trellis resizes the object to occupy a fixed portion. This discrepancy results in an unfair comparison. After aligning with Trellis, our results demonstrate significantly improved visual consistency in the mentioned failure examples. We will include this updated setting and analysis in the final version to ensure a fair and rigorous comparison.

Weaknesses3

A3: We conduct additional 3D understanding comparisons with recent and stronger methods, including ShapeLLM and GPT4Point. As shown in the table below, the best and second-best results for each metric are marked in bold and underline, respectively. Our method consistently outperforms these baselines, achieving state-of-the-art performance in 3D understanding tasks.

Questions1

A4: We thank the reviewer for this valuable suggestion. In our current pipeline, we first edit single-view image using GPT-4o, then perform image to 3D generation using Trellis. We agree that incorporating view-consistent editing and leveraging more large 3D reconstruction models could further improve the editing fidelity and geometric accuracy of the final 3D shapes. We will serve this suggestion as future direction in the conclusion section, and explore it in our future work.

Questions2

A5: We have provided a preliminary discussion on the performance gap in Section 4.2, specifically in the third paragraph under the subsection "Compared with Trellis." We believe the performance gap between our model and Trellis can be attributed to two main reasons:

Reason 1: Our model integrates six tasks (3D generation from images, 3D generation from text, 3D understanding, 3D editing, image understanding, and text reasoning) into a single model. In contrast, Trellis separates the tasks of 3D generation from images and 3D generation from text into two distinct models, each trained independently. The multi-task integration is inherently more challenging than training a single-task model. To demonstrate this, we conducted a small-scale fine-tuning experiment specifically for text-to-3D generation, using the existing pre-trained weights. During training, we removed extraneous prompt words, retaining only the basic prompt, and froze the gradients of the textual tokens corresponding to embeddings other than the mesh token. The hyperparameters used were: learning rate of 1e-5, context size of 1536, batch size of 4, and gradient accumulation of 2. We continued fine-tuning for 5 epochs. After fine-tuning, the model lost its ability to generate normal text outputs; however, its ability for text-to-3D generation improved significantly, yielding results on the test dataset comparable to those of Trellis. This experiment shows that our model, by sacrificing other capabilities, can achieve further breakthroughs in a single task.

Reason 2: We believe a key factor preventing ShapeLLm-4o from surpassing Trellis is the inherent advantage of diffusion/flow models over autoregressive architectures in visual content generation. The diffusion/flow models currently outperform autoregressive models in visual generation tasks. Overcoming this gap, where autoregressive architectures can outperform diffusion/flow models, remains a significant challenge in the academic community. However, the focus of our work is not to explore the upper bounds of autoregressive architectures for 3D generation but to enable unified generation and understanding in a model. Our goal is for LLMs to successfully leverage autoregressive paradigms to both understand and generate 3D content. This is a significant innovation in the field. Moreover, autoregressive visual generation is a highly active area of research, and the performance of the unified model architecture we propose will only improve as advances in autoregressive visual generation continue.

Questions3

A6: The main difference between our work and Trellis is not just swapping the flow transformer with a VLM backbone. There are two key distinctions: Generative Modeling Approach: Trellis uses a flow-based model, while our approach relies on an autoregressive model. These two methods are fundamentally different in how they generate data. Discrete vs. Continuous Representation: Trellis uses continuous latent spaces, whereas our model works with discrete tokens. This affects how the 3D content is represented and generated. So, the architectural difference is much more than just the backbone change — it's about the overall generative modeling approach and representation type.

We thank the reviewers for the constructive reviews. We provide our feedbacks as follows.

Weaknesses1

A1: We emphasize that 3D object-level generation and understanding tasks are fundamentally different from scene-level ones in terms of objectives, input modalities, and methods. Due to this gap, models designed for object-level tasks often cannot be directly applied to scene-level benchmarks, and vice versa. This distinction is well acknowledged in the community—existing models such as PointLLM, GPT4Point, Trellis, and ShapeLLM are all tailored specifically for object-level tasks. This is not a limitation of the algorithms themselves, but a reflection of the fundamental differences between the two problem settings. Therefore, existing benchmarks designed for 3D scene understanding are not directly applicable to our method and do not offer a fair comparison. Moreover, object-level tasks serve as the foundation for 3D generation and understanding, since the ability to accurately generate and interpret 3D objects is critical for applications in robotics, virtual reality, gaming, and other domains. Robust object-level modeling is necessary for scaling to more complex 3D scene-level tasks. We contend that ​the practical realism and methodological challenges inherent in object-level tasks are commensurate with those of 3D scene-level tasks.​ Despite the importance, research combining unified autoregressive models with 3D object generation and understanding is few. Our work focuses on object-level tasks, marking an important step toward unifying 3D object generation and understanding within a generative modeling framework.

Weaknesses2

A2: Our proposed unified autoregressive framework and VQ-VAE are specifically designed for object-level tasks, with tailored methodologies, data structures, and objectives not applicable to scene-level tasks. Therefore, directly employing existing scene-level 3D benchmarks to evaluate our framework is not only inappropriate but also fails to reflect the scope and advantages of our approach. Just as Trellis cannot be expected to generate complex scenes, our object-level trained model fundamentally lacks the architectural capacity for scene generation and understanding.

Weaknesses3

A3: While our data construction is GPT-assisted, as in prior works, our main contribution lies not in proposing a novel data generation pipeline, but in constructing a critical but underexplored dataset. Specifically, Grounded 3D-LLM emphasizes scene-level understanding, whereas our dataset targets object-level editing. To the best of our knowledge, object-centric 3D editing data remains extremely scarce in existing work. We believe our dataset fills this important gap and offers a valuable resource for advancing research in controllable 3D generation and editing.

Weaknesses4

A4: To quantitatively assess the quality of our GPT-annotated text-to-3D dataset, we randomly sample 1000 text–image pairs and evaluate their semantic alignment using CLIPScore and ViLT R-Precision. As shown in the Table 1, our dataset achieves high scores across both metrics, indicating strong correspondence between the captions and 3D content.

Similarly, to evaluate the alignment in 3D edting dataset, we compute the same metrics between the editing prompts and the rendered front-view of edited objects. The results, summarized in Table 2, indicate the editing prompts are well aligned with the resulting modifications.

Weaknesses5

A5: Our framework jointly addresses multiple tasks—including 3D generation, understanding, and language capabilities—which inherently increases the training complexity and resource requirements. In particular, maintaining strong language capabilities while integrating diverse 3D tasks demands substantial computation. We verified it by conducting a fine-tuning experiment specifically targeting text-to-3D generation, based on the pretrained weights. During fine-tuning, we removed redundant prompt tokens. Additionally, we froze the gradients of all text token embeddings except for the mesh tokens. The learning rate was set to 1e-5, with a context length of 1536, batch size of 4, and a gradient accumulation step of 2. The fine-tuning lasted for 5 epochs. After fine-tuning, the model lost general text processing ability but improved significantly on text-to-3D tasks, achieving better results on our test dataset. This shows the model can gain on a specific task by sacrificing general capabilities.

The core focus of ShapeLLM-4o, as indicated by its name, is on object-level generation, editing, and understanding tasks. Scene understanding is neither the target of our research nor falls within the scope of our technical contributions. We argue that it is unreasonable and non-comparable for reviewers to request a comparison between ShapeLLM-4o and models specifically designed for scene understanding, for three main reasons:

1. Analysis of Training Data Composition: Object-level understanding datasets primarily consist of dialogue data focused on the object itself, emphasizing the capture of the object's geometric shape and detailed features. In contrast, scene-level task datasets focus on the overall scene, with the core focus being on the spatial relationships between objects. Although scene datasets can provide basic object descriptions, the granularity of these descriptions is insufficient for fine-grained tasks and cannot support detailed object-level question-answering tasks. Therefore, requesting the direct application of a model trained on object understanding datasets to be tested on scene-level datasets introduces a significant unfairness in evaluation. To validate this claim, we specifically tested the performance of Chat-Scene on object-level tasks—using the same experimental setup as PointLLM. The results showed a clear deficiency in its object understanding capabilities (see the table below for specific values). To ensure the fairness of the experiment, I sought confirmation from the original Chat-Scene author team through an anonymous channel, and they explicitly stated that the model's inability to transfer to object-level tasks was an expected outcome, with the underlying reason being the inherent differences in training data. This conclusion aligns perfectly with the core argument I made earlier. Existing research data suggests that similar works, such as PointLLM, ShapeLLM, and GPT4Point, focus on a single domain (scene or object) and have never been asked to balance both capabilities. Based on the above empirical analysis and consensus in the field, I must raise a clear objection to this evaluation method. || B-1↑ | R-L↑ | M↑ | S-BERT↑ | S-CSE↑ | |:-----:|:------:|:----:|:-----:|:------:|:------:| | Chat-Scene|7.1|7.8|8.0|18.9|16.4|

2. Analysis from a Technical Implementation Perspective: Our model architecture and hyperparameter design are deeply optimized for object-level tasks. A typical example of this is the use of a 64-resolution voxel representation to balance fine-grained features with computational efficiency, along with the allocation of a 1024-length serialized encoding for each voxel. These design choices are primarily aimed at maximizing object performance. Directly transferring such specialized configurations to scene-level task testing would introduce evaluation bias due to the lack of domain adaptability.

3. However, our model architecture exhibits outstanding transfer and scalability capabilities, requiring only minimal adjustments to seamlessly support scene-level tasks. To validate this characteristic, we propose three simple and direct transfer strategies:

Strategy 1: Independently encode each object in the scene into a 1024 sequence, and increase the inference batch size. After completing parallel captioning, the model output can be queried using the user-provided description.

Strategy 2: Encode the scene as a voxel sequence and concatenate it with multiple views of the scene, directly inputting this into the model.

Strategy 3: We can increase the compression rate of the current VQ-VAE, reducing the sequence length to 128 in order to accommodate all the objects in the scene. Additionally, we can assign the coordinates of each object's bounding box after each object.

The three strategies outlined above will be explored in greater depth as part of our future work. It is important to emphasize once again that our model architecture has substantial value, enabling it to be seamlessly extended to scene understanding tasks.

Understanding tasks represent just one dimension of this study's contributions. Our core innovation lies in the unprecedented unification of four major task frameworks: text-to-3D generation, image-to-3D generation, 3D object understanding, and 3D object editing. After in-depth discussions with the reviewers, we also plan to integrate 3D scene understanding into our framework in the future. We kindly request that the reviewers fully acknowledge the systematic breakthroughs our research brings to the field of cross-modal 3D generation and understanding, while understanding that scene understanding is not the core focus of this paper.

Finally, we appreciate the reviewers' feedback, and we will incorporate this discussion into the main body of the text. Additionally, we will include relevant works (Chat-Scene, Grounded 3D-LLM, Robin3D, Chat-3D) in the related work section.

We thank the reviewers for the constructive reviews. We provide our feedbacks as follows.

Weaknesses1

A1: To assess the reconstruction quality of our 3D VQ-VAE model, we randomly select 1000 samples from the test set and feed them into the model. We then calculate several metrics between original and reconstructed voxel grids, including IoU, Recall, Precision, F1 and Chamfer Distance. These results, summarized in the table below, indicate that our 3D VQ-VAE model preserves geometric structure with high fidelity, providing a reliable reconstruction basis for following generation tasks.

Weaknesses2

A2: To provide a more comprehensive comparison, we quantitatively evaluate the image to 3D generation performance of OpenLRM, LGM, InstantMesh, and Unique3D on the same test set used in our paper. We also adopt the same evaluation metrics, including CLIPScore, FD_{incept}, and KD_{incept}. As shown in the table below, our method consistently outperforms all baselines and demonstrates superior 3D generation quality.

Weaknesses3

A3: Our current voxel-based representation provides effective structural alignment, enabling efficient tokenization with 3D VQVAE and seamless integration with autoregressive models. However, it does come with certain limitations. Specifically, voxel representations incur high memory and computational costs at higher spatial resolutions. Moreover, voxel-based representations are unable to model textures, and the use of low-resolution voxels limits ShapeLLM-4o's ability to understand 3D assets at a finer level of granularity. In future work, we aim to explore more efficient and finer-grained 3D representation methods.

Questions1

A4: Thank you for your valuable suggestion. We will add some discussion on this section in the main text. In fact, we have considered other encoding methods. For meshtron, encoding mesh into discrete tokens through spatial quantization can result in excessively long sequences, which is not conducive to training autoregressive models. For continuous features represented by vecsets, we believe that on the one hand, their information content will be relatively rich, and it may be difficult to train a complete 3D vqvae for encoding. On the other hand, vecsets are implicit features, while voxels are explicit features, and 3D editing may require more explicit features. Considering downstream tasks, prevalent approaches (e.g., Trellis) increasingly adopt coarse-to-fine generation frameworks. Such pipelines typically first generate coarse voxel structures followed by high-resolution mesh refinement. Consequently, leveraging voxel representations enables ShapeLLM-4o to seamlessly integrate with contemporary 3D generation workflows, establishing greater generalizability.

Limitations

To quantitatively assess the quality of our GPT-annotated text-to-3D dataset, we randomly sample 1000 text–image pairs and evaluate their semantic alignment using CLIPScore and ViLT R-Precision. As shown in the Table below, our dataset achieves high scores across both metrics, indicating strong correspondence between the captions and 3D content.

Similarly, to evaluate the alignment in 3D edting dataset, we compute the same metrics between the editing prompts and the rendered front-view of edited objects. The results, summarized in Table below, indicate the editing prompts are well aligned with the resulting modifications.