3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination

1. LEO [1] and SceneVerse [2] have proposed the scene graph-based data generation pipeline. The pipeline's technical contribution is very minor compared to previous work.
2. No existing 3D VLM is trained based on 3D-GRAND dataset. The author only mentions that they train 3D-GRAND models based on a specific subset of 3D-FRONT in the appendix. There should be 3D VLMs(existing models such as 3D-LLM, LEO, etc.) trained based on 3D-GRAND as evidence of the quality of the dataset. This results in poor integrity of the article and greatly reduces the contribution of the article.
3. Unfair comparison with the baseline models. The proposed baseline model 3D-GRAND takes the object locations and attributes as the input, which provides strong information on the whole 3D scene without any noise. While other models such as 3D-LLM, 3D-Vista, and LEO take the 3D point cloud or multi-view images as input.
4. The training set of the baseline model 3D-GRAND should be presented in section 3 instead of in the appendix.

[1] An Embodied Generalist Agent in 3D World [2] SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding

1. While 3D-GRAND demonstrates early signs of sim-to-real transfer, it relies heavily on synthetic data from the 3D-FRONT and Structured3D datasets. There’s limited analysis on potential biases that might arise from this reliance, especially since synthetic scenes can differ significantly from real-world 3D environments in object layout and scene composition. This limitation could affect the model’s performance on real-world tasks where variability is higher.
2. Although the paper introduces a hallucination filter, the methodology behind it is not fully explained. For instance, it’s unclear how specific filtering criteria are determined, or how consistently the filter distinguishes valid object references from hallucinated ones. This could lead to inconsistencies in the data and, consequently, model performance. A more transparent breakdown of the filter’s effectiveness, or examples of common hallucination cases, would enhance confidence in the filtering approach.
3. Dense grounding annotations at the noun-phrase level are presented as beneficial, but the paper does not discuss potential downsides, such as increased model complexity or the risk of overfitting to these annotations. It’s also unclear whether dense grounding provides a significant advantage over sparse grounding for all task types or only specific ones. A comparison with sparse grounding on certain benchmarks could clarify this point.

• The benchmark primarily focuses on binary existence questions, lacking evaluation on more complex tasks such as spatial reasoning, attribute-based grounding, and relational understanding, which are essential for fully assessing a model’s hallucination in 3D comprehension.
• The proportion of data generated via templates versus LLM-generated annotations is unclear, and the paper does not provide specific examples of the templates used.
• How does the paper handle metric calculation in cases where objects in the model output are ungrounded? For example, if the model describes a “blue chair” that does not exist in the scene or fails to reference an existing object, how are grounding accuracy and hallucination rates calculated in such instances?
• In addition to grounding experiments, the paper could benefit from including language tokens prediction experiments, which would further demonstrate the dataset's effectiveness in training language understanding components of 3D-LLMs.
• The model design is relatively straightforward, relying on scene graphs to process 3D scenes and output JSON representations. This approach lacks advanced reasoning capabilities, which could limit the model’s ability to handle tasks requiring deeper understanding and inference in 3D environments.
• The figures in the paper need to be refined to improve their quality.