Cue3D: Quantifying the Role of Image Cues in Single-Image 3D Generation

Thank you for your informative and thoughtful review!

Q1a: Zeroverse domain gap: GSO and Toy4K are based on natural/real-world objects and shapes, with internal symmetries and are relatively "smooth", Zeroverse is not; Model has not see Zeroverse data.

A1a: Thanks for pointing this out. We agree with the reviewer’s observation. We will first describe our motivation of using Zeroverse, and then present an alternative quantitative experiment shape cutmix. This experiment validates that when we (i) reduce the domain gap, (ii) preserve more shape cues like symmetry and smoothness, (iii) on data similar to the training data domain, shape meaningfulness remains an impactful cue.

Our motivation for using zeroverse is because it completely destroys the shape meaningfulness cue. It could be seen as an upper-bound experiment: if the model reasons about 3D shape in a completely geometric way and does not require shape to be meaningful, it should still be able to generate Zeroverse objects. We do agree we should have more experiments on the middle ground: we try to reduce the domain gap and preserve more shape cues when weakening shape meaningfulness.

We introduce several different settings for our CutMix experiments:

1. Half-and-half: We mix half of mesh M with the other half of mesh N to construct a new mesh from GSO meshes. This limits the distribution shift and preserves many local and global shape cues (e.g., surface smoothness, local symmetry), and also preserves a significant amount of shape meaningfulness to human perception. We show 3 variants: front-back, left-right, and top-bottom.

2. Standard cube-based CutMix. We follow the CutMix paper and randomly sample an axis-aligned 3D cube within the bounding cube of the object. We replace the part of mesh M that falls into the cube with the part from mesh N that falls into the same cube. When sampling the cube, we pin one of its corners at the corner of the object bounding cube to avoid significant discontinuity in the output shape. The length ratio(length_{sampled cube}/length_{bounding cube}) is uniformly sampled from [0.4, 0.6]. Most parts of the object M are outside the chosen cube and remain intact. Meanwhile, the local shape cues are mostly preserved.

3. OctantMix. We center each mesh and split it into 8 octants by the coordinate planes (xy, yz, and xz planes). Then we replace the part in each octant by the corresponding part from other random meshes from the same dataset. This variant still preserves the local shape cues, and it has a significantly smaller distribution gap than zeroverse compared to our original evaluation data (GSO).

We show the result on the GSO dataset, compared to Zeroverse numbers for reference. We use the CDx1000 metric following Table 3, lower is better.

The results show that all variants still significantly harms performance. In particular, despite a significantly smaller domain gap to GSO, OctantMix leads to a similar performance drop as zeroverse, indicating catastrophic failure. Standard CutMix only alters approximately 1/8 of the mesh volume, but it also causes a very significant drop(e.g., 20 points for Hunyuan3D-2). Even for the minimal half-and-half perturbations, where shape meaningfulness still largely remains, we still observe significant performance drop of >10 points most of the time. This performance drop is still very significant compared to the effect of other cues. These results highlight the importance of shape meaningfulness, since the models have significant difficulty to generalize even when we only partially perturb this cue, especially compared to other cues like texture meaningfulness.

Q1b: Some models are based on text-to-image models, some are trained on text prompts. Text signal corresponding to Zeroverse is unseen.

A1b: We appreciate your comment and want to clarify a potential misunderstanding: we did not use text as input in our experiments, and none of the models have directly seen paired text and 3D assets. Regarding Trellis specifically, we used its image-to-3D checkpoint, which indeed was not trained on any text. You are correct that Multi-view diffusion (MVD)-based models (LGM, CRM, InstantMesh) starts from text-to-image models before further training on Objaverse, while the other models haven't seen any text during training. While text-based pretraining might influence the capabilities of the three MVD-based models, we believe this is not a major concern because both text-pretrained models and those without text pretraining exhibited similarly significant performance drops under shape meaningfulness perturbation. Since our main focus is on image-to-3D modeling, we believe exploring text-based generalization would be valuable future work.

Q2: Paradigm-wise sensitivity to cues and why; Percentage of degradation A2: Thank you very much for this helpful suggestion. We completely agree that grouping the models by paradigm and quantifying their relative sensitivity helps clarify which cues each family relies on most. Below, we summarize our current results paradigm-wise using percentages and provide additional interpretation. The following table shows the percentage of performance change on the GSO dataset. Since directly computing percentages on Chamfer Distance is not meaningful, we first calculate an F1 score at a Chamfer Distance threshold following SF3D [1], and then compute the percentage change based on this F1 score:

It's worth noting that no metric is perfectly linear with respect to model performance, so direct percentage comparisons might not be entirely fair between higher- and lower-performing models. Nonetheless, we can still draw several interesting conclusions. For instance, multi-view models seem less affected by shading due to their rich pretraining on text-to-image data. Native 3D models appear more robust to silhouette perturbations, possibly due to their generative training objectives. We'll include a more detailed discussion of these insights in the final version of the paper.

Q3: More discussion on how proposed insights can contribute to new image-to-3D models.

A3: We sincerely appreciate this question, and we fully understand your intention to help improve our paper. We agree that illustrating how our conclusions can inspire future research is valuable. Due to limited time and resources, we were not able to retrain or fine-tune multiple models. However, we made an effort to illustrate how our insights could inspire new research directions. Specifically, we explored a line-art-to-3D application, where the input is a simplified line-art image (in our case, extracted from GSO images), and the goal is to recover the underlying 3D structure. Below, we report results in terms of Chamfer Distance x 1000 following Table 3 (lower values indicate better performance):

Our Cue3D analysis emphasizes that semantics alone are not sufficient and highlights the importance of geometric cues (L57). This point is especially relevant in the line-art-to-3D scenario, as line-art inherently lacks detailed surface properties to inform 3D reconstruction. Directly using line-art images results in significantly worse 3D reconstructions than original images. Inspired by our analysis, we leveraged an image diffusion model (Flux ControlNet) conditioned on line-art, prompting it to generate geometric cues (e.g., shading and texture) through prompting for 3D rendering-style outputs. This approach significantly improved performance, validating that our proposed insights could meaningfully contribute to future research in image-to-3D.

Q4: Occlusion experiment clarification; Occluded version could be a valid object; Fine-tuning a model using occluded objects.

A4: Thanks for bringing up this important point. To clarify, all models evaluated take masks as input and have been trained specifically on image+mask combinations. Your suggestion that occluded versions of objects could sometimes represent valid objects is indeed insightful, as occlusion naturally introduces ambiguity. However, our primary objective is to evaluate whether the models have learned a robust prior distribution of plausible 3D shapes. Ideally, models should reconstruct shapes that align with their learned 3D priors, despite ambiguity from occlusions. Additionally, we fully acknowledge that fine-tuning on occluded images could further improve performance. To explore this, we're currently evaluating Amodal3R [2], a version of Trellis fine-tuned explicitly on occluded images, on our occlusion perturbation. We will include these results as soon as the evaluation is complete.

Reference

[1] Boss et. al., SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement, 2025

[2] Wu et. al., Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images, 2025

Response to Reviewer bRRe:

Thank you so much for your thoughtful and helpful comments!

Q1: How do the cues interact with each other in more complex ways?

A1: This is a really interesting point. The main difficulty in exploring more complex interactions between cues is that certain cues(e.g., texture and style) are inherently difficult to be combined in a meaningful way. We initially selected these cues because of their perceptual importance and interpretability to humans, rather than strict orthogonality. However, we completely agree that exploring these more nuanced interactions is a great direction for future research!

Q2: Are there other important cues?

A2: Great question! Yes, there are other important cues worth investigating. For example, as Reviewer MKBa also mentioned, material properties (like specularity and roughness), variations in lighting conditions, and viewing angles could all provide interesting insights. We added an initial experiment exploring the interaction between shading and material properties (e.g., specularity). Specifically, we conduct this additional experiment on the GSO dataset and report results in CDx1000 following Table 3 (lower is better). We investigated combinations of environment map lighting vs. directional lighting, and default vs. large specularity.

This experiment yields interesting observations! Surprisingly, directional lighting significantly degraded performance for most models, except Hunyuan3D-2. Additionally, large specularity slightly improved performance in several cases, although not always, with the exception of OpenLRM and Hunyuan3D-2. We speculate that this might be related to the different lighting setup used in each model’s training.

One major benefit of Cue3D is that it is easily extensible. We expect more cues like this could be integrated in the future into our Cue3D framework.

Q3: How generalizable are the findings? Test the robustness of the conclusion.

A3: Thank you for raising this important point! We are quite confident that our findings are generalizable, since our findings generalizes across scanned real-world object dataset (GSO) and synthetic 3D assets dataset (Toys4K). We will also include a figure that analyzes per-category performance impact on Toys4K in the final version of the paper. Our conclusions are consistent across these different datasets and categories. Regarding the robustness, we include Appendix Table 4 to illustrate that our conclusions are robust to random seed. Meanwhile, also in response to Reviewer wyxx Q2, we will include statistical tests(specifically, Wilcoxon signed-rank tests) to clearly demonstrate that the observed drops in performance metrics are statistically significant.

We sincerely appreciate your valuable review. If you have any further comments or questions, we would be happy to discuss further!

Thank you for your insightful comments!

Q1: Whether style variations are meaningful.

A1: We appreciate your thoughtful observation! We used a state-of-the-art style transfer model, but we acknowledge it is not perfect. While it successfully applies the desired style in many cases, there are scenarios where it does not fully capture the reference style. However, our goal for using style transfer is to perturb geometric cues while preserving semantic content (L175). Even when the style transfer is not perfect, it still generally achieves our goal of introducing meaningful perturbations.

Q2: More cues and transforms; Effect of varying viewing angle.

A2: Thanks you for the suggestions! Given the limited time and resources during the rebuttal period, we prioritized performing a more in-depth experiment exploring the interaction between shading and material properties (e.g., specularity). Specifically, we conducted this additional experiment on the GSO dataset and report results in CDx1000 following Table 3 (lower is better). We investigated combinations of environment map lighting vs. directional lighting, and default vs. large specularity.

This experiment yields interesting observations! Surprisingly, directional lighting significantly degraded performance for most models, except Hunyuan3D-2. Additionally, large specularity slightly improved performance in several cases, although not always, with the exception of OpenLRM and Hunyuan3D-2. We speculate that this might be related to difference lighting setup used in each model’s training. Meanwhile, investigating the effect of varying viewing angles would be valuable, and we believe this is a promising direction for future research.

Q3: Zeroverse comparison may be unfair due to difficulty.

A3: We appreciate the reviewer’s observation. We additionally conducted several shape CutMix experiments, which mixes different assets from the GSO dataset to create new meshes, with the objective of partially removing shape meaningfulness while maintaining a similar difficulty. We introduce several different settings for our CutMix experiments:

1. Half-and-half: We mix half of mesh M with the other half of mesh N to construct a new mesh from GSO meshes. This limits the distribution shift and preserves many local and global shape cues (e.g., surface smoothness, local symmetry), and also preserves a significant amount of shape meaningfulness to human perception. We show 3 variants: front-back, left-right, and top-bottom.

2. Standard cube-based CutMix. We follow the CutMix paper and randomly sample an axis-aligned 3D cube within the bounding cube of the object. We replace the part of mesh M that falls into the cube with the part from mesh N that falls into the same cube. When sampling the cube, we pin one of its corners at the corner of the object bounding cube to avoid significant discontinuity in the output shape. The length ratio(length_{sampled cube}/length_{bounding cube}) is uniformly sampled from [0.4, 0.6]. Most parts of the object M are outside the chosen cube and remain intact. Meanwhile, the local shape cues are mostly preserved.

3. OctantMix. We center each mesh and split it into 8 octants by the coordinate planes (xy, yz, and xz planes). Then we replace the part in each octant by the corresponding part from other random meshes from the same dataset. This variant still preserves the local shape cues, and it has a significantly smaller distribution gap than zeroverse compared to our original evaluation data (GSO).

We show the result on the GSO dataset, compared to Zeroverse numbers for reference. We use the CDx1000 metric following Table 3, lower is better.

The results show that all variants still significantly harms performance. In particular, despite a significantly smaller domain gap and a more similar difficulty to GSO, OctantMix leads to a similar performance drop as zeroverse, indicating catastrophic failure. Standard CutMix only alters approximately 1/8 of the mesh volume, but it also causes a very significant drop(e.g., 20 points for Hunyuan3D-2). Even for the minimal half-and-half perturbations, where shape meaningfulness still largely remains, we still observe significant performance drop of >10 points most of the time. This performance drop is still very significant compared to the effect of other cues. These results highlight the importance of shape meaningfulness, since the models have significant difficulty to generalize even when we only partially perturb this cue, especially compared to other cues like texture meaningfulness.

Q4: Can we be confident that test sets are not seen in training.

A4: This is a valid and important question. We are quite confident that none of the methods have seen the GSO or Zeroverse datasets during training, as GSO is widely reserved as a test-only benchmark, and Zeroverse is also not typically used for training single-image-to-3D models. We are fairly confident that Toys4K is not seen by all the models(Page 6 of the trellis paper: “For quantitative evaluations, we use Toys4k, which is not part of our training set or those of the compared methods.”), except that we are not absolutely certain about Hunyuan3D-2, since they do not release or list their training data. Since Hunyuan3D-2 does not behave significantly differently from trellis on Toys4K, it is likely that they also did not use Toys4K in training.

Q5: Limitation section and formatting.

A5: Thank you for the suggestion! We will move the limitation section to the main paper and make it more comprehensive in the final version. We will also fix the reference formatting issues.

Thank you once again for your insightful and helpful feedback. Please feel free to share any additional suggestions or questions you may have!

Thank you for your comprehensive and detailed review! We appreciate your suggestions greatly.

Primary questions (1,4, and 5):

Q1: Plans for adding different visualizations to better illustrate the data in Table 3 and 5.

A1: We appreciate your point that Tables 3 and 5 (cue-ablation results) can be difficult to quickly interpret. In our current version, the tables present raw results to faithfully represent all our experiments. However, we completely agree with your suggestion and will include the following visualizations inspired by VBench in the final version:

1. Radar plots for each family of methods and individual methods. These radar plots illustrate sensitivity to different cues and contextualizes performance across regression, multi-view, and native 3D models.
2. A color-coded Cue x Model heatmap that summarizes per‑cue degradation across methods and datasets.
3. For cues with variable strength (e.g., mask dilation, occlusion), bar plots showing performance drops at different perturbation strengths. We believe these visualizations will clearly highlight key insights and would be very happy to hear any additional suggestions from you to further improve the clarity of our presentation.

Q4: Importance of shape meaningfulness vs. generic OOD failure; Shape CutMix experiments.

A4: We first clarify our claims on shape meaningfulness, and then present the results of the shape CutMix experiments suggested by the reviewer, which support our conclusion derived from Zeroverse that shape meaningfulness is indeed crucial.

The reviewer suggests that the failure of the models on shape meaningfulness perturbation (Zeroverse) could be a generic OOD failure. However, our goal is precisely to identify along which dimension models fail to generalize OOD. Our specific claim in the paper is that meaningful shape is critical for single-image-to-3D generalization, whereas texture meaningfulness is not. We contrast the OOD effects of shape versus texture explicitly, showing that changing to meaningless textures(while keeping other cues unchanged) had minimal impact.

Following your recommendation, we conducted several shape CutMix experiments of varying difficulty:

1. Half-and-half: We mix half of mesh M with the other half of mesh N to construct a new mesh from GSO meshes. This limits the distribution shift and preserves many local and global shape cues (e.g., surface smoothness, local symmetry), and also preserves a significant amount of shape meaningfulness to human perception. We show 3 variants: front-back, left-right, and top-bottom.

2. Standard cube-based CutMix. We follow the CutMix paper and randomly sample an axis-aligned 3D cube within the bounding cube of the object. We replace the part of mesh M that falls into the cube with the part from mesh N that falls into the same cube. When sampling the cube, we pin one of its corners at the corner of the object bounding cube to avoid significant discontinuity in the output shape. The length ratio(length_{sampled cube}/length_{bounding cube}) is uniformly sampled from [0.4, 0.6]. Most parts of the object M are outside the chosen cube and remain intact. Meanwhile, the local shape cues are mostly preserved.

3. OctantMix. We center each mesh and split it into 8 octants by the coordinate planes (xy, yz, and xz planes). Then we replace the part in each octant by the corresponding part from other random meshes from the same dataset. This variant still preserves the local shape cues, and it has a significantly smaller distribution gap than zeroverse compared to our original evaluation data (GSO).

We show the result on the GSO dataset, compared to Zeroverse numbers for reference. We use the CDx1000 metric following Table 3, lower is better.

The results show that all variants significantly harm performance. Notably, OctantMix causes a performance drop similar to zeroverse despite a smaller domain gap. Standard CutMix, which modifies only about 1/8 of the mesh volume, still results in large drops (e.g., 20 points for Hunyuan3D-2). Even minimal half-and-half perturbations, where shape meaningfulness is mostly preserved, typically cause performance drops of over 10 points, which is greater than for most other cues. This shows that shape meaningfulness is crucial, since models struggle to generalize even with partial perturbations of this cue, especially compared to other cues like texture meaningfulness.

Q5: How to ensure that each individual cue is entirely isolated? Analysis on the correlation of cues.

A5: Great question! We would like to clarify the principles behind our cue selection and provide additional quantitative analysis on correlations between the effects of different cues.

We chose these cues primarily based on their perceptual importance and interpretability to humans, rather than strict orthogonality. As noted in Section 2 (Lines 104-111), our selected cues originate from psychological studies of human visual perception and have strong foundations in prior vision research, as they represent factors that humans typically find meaningful. While some cues naturally remain disentangled (such as shading versus texture), others inherently overlap to some extent (like style and texture, as you correctly pointed out).

We appreciate your suggestion to quantitatively analyze cue correlations. However, some perturbations, such as style and texture, are difficult to combine meaningfully, making full combinational analyses like PCA impractical. Instead, we assess whether cues impact objects similarly by calculating per-object performance drops in CD for each cue and then computing the Spearman rank correlation between pairs of cues. This produces a correlation matrix showing how similarly each pair of cues affects the same set of objects. Note that this analysis only correlates with cue correlation but does not directly indicate cue correlation.

The result suggests that overall the correlation is low. Interestingly, texture and shading cues seem to affect a set of objects in similar ways, though they are inherently disentangled.

Other questions (2, 3, 6):

Q2: Provide some statistical tests to show that the drop in metrics are actually significant.

A2: Following your valuable suggestion, in the final version, we will include statistical tests (specifically, Wilcoxon signed-rank tests) to clearly demonstrate that the observed drops in performance metrics are statistically significant.

Q3: How is the performance robustness of each cue determined in Fig.1 (right)?

A3: The radar plot in Figure 1 illustrates normalized increases in Chamfer Distance (CD) when using perturbed images relative to original images. Specifically, each axis is normalized from 0 to 1 based on the largest observed drop across all models and cues. Detailed descriptions for how we compute each axis are provided in Appendix Section 2.3.

Q6: The dilation effect for the silhouette cue confounds internal texture; instead, use the mask from one object to mask out a different object.

A6: Thank you for highlighting this! We agree this is a valid concern, as dilated regions were filled with solid black. We will explain our design decision and provide results following your recommended approach. In our dilation perturbation, no image cue is removed, only black space is added. If the model ignored silhouette cues, it would rely on other cues and reproduce the original shape, which did not occur, suggesting the model depends on silhouette information.

Following your suggestion, for each mesh M, we randomly sample another mesh N and combine the image content of M with the silhouette of N (scaling M to avoid blank space). We then evaluate each model’s output against both the ground truth by image content (M) and by silhouette (N). Results are provided below.

Results indicate that performance significantly decreases with both ground truths. However, performance against the silhouette ground truth (N) is substantially better, clearly indicating that all models indeed rely heavily on silhouette cues to infer shape. These findings align well with our original conclusions, and we will include these results in the final version of our paper.

Thank you again for your thoughtful and constructive suggestions! We also welcome your further suggestions or questions!