Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention

Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.

[Q1]: Novelty of the proposed SSA.

[A1]: Thank you for your comments. The main contribution of our SSA is to address the scalability and efficiency limitations in training with ultra-long tokens in 3D data, where the number of tokens increases cubically with resolution. The SSA is also well acknowledged by all other three reviewers: Reviewer WHua noted, "the method addresses the core inefficiency in high-resolution 3D generation with a well-motivated and technically solid solution". Reviewer uJi9 said our approach is "a novel framework for scalable 3D generative modeling", giving it a significance score of 4 (excellent). Reviewer y1dk commented that "the implementation of SSA is non-trivial and could benefit the 3D community overall as a general mechanism for harvesting sparsity naturally existing in 3D data".

Although our core idea of accelerating the attention mechanism via block partitioning is inspired by NSA, NSA is only applicable to structurally organized 1D sequences, as it partitions tokens based on their sequential order. This approach loses critical information when applied to unstructured tokens, such as sparse 3D voxels, and suffers from two key limitations. First, even though tokens within the same block may share consecutive indices, they are not necessarily spatially adjacent in 3D space. Second, due to the nature of sparse voxel structures, blocks with the same index across different samples may correspond to entirely different spatial regions. These issues collectively lead to unstable training convergence and a loss of detail in the generated meshes. To address this, we propose a novel method that partitions tokens into blocks based on their 3D spatial locations, followed by blockwise token selection. As shown clearly in Figure 6 of the paper, meshes generated using SSA exhibit smoother surfaces and more fine-grained details. We believe the novelty of SSA is well supported, as also highlighted by Reviewers WHua, uJi9 and y1dk.

[Q2]: Sparse Conditioning Mechanism for improved alignment.

[A2]: Thanks for your valuable suggestions. We have presented a qualitative comparison in Figure 5 of the supplementary material between results generated using cross-attention over the full input images and those using sparse conditioning. The regions highlighted in red indicate areas where the meshes generated by the former does not align well with the input images. After applying sparse conditioning, the generated results are corrected in these regions. To avoid potential cherry-picking, we perform quantitative comparisons on a subset from Objaverse dataset, which comprises 500 meshes disjoint from the training set. Unlike the approach in our paper which uses metrics such as Uni3D and ULIP-2 to measure semantic similarity between generated meshes and input images, we employ the FID metric to more rigorously evaluate the geometry quality of generated meshes, which requires GT meshes. We render paired normal maps for both the generated meshes and GT meshes from the same viewpoints, then calculate the Normal-FID between them to assess model performance. The results are as follows:

It can be observed that the sparse conditioning module brings a certain level of performance improvement, demonstrating its effectiveness. However, we would like to clarify that the main performance gains and primary contributions of our method come from the proposed fully symmetric sparse SDF VAE and the SSA mechanism, which significantly improve both training and inference efficiency. These advances overcome the resolution limitations of previous methods and enable the generation of 1024³ resolution meshes.

[Q3]: What does "additional valid voxels" in line 144 refer to.

[A3]: Thank you for pointing this out. Our Sparse SDF VAE encodes sparse SDF grids of resolution N³ into sparse latent tokens at a lower resolution of (N/8)³, and then decodes them back to N³ sparse SDF grids. For each sparse latent token, the decoder upsamples it to 8³ grids. However, only a subset of these 8³ grid points correspond to the original input sparse SDF grids, and the remaining grid points fall into the category of extra valid voxels. We apologize for the confusion and misunderstanding this may have caused. In the final version, we will include a clear explanation of extra valid voxels to improve clarity.

Thank you sincerely for your kind follow-up and for raising your rating. We are glad our rebuttal has addressed your concerns, and we appreciate your constructive review.

Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.

[Q1]: It is unclear in the paper how the sparse structure is generated at inference time.

[A1]: Thank you for raising this point. Our Direct3D-S2 employs an additional dense 3D VAE and DiT trained in a manner consistent with Trellis to predict indices of sparse latent tokens within voxel grids, which constitute the sparse structure. This is noted in Lines 29-30 of the supplementary material. We acknowledge this may cause confusion and incomprehension during paper reading, and we sincerely apologize for this oversight. In the final version, we will restructure the method section to include explicit descriptions of the sparse structure generation process, thereby avoiding any misunderstanding. We deeply appreciate your suggestion.

[Q2]: The speed of generation is unclear.

[A2]: Thanks for your comments. Within the DiT architecture, the self-attention module exhibits O(N²) complexity for N tokens, while other components such as feed-forward MLPs maintain linear complexity O(N). At high resolutions where the number of tokens N become substantial, self-attention accounts for over 90% of the model’s runtime. Therefore, although SSA only accelerates the self-attention module, it still leads to a significant improvement in the overall generation speed. In the table below, we compare the time required to denoise a single step using SSA versus full attention:

For 512³ resolution, around 30 denoising steps are needed, while for 1024³ resolution, around 15 steps suffice. Predicting the sparse structure takes approximately 5 seconds. Therefore, excluding post-processing steps such as mesh simplification, generating a 512³ resolution mesh takes about 0.5 minute, and a 1024³ resolution mesh takes around 2 minutes, which meets the requirements for practical applications.

[Q3]: Experiments compute resources.

[A3]: Thanks for your comments. As documented in the Implementation Details section of supplementary material, we have already specified the type of GPU and training durations. For further clarity, we provide additional experiments compute resources in the table below:

Note that we trained our Direct3D-S2 using 8 GPUs, and the table above lists the GPU memory usage per single card for training.

[Q4]: No information about data curation.

[A4]: Thanks for your comments. In fact, we described the data processing pipeline in Section 5.1 in the paper. Here, we provide a more detailed explanation of the data curation process to avoid any misunderstanding:

(1) Data Filtering. We first render all meshes from the ShapeNet, Objaverse, and Objaverse-XL datasets from six fixed viewpoints. We then randomly select several thousand 3D models and manually annotate whether each one is of high quality. For each mesh, we extract CLIP features from its six rendered images and concatenate them. Based on this, we train a lightweight binary classification network. Using this network, we filter out 452k high-quality meshes from the datasets for training.

(2) Geometry processing. We follow the same strategy as CLAY [1]: for each non-watertight mesh, we first compute a UDF grid, and then use ray tracing to determine the visibility of each grid point. For invisible interior points, we assign negative UDF values. The resulting mesh is made watertight via marching cubes. We then extract its SDF grid for training.

(3) Rendering conditional images. We rendered 45 RGB images per mesh at 1024x1024 resolution with random camera parameters. The camera configuration space is defined as follows: elevation angles ranging from 10° to 40°, azimuth angles spanning [0°, 180°], and focal lengths varying between 30mm and 100mm.

[Q5]: Additional limitations.

[A5]: Thank you for your suggestions. Another limitation of our method is that, since the final output of our model is a sparse SDF grid, we need to assign a default SDF value to the remaining grid points before applying marching cubes. As a result, the extracted mesh is hollow and requires post-processing, similar to Trellis, to remove internal structures. This issue could also be addressed by implementing a sparse marching cubes algorithm, which we leave as future work.

[1] Zhang, Longwen, et al. Clay: A controllable large-scale generative model for creating high-quality 3d assets. TOG 2025.

Thank you for your thoughtful feedback, and we are pleased to hear that our rebuttal addressed your concerns well. Your constructive suggestions are very helpful, and we sincerely appreciate your consideration.

Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.

[Q1]: Lacks numerical ablation studies across different resolutions in both VAE and generation.

[A1]: Thanks for your valuable suggestions. We present the numerical ablation studies of VAE across different resolutions below. The evaluation metrics includes Chamfer Distance and F1 score with thresholds of 0.005 and 0.001. The metrics are multiplied by 10000, 100 and 100, respectively.

We perform quantitative comparisons of generation across different resolutions on a subset from Objaverse dataset, which comprises 500 meshes disjoint from the training set. Unlike the approach in our paper which uses metrics such as Uni3D and ULIP-2 to measure semantic similarity between generated meshes and input images, we employ the FID metric to more rigorously evaluate the geometry quality of generated meshes, which requires GT meshes. We render paired normal maps for both the generated meshes and GT meshes from the same viewpoints, then calculate the Normal-FID between them to assess model performance. The results are as follows:

There quantitative results demonstrate the scalability of our Direct3D-S2 in terms of resolution.

[Q2]: The 256 resolution results in Figure 3 appear to be significantly worse than those of Trellis.

[A2]: Thanks for your comments. We acknowledge that at 256³ resolution, our method performs worse than Trellis, an expected outcome for two reasons:

(1) Trellis' VAE employs a 53-dimensional FlexiCubes parameters at 256³, whereas our Sparse SDF VAE uses only 1D SDF representations at the same resolution.

(2) Trellis' VAE is supervised by high-resolution normal maps rendered from GT meshes, while we use 256³ SDF grids as supervision signals, which provide significantly inferior geometric fidelity compared to mesh-derived normals.

However, our approach offers critical advantages over Trellis:

(1) Model scalability: Trellis’ VAE requires rendering normal maps via FlexiCubes representations for supervision, demanding prohibitive GPU memory (e.g., out-of-memory errors at 512³). Our Sparse SDF VAE scales seamlessly to 1024³ resolution.

(2) Training efficiency: Trellis’ asymmetric VAE (point cloud → latent → FlexiCubes with normal map supervision) exhibits high training complexity, requiring 64 A100 GPUs for several days to converge. Our symmetric encoder-decoder architecture (SDF grid → latent → SDF grid) reduces optimization difficulty, converging from 256³ to 1024³ in just 2 days on 8 A100 GPUs.

(3) Long-token modeling: Trellis’ DiT uses full attention, becoming prohibitively slow with long tokens. Our proposed SSA substantially improves the training and inference efficiency.

(4) Progressive quality gain: While 256³ SDF grids offer limited quality, this resolution serves merely as a warm-up phase to save training time. Quality progressively improves at 512³, with 1024³ SDF grids approaching GT mesh fidelity, enabling significant performance gains at higher resolutions.

In summary, Trellis achieves strong 256³ reconstruction quality but suffers from inefficiency and 256³ resolution limitations. Our Direct3D-S2 framework is explicitly designed to overcome this, enabling scalable VAE resolution (up to 1024³) and introducing SSA for efficient DiT training/inference with ultra-long tokens.

[Q3]: Some unfaithful details are present in the generated results.

[A3]: We sincerely appreciate you highlighting this observation. The surface details occasionally manifesting in our generated meshes but absent in input images stem from a dataset limitation: a subset of textureless meshes in Objaverse renders geometrically ambiguous conditional images. This creates a supervision mismatch during training, where input images lack discernible details while target meshes retain them, causing the model to potentially generate unfaithful surface details misaligned with the conditioning image. To resolve this, we will filter textureless meshes from the training set and retrain the DiT model. The updated model will also be publicly released. We deeply value your insight, which significantly improves our method’s robustness.

[Q4]: The paper lacks generation results on in-the-wild images.

[A4]: Thank you for your suggestion. Our test cases collected from professional communities encompass diverse image types, including samples generated by text-to-image models, images drawn by artists, and in-the-wild images, on which our method consistently demonstrates robust performance. Due to current conference policy prohibiting PDF attachments or external links during rebuttal, we will include additional results on in-the-wild images in the final version.

Thank you for taking the time and effort to review our response. We are glad that our rebuttal has addressed most of your concerns. If you have any further questions, please let us know and we will respond promptly!

Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.

[Q1]: Quantitative results of ablation studies.

[A1]: Thanks for your valuable suggestions. We perform quantitative comparisons on a subset from Objaverse dataset, which comprises 500 meshes disjoint from the training set. Unlike the approach in our paper which uses metrics such as Uni3D and ULIP-2 to measure semantic similarity between generated meshes and input images, we employ the FID metric to more rigorously evaluate the geometry quality of generated meshes, which requires GT meshes. We render paired normal maps for both the generated meshes and GT meshes from the same viewpoints, then calculate the Normal-FID between them to assess model performance. The results are as follows:

(1) Generation results of our Direct3D-S2 at various resolutions:

(2) Ablation results for different modules of SSA at 512³ resolution:

(3) Comparative results of SSA against Full Attention and NSA at 512³ resolution:

(4) Ablation results for sparse conditioning at 512³ resolution:

[Q2]: Results when only cmp or only slc is used in SSA.

[A2]: Thanks for your suggestions. We have added supplementary experiments using only cmp or slc, with results presented in [Q1](2). When using only the cmp module, which provides exclusively coarse-level features, the model's performance significantly degrades. When using only the slc module, the model consistently collapses (with loss diverging to NaN) after several hundred training iterations due to the absence of the cmp module which learns block-level attention scores. Consequently, the slc module must operate in conjunction with the cmp module.

[Q3]: Quantitative comparisons of Sparse SDF VAE.

[A3]: Thanks for your suggestions. We present the quantitative comparisons of VAE below. The evaluation metrics includes Chamfer Distance and F1 score with thresholds of 0.005 and 0.001. The metrics are multiplied by 10,000, 100 and 100, respectively. Regarding training efficiency, Dora requires 32 A100 GPUs for two days of training, while Trellis necessitates 64 A100 GPUs. Other compared methods do not report training time. In contrast, our Sparse SDF VAE requires only 8 A100 GPUs for one day to train at 512³ resolution, and merely an additional day to progress to 1024³ resolution.

[Q4]: Clarification on performance attribution and results at 256³ resolution.

[A4]: Thanks for your comments. We acknowledge that at 256³ resolution, our method performs worse than Trellis, an expected outcome for two reasons:

(1) Trellis' VAE employs a 53-dimensional FlexiCubes parameters at 256³, whereas our Sparse SDF VAE uses only 1D SDF representations at the same resolution.

(2) Trellis' VAE is supervised by high-resolution normal maps rendered from GT meshes, while we use 256³ SDF grids as supervision signals, which provide significantly inferior geometric fidelity compared to mesh-derived normals.

However, our approach offers critical advantages over Trellis:

(1) Model scalability: Trellis’ VAE requires rendering normal maps via FlexiCubes representations for supervision, demanding prohibitive GPU memory (e.g., out-of-memory errors at 512³). Our Sparse SDF VAE scales seamlessly to 1024³ resolution.

(2) Training efficiency: Trellis’ asymmetric VAE (point cloud → latent → FlexiCubes with normal map supervision) exhibits high training complexity, requiring 64 A100 GPUs for several days to converge. Our symmetric encoder-decoder architecture (SDF grid → latent → SDF grid) reduces optimization difficulty, converging from 256³ to 1024³ in just 2 days on 8 A100 GPUs.

(3) Long-token modeling: Trellis’ DiT uses full attention, becoming prohibitively slow with long tokens. Our proposed SSA substantially improves the training and inference efficiency.

(4) Progressive quality gain: While 256³ SDF grids offer limited quality, this resolution serves merely as a warm-up phase to save training time. Quality progressively improves at 512³, with 1024³ SDF grids approaching GT mesh fidelity, enabling significant performance gains at higher resolutions.

In summary, Trellis achieves strong 256³ reconstruction quality but suffers from inefficiency and 256³ resolution limitations. Our Direct3D-S2 framework is explicitly designed to overcome this, enabling scalable VAE resolution (up to 1024³) and introducing SSA for efficient DiT training/inference with ultra-long tokens.

[Q5]: Potential for text-to-3D extension.

[A5]: Thanks for your valuable suggestions. While our framework could technically enable text-to-3D generation by substituting image conditioning with text conditioning during training and inference, this approach faces fundamental data scalability limitations since text-to-image/video models typically require billions of training samples whereas available 3D mesh datasets are orders of magnitude smaller (∼1M total, reduced to ∼100k usable samples after quality filtering). This severe data scarcity prevents training sufficiently generalizable text-to-3D models, evidenced by Trellis’ text-to-3D variant exhibiting significantly inferior stability and generalization versus its image-to-3D counterpart. Consequently, state-of-the-art 3D generation works (e.g., Hunyuan3D 2.0 [1], Craftsman [2], Triposg [3], Hi3DGen [4]) predominantly adopt text-to-image-to-3D pipelines leveraging pre-trained text-to-image models, and notably in our evaluation cases where a substantial portion of images sourced from communities were synthesized by text-to-image models, our Direct3D-S2 achieves compelling results, demonstrating seamless compatibility with such text-to-image-to-3D workflows.

[Q6]: Redundant mention of ULIP-2, Uni3D, and OpenShape in Lines 256 and 264.

[A6]: Thanks for your suggestions. We will reorganize these sentences in the final version.

[1] Zhao, Zibo, et al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv 2025.

[2] Li, Weiyu, et al. CraftsMan3D: High-fidelity Mesh Generation with 3D Native Diffusion and Interactive Geometry Refiner. CVPR 2025.

[3] Li, Yangguang, et al. Triposg: High-fidelity 3d shape synthesis using large-scale rectified flow models. arXiv 2025.

[4] Ye, Chongjie, et al. Hi3dgen: High-fidelity 3d geometry generation from images via normal bridging. ICCV 2025.

Thanks for your positive feedback and for taking the time to consider our responses. We are pleased that our rebuttal has addressed your concerns, and we will include the additional ablation in the final version.