GALA: Geometry-Aware Local Adaptive Grids for Detailed 3D Generation

1. More datasets other than ShapeNet are preferred:

   We have now provided additional Text-to-3D generation results on Objaverse. Please refer to Appendix F of the revised version.

2. Large Shape Reconstruction Results:

   We have tested our GALA representation on large shapes with more than a million triangle faces from the Stanford 3D Scanning dataset. We can see from Appendix D of the revised version that our GALA-based approach can capture rich and accurate geometry details of the large and complicated shapes and outperform other baselines.

3. “The authors claim to introduce a 3D representation, yet, they rely on a generative model to encode and decode shapes. The soundness of the representation is never evaluated.”:

   Thank you for your feedback. To clarify, yes, the representation is specially designed for the generation in 3D. While tightly integrated into the generative process, its soundness as a representation is evaluated through reconstruction tasks (Section 4.3) and ablation studies (Section 4.5), demonstrating its standalone capacity to model surfaces effectively. We will revise the manuscript to make this dual role clearer.

1. Error propagation through steps of cascaded generation:

   Good question. Our method, along with many other sequential or cascaded generations, including the two you mentioned (XCube and OctFusion), all need to address error accumulation due to the training and inference disparity. We mainly rely on noise augmentation used in previous works [1, 2, 3], where noise is injected to GT conditions by the diffusion forward process with random noise levels. From our results as well as others, it seems that the error can be effectively managed using similar techniques.

2. Comparison to OctFusion and XCube:

   We now show reconstruction and generation comparisons with OctFusion and XCube in Appendix C of the revised version, demonstrating that our method leads to more accurate and efficient representation than the two baselines, as well as its capability of generating better geometric details. Detailed evaluation protocols can be found in Appendix C.

3. Training on more complicated datasets such as Objaverse dataset:

   We are now showing additional text-to-3D experiments in Appendix F of the revised version. This demonstrates our ability to adapt to larger and more complicated datasets.

4. What is the difference between Mosaic-SDF and “Ours w/o A”?:

   Yes, “Ours w/o A” is similar to Mosaic-SDF except that we still have the additional octree forest setting in “Ours w/o A”.

5. More comparisons with Mosaic-SDF:

   In Figure 7 of the main paper, it is noted that we can model complicated geometries such as the interior of the car model with fewer and quantized parameters than Mosaic-SDF. We also provide additional Mosaic-SDF comparisons in Appendix E of the revised version. In Appendix E, we can see that GALA can represent complex geometry more accurately both in low and default parameter counts.

[1] Ho, Jonathan, et al. "Cascaded diffusion models for high fidelity image generation." Journal of Machine Learning Research 23.47 (2022): 1-33.

[2] Saharia, Chitwan, et al. "Photorealistic text-to-image diffusion models with deep language understanding." Advances in neural information processing systems 35 (2022): 36479-36494.

[3] Xu, Xiang, et al. "Brepgen: A b-rep generative diffusion model with structured latent geometry." ACM Transactions on Graphics (TOG) 43.4 (2024): 1-14.

1. What if we use the same Flow Matching strategy as Mosaic-SDF:

   Flow Matching (FM) [1] has shown some advantages over general diffusion schemes. Both in theory and practically, FM should be adaptable to our generation training process. The reasons we chose v-prediction diffusion as our generation training scheme are: (1) At the early development stage, diffusion training frameworks were more mature and easy to get access to; (2) Other baselines other than Mosaic-SDF adopted diffusion, and Mosaic-SDF itself did not conduct an ablation study to show performance boost by shifting usual diffusion paradigm to FM in its specific case. Adopting FM to GALA would be interesting to explore in future work.

2. The grouping of citations:

   We are sorry about this. It is fixed now.

3. How is each grid subdivided: In Figure 1, as well as in the introduction, the subdivisions are based on octrees, where the space in a cube is evenly subdivided into 8 smaller cubes. After recursive subdivision to a certain depth, adaptive grids will be extracted in non-empty nodes (green ones visualized in Figures 1 and 3). We have made this information more consistent and clear by changing the wording in Sec 3.1.1 and Figure 3 as well as adding more explanations in Appendix B.4, in the revised version. How is a mesh extracted from this adaptive grid: We did the former way as you mentioned, by first extracting values within locations of a certain resolution regular grids by interpolation via Equation 2. We have addressed this concern in Sec 3.1.3 as well as Appendix A.1, in the revised version.

4. What does “the weighting function which … rotation matrix $O_i$” mean:

   Sorry for any confusion. Yes, it is about Equation 3. What we want to explain here is that the neighborhood, on which the weight function is defined, is determined by $\ell_\infty$-norm centered at center $\mathbf{p}_g^i$, orientated by $\mathbf{O}_i$ and scaled via $\mathbf{s}_i$. We have changed the wording there.

5. Is $\mathbf{x}_{V}$ and $X_V$ the same thing?:

   $X_V$ means a set of vectors and $x_{V}$ means a single vector, and $x_{V} \in X_V$. To clarify this further, every node’s information will be flattened into a vector. Since we have multiple nodes, each level of information forms a set of vectors. We have made this part more clear in the revised version, please refer to line 302-305 of Sec 3.2.

6. The later two stages will not be affected by the conditioning?:

   No. All three stages will be conditioned on the input labels. As formulated via $P(X_o|y)$, $P(X_{\bar{V}}|X_o,y)$ and $X_V = F(X_{\bar{V}},y)$ in lines of 320, 322 and 323, there are $y$ labels in all three equations. You can also find more detailed figure illustration in Figure 19 of Appendix A.5. [1] Lipman, Yaron, et al. "Flow matching for generative modeling." arXiv preprint arXiv:2210.02747 (2022).

1. Failure cases show the method cannot cover long thin parts:

   The inability to fully handle very thin features is common to all methods, e.g., see the last row of Fig. 6 for some reconstruction results. Whatever the method, its representational capacity will always be limited while there will always be features that are too small to capture. Fig. 6 also shows a typical example, that our method with GALA outperforms the competitors in this regard.

   Note that the imperfect result that the reviewer was looking at, in Fig. 27, was that of generation. As explained in Appendix G, the disconnections in the thin part during generation were caused by the disconnected root voxels that were generated, while the thin structures were correctly captured within each root voxel. More advanced generation techniques, which are parallel to our current contribution, may help us cover long thin parts better in the generation phase. We leave that for future work.

2. Will the code be released publicly in the future?: Yes, the code and any data we used will be made publicly available upon paper acceptance.