GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussians

W1. More results
Thank you for your advices. In the appendix of the paper, we provide additional examples from Mip-Nerf360 as well as multi-view results.

W2. Reconstruction quality
The drop in reconstruction performance is attributed to cumulative errors introduced by the relatively coarse nature of level-1 superquadrics (42 vertices and 80 faces per primitive). Our 3DGS is initialized and densified based on this learned coarse level-1 superquadric representation, which limits the fidelity of the reconstruction. To improve reconstruction quality, we could increase the resolution of the superquadric by using a level-2 icosphere, which has 162 vertices and 320 faces. However, we didn't do so in this work for the following reasons: 1. Unfair comparison with the baseline (DBW): DBW also uses a level-1 icosphere, and increasing the resolution of our superquadric would create an unfair advantage. 2. Higher memory overhead: A level-2 icosphere significantly increases memory usage.

That said, if we were to use a level-2 superquadric to fit the scene, it would significantly improve reconstruction fidelity, achieving 96.9, 32.7, and 9.2 on SSIM, pSNR, and LPIPS, respectively, on the DTU dataset. These results are already highly comparable to the original 3DGS, allowing us to retain the attractive editability while achieving more comparable reconstruction quality. In future work, we will explore how to efficiently apply our pipeline to high-resolution and more detailed primitives to achieve low overhead with higher reconstruction quality. For instance, one potential direction is to adopt a sparse sampling strategy when deriving the attention-guided centering loss, which could significantly reduce computational costs while preserving effectiveness.

W3 Training & editing time and FPS
Although we cannot achieve the training speed of the original Gaussian splitting, in relatively simple scenes like DTU, we only need 1.7 times the time (approximately 5K iterations) to reconstruct comparable high-fidelity scenes. Additionally, the scenes we reconstruct possess part-aware and semantic coherent decomposition ability, and inherent editability. Apart from that, we can also achieve real-time editing and rendering because the superquadric itself does not participate in the rendering process. Instead, it provides an editable block skeleton structure, while only the Gaussian representation is used for rendering. As a result, our method achieves real-time rendering performance exceeding 30 FPS, comparable to the original 3DGS baseline.

Q1. motivation clarfication and comparsion with [1-3] "combining existing works"
We would like to emphasize that our work is not a straightforward combination of [1]/[2] and [3], and it differs significantly from theirs in several key aspects: (1) decomposition achievement compared to [1] and [2]. Garfield[1] leverages contrastive learning to group 3D representations into pre-segmented part-level masks, which relies heavily on the quality and consistency of pre-segmented masks. In contrast, our approach introduces dual-rasterization and attention-guided clustering to implicitly derive self-supervised semantic priors. This strategy not only eliminates the labor-intensive process of ensuring multi-view and fine-grained consistency for 2D masks but also allows for controlling the granularity of part-level decomposition by adjusting the threshold. On the other hand, Total-Decom[2] only focuses on decomposing indoor scenes into foreground objects and background, which cannot acheive part-level decomposition like our method. (2) editing motivation compared to [3] SC_GS [3] enables editing by reconstructing 4D motion assets from dynamic video sequences, which are not available in our setting, as there are no motion-based input references or guidance.Instead, our editing capability benefits from the actionability of primitives and the binding and local transformations enabled by the hybrid representation strategy.
In conclusion, [1]/[2] + [3] can't achieve the effects of our work, since our work is not a simple combination of these contemporary works. In contrast, our work focuses on leveraging a hybrid representation combined with self-supervised semantic priors to achieve part-aware, decomposable reconstruction while simultaneously enabling interpretable editing.

1. Computational Cost
Regardless of the training cost, our method also supports real-time rendering and editing during the inference stage, which we believe is a more critical feature for 3D reconstruction and editing tasks. Moreover, compared to Garfield, our method requires only 1.6 times more computational time and incurs a similar memory cost on the DTU dataset, as both approaches use the same SAM model to assist with semantic awareness. However, our method achieves better part-aware decomposition, and editing capabilities across diverse scenes.

2. Comparison with Garfield
Since few part-aware decomposition ground truths are available in multi-view image datasets, it is challenging to provide convincing quantitative decomposition results. Instead, in this anonymous link https://qpkl9034.github.io/, we provide additional visual comparisons between our method and Garfield, including dense part-aware decomposition (scale = 0.05) and Garfield-based editing results. From the comparison, it is evident that, due to the effectiveness of our attention-guided centering loss, our approach achieves superior part-aware decomposition with consistent semantic coherence. Additionally, by effectively leveraging the inherent actionability of superquadrics through our hybrid representation-based pipeline, our method delivers more satisfactory and multiple-part editing results.

Besides, we also compared the quantitative reconstruction quality comparison of Garfield and our method on the DTU dataset, further demonstrating that our hybrid representation-based framework not only achieves high-quality part-aware decomposition but also maintains superior rendering quality.

We will cite the Garfield paper and continue to update additional comparison results, which will be included in the final version of the paper.

W1. Multi-view results
Thank you for your suggestion. In the appendix of the paper, we provide additional examples of multi-view results. Furthermore, we include related demonstration videos in the supplementary materials for better illustration.

W2 & W3. "Is Superquadrics + 3DGS a good design?", and comparison with "Mixture-of-Volumetric-Primitives"
We believe superquadric + 3DGS is a good design. Compared to Mixture-of-Volumetric-Primitives, which are less flexible for modeling highly detailed or irregular structures[1] and come with significant memory overhead[2], leveraging a small number of mid-level primitives such as superquadrics combined with 3D Gaussian Splatting (3DGS) provides a balance of actionability of primitives and the advanced expressiveness and real-time rendering capabilities of Gaussian representations. However, as discussed in the Limitations section of the paper, similar to our baseline DBW, the current use of superquadrics and Gaussian representations does have certain limitations and may not be the optimal choice. Besides, the key contribution of our work lies in the proposal of self-supervised implicit semantic guidance and the provision of a new insight into exploring the feasibility of combining primitives with Gaussian representations for part-level decomposition. In future work, we plan to explore ways to hybrid superquadrics with more diverse primitives, such as curved surfaces integrated with 3DGS, to further improve our hybrid representation and achieve better results.

W4. Comparison with deformation works
Our approach differs from deformation-based methods in several key aspects: (1) motivation. Instead of extracting a single deformable mesh, our work focuses on creating part-aware, decomposable 3D assets. While both approaches enable editing capabilities, our framework is specifically designed to support more part-aware downstream tasks such as segmentation, part-level decomposition and editing. For example, deformable methods cannot achieve edits like “swap” or “separate then move”, as demonstrated in Figure 4, because they model the entire scene as a single object, lacking the part-aware decomposition necessary for such operations.

(2)contribution The deformation-based work you mentioned primarily focuses on optimizing a cage to represent the scene as a single “clay-like” deformable object. In contrast, our contribution lies in exploring the semantic relationships between parts and leveraging the advantages of different representations to simultaneously achieve semantic coherence, decomposability, and editability. This exploration enables a richer and more structured understanding of the scene, making our approach distinct in its objectives and capabilities.

Thank you for your valuable suggestions on our work. We will incorporate the above discussion into the Related Work section in the final version of our paper.

Q1-1. "requires bounding box and point prompts along with input posed images..., unfair comparisons"
The bounding box and point prompts we used in our pipeline are not ground truths. In fact, In fact, the bounding box and point prompts are derived through differentiable dual soft rasterization, based on the projections of the current superquadric vertices. Thus, these prompts are also dynamically optimized during the training process, ensuring that no unfair comparisons are introduced.

Q1-2. additional baseline where "using 3DGS to reconstruct the semantic scenes where the SAM segmented images are used as training images".
In fact, our paper already compares with an more advanced baseline using the similar idea as mentioned. Specifically, in the baseline algorithm OmniSeg, contrastive learning is used to constrain Gaussians within segmented parts. While this approach does generate semantic-aware decomposition masks, it faces limitations due to the nature of the Gaussian representation. As a result, the reconstructed scene cannot support controllable editing of these parts, as it is not feasible to apply distinct transformations to individual Gaussians.

Q2. "sensitive to segmentation failure"
Instead of obtaining consistent per-part masks across multiple views, as done in methods such as SOTA relate work SegAnyGaussian，our method requires only silhouette preprocessing of the input images to locate the object for decomposition as shown in Fig.2. Thanks to advancements in large vision foundation models, extracting a binary silhouette for objects has become relatively straightforward, with minimal need for manual correction and a low likelihood of failure. Therefore, even when handling more complex scenes such as flowers and leaves, our method in Stage 1 only requires obtaining their overall silhouette.

1. Inherit Limitations from DBW In Table 2, Figure 8, and Figure 9 of the paper, we provide a detailed comparison and showcase the reconstruction, decomposition, and editing quality of our method across diverse scenes, including DTU, BlendedMVS, and Mip-NeRF360. The results clearly demonstrate that, compared to the baseline method DBW, our approach achieves significant improvements in terms of texture quality, decomposition accuracy, and editing capabilities. These substantial advancements highlight the effectiveness of our proposed attention-guided centering loss and the hybrid representation-based pipeline.

2. Limitations
Our main contribution lies in proposing the attention-guided centering loss and the hybrid representation-based pipeline, enabling a part-aware decomposable and editable reconstruction framework. As discussed in the Limitations section of the paper, our method does have certain limitations, primarily due to the mid-level nature of superquadrics. In future work, we plan to enhance our approach by augmenting superquadrics with a more diverse and detailed set of hybrid primitives, such as cuboids and curved surfaces, to effectively address these limitations.

W1. Reconstruction quality
Indeed, we cannot achieve exactly the same reconstruction quality as the original 3DGS. This is because our 3DGS is initialized and densified based on the learned superquadric representation, which introduces cumulative errors due to the relatively coarse nature of level-1 superquadrics (42 vertices and 80 faces per primitive). To improve reconstruction quality, we could increase the resolution of the superquadric by using a level-2 icosphere, which has 162 vertices and 320 faces. However, we didn't do so in this work for the following reasons: 1. Unfair comparison with the baseline (DBW): DBW also uses a level-1 icosphere, and increasing the resolution of our superquadric would create an unfair advantage. 2. Higher memory overhead: A level-2 icosphere significantly increases memory usage.

That said, if we were to use a level-2 superquadric to fit the scene, it would significantly improve reconstruction fidelity, achieving 96.9, 32.7, and 9.2 on SSIM, PSNR, and LPIPS, respectively, on the DTU dataset. These results are already highly comparable to the original 3DGS and other baselines.

In future work, we will explore how to efficiently apply our pipeline to high-resolution and more detailed primitives to achieve low overhead with higher reconstruction quality. For instance, one potential direction is to adopt a sparse sampling strategy when deriving the attention-guided centering loss, which could significantly reduce computational costs while preserving effectiveness.

Q1. Faliure case
Thank you for your suggestion. In the appendix of the paper, we have also included our failure cases to provide a comprehensive evaluation of our method.

Q2. K value decision and robustness
For fairness in experimental comparison, we used the default initial number $K$ of primitives following DBW's setting in both our experiments and DBW’s, where $K=10$. In fact, due to the introduction of the dynamic adaptive splitting and fusion strategy, the initial value of $K$ is relatively robust, where $K$ has a minimal impact on the final number of effective primitives obtained. The following table presents the initial $K$ and the final visible superquadrics obtained after training for different scenes in the DTU dataset. These results demonstrate that the $K$ is relatively robust within our pipeline.

W1. Multi-view results
Thank you for your suggestion. In the appendix of the paper, we provide examples of multi-view results.

W2. Reconstruction quality
Our method achieves lower reconstruction quality compared to the 3DGS algorithm primarily due to cumulative errors caused by the relatively coarse nature of level-1 superquadrics. In the future, we plan to enhance reconstruction quality, particularly for highly-detailed scenes, through two approaches: 1. Using higher-resolution superquadrics: Tests conducted on the DTU dataset indicate that reconstruction quality improves significantly when using level-2 superquadrics(2.4↑ on SSIM, 2.0↑ on PSNR, and 1.1↓ on LPIPS). 2. Assisting superquadrics with more fine-grained hybrid primitives: Incorporating detailed primitives may also help mitigate the cumulative errors introduced by coarse superquadrics, further enhancing reconstruction fidelity.

W3. Background limitation
As discussed in the Limitations section in paper, similar to our baseline DBW, our method is still chanllenging to to handle backgrounds. This is due to the fact that backgrounds typically have more complex and detailed structures with richer semantic information. In future work, we plan to enhance our pipeline by assiting superquadrics with a more diverse set of hybrid primitives, such as cuboids and curved surface to address these limitations.

Q1. FPS and training time clarification
We can achieve real-time editing and rendering because the superquadric itself does not participate in the rendering process. Instead, it provides an editable block skeleton structure, while only the Gaussian representation is used for rendering. As a result, our method achieves real-time rendering performance exceeding 30 FPS, comparable to the original 3DGS baseline. Besides, the 6-hour training time typically includes 5 hours for Stage 1 and 1 hour for Stage 2.

W1 & Q1. Control over granularity
The granularity of parts can be controlled through the splitting threshold, $\beta$, as illustrated in Figure 5(c). Specifically, when the distance between the semantic cluster centroids within a superquadric exceeds the threshold $\beta\check B_k$, where $\check B_k$ is the diagonal of the corresponding bounding box, the superquadric is subdivided into multiple parts. Consequently, a smaller value of $\beta$ tends to resulting in a finer-grained decomposition, while a larger value leads to a coarser one. By employing this dynamic splitting strategy, we are able to decompose the 3D asset at varying levels of granularity.

W2 & Q2 & W3 & Q5. Editing over multiple parts, Multi-view results, more results
Our method enables editing of multiple parts simultaneously. In the appendix of the paper, we provide additional examples from Mip-Nerf360, multi-part editing as well as multi-view results.

Q4. LLFF dataset
The primary contribution of our work is the introduction of a novel semantic-aware, decomposable pipeline that leverages a hybrid representation to achieve part-aware, semantically coherent, and actionable reconstructions. As discussed in the Limitations section in paper, similar to our baseline DBW, the current use of superquadrics does has certain limitations, such as difficulty in fitting concave shapes and capturing highly detailed structures as appeared in LLFF. However, our work offers a new perspective and approach to exploring the feasibility of combining primitives with Gaussian representations for part-level decomposition. In future work, we plan to enhance our pipeline by assiting superquadrics with a more diverse set of hybrid primitives, such as cuboids and curved surface to address these limitations.

Q6. Training time clarification
The 6-hour training time, which includes both Stage 1 and Stage 2, represents the average training time across all datasets presented in the paper. In practice, the training time largely depends on the complexity of the structures in the dataset. For simpler structures, such as the Smurf model in DTU or the camera data in BlendedMVS, the training time is approximately 3 hours. However, for more complex structures, such as the Gundam model in BlendedMVS, the training time may extend to around 7 hours.