Coherent 3D Scene Diffusion From a Single RGB Image

1. Reconstructing background objects

Indeed, structural elements can be considered part of holistic scene reconstruction. Including structural elements in the scene prior can certainly be beneficial and can improve the coherence of the scene. We will extend our formulation with the structural elements of the scene and train a layout estimator to reconstruct them. We will report our findings in the final paper - thanks a lot for this suggestion!

2. Global shape prior

Thanks for the comment. While our model produces clean shape modes even under heavy occlusions compared to blurry or incomplete shapes of prior work, our learned shape prior does not necessarily match the input image perfectly. In our alignment loss formulation, we sample points from the 3D Gaussians, which represent the coarse shape of the object. Incorporating the per-Gaussian intrinsics, which encode local details, into the loss formulation is an interesting experiment.

3. Reconstruction of natural objects, e.g. plants, trees

Thanks for the interesting question. In Fig. 3, 5 and 9, we show that our geometry-based model is able to reconstruct certain fine details, such as individual braces at the back of a wooden chair or thin table legs. However, for natural, intricate objects such as plants, geometric approaches generally struggle to reconstruct very fine details. To study the limits of our approach, we will include a comparison with an increased number of 3D Gaussians in the shape model. To further mitigate this issue, our approach can reconstruct the geometry of the object, and additional fine details, such as individual leaves, can be further represented by local textures, as has been done in computer games, or with 3D Gaussian splats.

4. Resolving scale ambiguity

Since our model learns object poses and scales together with a strong scene prior, our model naturally reasons about common scene scales and hence is less prone to ambiguities of individual objects given the scene context.

5. RGB image input

Thanks for the comment. We will go through the manuscript thoroughly and improve the writing and fix minor typos. Eq. 1 defines the overall problem setup of reconstructing multiple objects from a single RGB image ($\mathbf{I}$, L121). Instead of conditioning on the input image directly, we use an image feature extractor $\Theta_i$ and instance segmentation on the input image to extract per-detection image features, which are then used to condition the diffusion model ($\text{feat}_i$, L135, Eq. 6).

6. Gradient backpropagation through point sampling

For sampling we use torch.distributions.MultivariateNormal function with the estimated per-Gaussian centers $\mu$ and covariance matrices $\Sigma$. While the sampling itself is not differentiable, we can compute the gradients on the sampled points. During joint training with the proposed efficient alignment loss, we omit the additional denoising of the intrinsic vectors and the costly rejection of point samples using the shape decoder. Hence, our point sampling formulation is different from the shape decoding of SPAGHETTI [1] or SALAD [2].

7. Textures of objects

Our model does not predict object colors or textures, however, this is an interesting future direction. For visual clarity, we color each object according to its semantic class - we will indicate this in the captions of the figures.

References

• [1] Hertz et al. - “SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation”, SIGGRAPH (TOG) 2022
• [2] Koo et al. - “SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation”, ICCV 2023

1. Clarification of model architecture

Thank you for the feedback regarding the architecture description of our model in Section 3.7. We will improve the clarity of this section in the final paper. In the rebuttal PDF, we have included an architecture figure of the shape model (Fig. 4).

In particular, the pose diffusion model is conditioned on all objects’ image features of a scene simultaneously (Eq. 6 & 7). This setup allows the model to learn a scene prior via ISA by exchanging relational information between the objects and to denoise the individual object poses. During joint training, the entire model takes as input all scene object features (Eq. 6 & 7), estimates the scene context implicitly via ISA, predicts the object poses and the intermediate shape Gaussians of each object. These Gaussians are transformed into world space using the specific object pose to compute the Alignment Loss (Fig. 2, right).

2. Clarification of “Noisy Shape Gaussians” in Figure 1

The “Noisy Shape Gaussians” in Figure 1 depict the anisotropic 3D Gaussians of 3 objects at the beginning of the denoising process. During the backward diffusion process, these 3D Gaussians get denoised to form the intermediate, scaffolding representation of each shape. We will clarify it in the figure and the caption.

3. 3D supervision

Yes, the pose and shape models are first trained with respective 3D supervision before joint fine-tuning. We follow the training scheme of previous works, Total3D [3], Im3D [4], and train the poses on SUN RGB-D [5] and shapes on Pix3D [6]. Hence, our approach learns marginal distributions for layout and shape first. We then fine-tune for the joint distribution by using the provided depth map from SUN RGB-D and the proposed alignment loss.

We agree that training on depth data only poses an interesting and challenging problem setting. In order to directly learn the joint distribution, image-based representation methods, such as EG3D [7], could be employed to learn meaningful shape priors from RGB-(D) data. These could be combined with depth and pose estimation like NOCs [8]. We will address this in the limitation section.

4. Context-learning via attention

Thanks for pointing us to this relevant work, we will include it in the related work section.

5. Surface alignment of Gaussians

In our experiments, we observed that the intermediate 3D Gaussians usually align well with the object’s parts (see Fig. 1 and Fig. 2), leading to sampled points that are close to the surface. For future work, it would be interesting to integrate surfel-based representations [9, 10] as promising alternatives for binding Gaussians to the surface of the object (2DGS [11]).

6. Varying number of scene objects

Our model can in fact handle a varying number of objects in the scene, because the dot-product attention in our ISA formulation is agnostic to the number of objects during train and inference time. In our experiments, we have not experienced negative impacts of false positives/negatives from the image detection backbone, however it is possible to perform NMS on the reconstructed 3D shapes to further filter out false detections.

7. Object pose normalizations

During our experiments, we did not try different normalization factors for the object pose parameters. We will include a comparison with different, e.g., unnormalized, object poses in the final paper.

8. Convergence of disentangled of pose and shape

Thank you for the interesting question. Investigating the convergence properties of the presented disentangled pose and shape formulation is indeed a meaningful addition. We will perform a comparison in a controlled environment, e.g., using synthetic data with full 3D ground truth such as 3D-FRONT [12], and include it in the final paper.

9. Comparison with SOTA single-object methods

We show comparisons for single object reconstruction with InstPIFU [13] in Figure 8.

10. Can it be applied to part-level object reconstruction

In our experiments, the scaffolding 3D Gaussians align well with individual objects parts. For a part-level reconstruction, each vertex of the output shape mesh can be assigned to its nearest 3D Gaussian to get a part-level segmentation of the object. We will include a visualization of that in the final paper.

References

• [1] Hertz et al. - “SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation”, SIGGRAPH (TOG) 2022
• [2] Koo et al. - “SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation”, ICCV 2023
• [3] Nie et al. - “Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image”, CVPR 2020
• [4] Zhang et al. - “Holistic 3D Scene Understanding from a Single Image with Implicit Representation”, CVPR 2021
• [5] Sun et al. - “Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling”, CVPR 2018
• [6] Song et al. - “SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite”, CVPR 2015
• [7] Chan et al. - “EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks”, CVPR 2022
• [8] Wang et al. - “Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation”, CVPR 2019
• [9] Pfister et al. - “Surfels: Surface elements as rendering primitives”, SIGGRAPH 2000
• [10] Zwicker et al. - “EWA volume splatting”, Visualization 2001
• [11] Huang et al. - “2DGS: 2D Gaussian Splatting for Geometrically Accurate Radiance Fields”, SIGGRAPH 2024
• [12] Fu et al. - “3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics”, ICCV 2021
• [13] Liu et al. - “Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes”, ECCV 2022

1. Generalization to unseen/uncommon object categories

Our shape prior is learned across all categories and in our experiments, we have seen that the shape Gaussians are indeed quite stable and align well with geometric parts (Figs. 1 & 2). However, we do rely on strong priors for seen categories, making generalization to unseen categories challenging. This limitation could be mitigated by training on larger shape datasets such as Objaverse [1].

2. Comparison to a retrieval baseline

We include qualitative results of the CAD retrieval method ROCA [2] in the rebuttal PDF (Fig. 3). While the retrieved CAD models are complete by definition, ROCA struggles to capture the correct mode of instances and produces misaligned and intersecting objects. We will include a detailed evaluation and comparison of this baseline in the final paper.

3. Improvements

In contrast to the regression-based baselines, such as Total3D [3], Im3D [4] and Mesh R-CNN [4], we introduce a fully probabilistic approach to model the joint distribution of object arrangements and shapes. As our quantitative and qualitative results show, our formulation leads to more accurate shapes and object arrangements. In particular, for ambiguous and challenging cases, our diffusion prior still yields plausible results where the baselines tend to produce inaccurate geometries and unrealistic poses.

References

• [1] Deitke et al. "Objaverse: A universe of annotated 3d objects.", CVPR 2023.
• [2] Gümeli et al. - "ROCA: Robust cad model retrieval and alignment from a single image.", CVPR 2022
• [3] Nie et al. - “Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image”, CVPR 2020
• [4] Zhang et al. - “Holistic 3D Scene Understanding from a Single Image with Implicit Representation”, CVPR 2021
• [5] Gkioxari et al. - “Mesh R-CNN”, ICCV 2019

Thank you for the rebuttal. My "borderline accept" rating remains the same. I agree with other reviewers that the paper is above threshold.

1. Clarification of shape learning

To support our latent diffusion approach for modeling shapes, we represent 3D shapes following the disentangled formulation of SPAGHETTI [1]. SPAGHETTI learns the disentanglement into 16 Gaussians and per-Gaussian “intrinsics” in a self-supervised way through augmentation with rigid transformations of the shapes. Since the official code only provides the inference part and checkpoints for chairs and airplanes, we re-implement the method according to the paper and train it across all categories on Pix3D [2]. From this trained model, we extract the shape codes $\sigma_i$ for supervision of the shape diffusion model. The shape decoder mentioned in L213 refers to the one of SPAGHETTI. We will add a more detailed description of our re-implementation and training in the paper as well as release the code and checkpoints together with the final paper.

2. Intermediate scene representation

The intermediate scene presentation consists of the per-shape 3D Gaussians from the shape diffusion model and the object pose from the object pose diffusion model. The 3D Gaussians are transformed to world space by the predicted object poses. Following baseline protocols and Sec. 4.2, we train our model in stages and use the respective supervision: The scene prior and object poses are trained on SUN RGB-D [3]; Shape reconstruction is trained on Pix3D [2] using the preprocessed 3D shape codes $\sigma_i$ (see above). The final model is then jointly trained on SUN RGB-D. We do not constrain the 3D Gaussians to be non-overlapping or non-trivial, however by restricting the number of Gaussians (16 in our experiments), this encourages the shape model to distribute the Gaussians effectively in 3D space.

3. Intra-Scene Attention

Since we condition the model on all scene objects simultaneously, the intra-scene attention allows our model to learn a joint scene prior, e.g., properties such as object-object relationships and scene contexts. This is different from methods that individually regress the objects’ pose/shape (MeshRCNN [5], ROCA [6]) or refine the object poses in a second step (Im3D [4]).

4. Notion of cls in Eq. 6

‘cls’ in Eq. 6 refers to the semantic class in L135. We will clarify this in the paper.

5. Figures for ablation studies

We included two additional figures (Fig. 1 & Fig. 2) for qualitative results of our ablation studies in the rebuttal PDF.

6. Additional related works

Thanks for pointing us to additional related works, we are happy to include and discuss them in the paper.

References

• [1] Hertz et al. - “SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation”, SIGGRAPH (TOG) 2022
• [2] Sun et al. - “Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling”, CVPR 2018
• [3] Song et al. - “SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite”, CVPR 2015
• [4] Zhang et al. - “Holistic 3D Scene Understanding from a Single Image with Implicit Representation”, CVPR 2021
• [5] Gkioxari et al. - “Mesh R-CNN”, ICCV 2019
• [6] Gümeli et al. - "ROCA: Robust cad model retrieval and alignment from a single image.", CVPR 2022