Large Spatial Model: End-to-end Unposed Images to Semantic 3D

We thank reviewer 4 (8Lhg) for recognizing the contribution of our paper and offering insightful comments. Please find our response to the feedback below.

[W1]: Ablate the multi-task design choice?
We ablate the “novel view feature synthesis (Eq.4)” and “geometry prediction (Eq.1)” tasks in the following table. The row labeled “no Feat.” indicates the removal of the multi-scale fusion of LSeg features (green part in Point-wise Aggregation of Fig. 2). We observe that incorporating semantic task learning into point-wise aggregation enables the model to lift view-inconsistent feature maps to a view-consistent feature field, while it does not impact view synthesis and depth tasks a lot. We believe the input to “Point-wise Aggregation” contains pixel-wise point maps (x, y, z) and RGB (r, g, b) from the support images, which already provide the complete observation within the input views for interpolating novel views and depth prediction. Thus, additional incorporation of feature maps does not provide visual cues to enrich the radiance field. Removing geometric supervision leads to the model diverging quickly, which means pixel-aligned depth regression helps the model recognize correspondences across input views, guiding the model's learning when it is unaware of the camera parameters. The experiment setting follows Table 4 of the main draft.

[W2]: Update Figure 5, and the pose divergence for the novel views?
We will include the source (support) views in the revision; thank you for the suggestion. We specify the rendered novel camera positions and orientations by interpolating between the first and second reference (or support) images for free-view exploration (see the supplementary webpage). Specifically, the orientation interpolation is conducted using Spherical Linear Interpolation [1] between the two sets of quaternions, while the position interpolation is conducted using linear interpolation.

[W3]: More accurate terminology?
We will take your suggestion into account and revise it accordingly.

[Q1]: Typos and highlighting the table?
We will fix it.

[Q2]: Why is “+ Multi-scale Fusion” indented in Table 4?
Indented row (Multi-scale Fusion) means it is based on the rows above (Fuse LSeg Feat. and Fuse Encoder Feat.), indicating that we apply multi-scale fusion to both. We have clarified this part in the revision.

[Q3]: Input view or novel view in Figure 3?
They are novel views.

[Limitation]
Thanks for pointing this out. Our method, along with all the adopted methods (pixelSplat, Feature-3DGS, NeRF-DFF), belongs to the category of reconstruction models, which focus on building the 3D based on input observations. A discussion about extending to generation for synthesizing new viewpoints with large pose divergences will be included in our revision.

Reference:
[1]. Slerp from wikipedia (we cannot provide the direct link in author response)

We thank reviewer 3 (MCB6) for recognizing the contribution of our paper and offering insightful comments. Please find our response to the feedback below.
[W1.1]: Reasons for deviation from Replica, and performance discrepancy.
The reasons for the deviation are threefold:

1). The processed Replica datasets (by Feature-3DGS) lack ground-truth depth maps for geometric accuracy. Feature-3DGS uses COLMAP for SfM results, leading to misaligned coordinate systems. 2). Feature-3DGS uses LSeg as GT for evaluating semantic quality, while we use annotated GT semantic labels. 3). Replica datasets are simulated under ideal environments, whereas our draft focuses on real-world textures, lighting changes, and sensor noise.

The performance discrepancy of Feature-3DGS stems from resolution and semantic ground-truth differences. Our experiments follow the generalizable NVS method, pixelSplat, using 256x256 resolution and cropped boundaries, reducing the valid field-of-view. We also use annotated ScanNet for semantic segmentation, rather than LSeg as ground-truth.

[W1.2]: Results on Replica Dataset with explanation.
Comparing our methods with NeRF-DFF and Feature-3DGS is challenging due to their assumptions of dense views, long optimization steps, and precomputed camera poses, while ours uses sparse and pose-free views. pixelSplat also assumes known camera parameters. This places our method at a disadvantage, but Tab. 1 in our submission demonstrates its strength.

To further validate, we compare with Feature-3DGS on the Replica dataset, using Semantic-NeRF [1] preprocessed data with annotated RGB-D-Semantics (256x256 resolution). We randomly sample 2,000 points from GT depth maps for initializing Feature-3DGS and use 70 images for training and 10 for testing, following their setup.

LSM generalizes well from the real-world ScanNet test set (25.44dB) to the simulated Replica test set (23.10dB) in zero-shot, even without GT camera parameters. Our model builds the feature field in a feed-forward pass, achieving high quality. Notably, our method achieves the best geometric accuracy without using epipolar attention (pixelSplat, need pose) or requiring test-time optimization with dense views and GT camera parameters (Feature-3DGS, need pose).

To be more aligned with our assumed setting, which are more practical in real applications, we proposed to benchmark the performance of investigated baselines under the sparse view setting. We reduce the support view number for all methods from 70 to 20, while preserving the same test sets. In the table below, Feature-3DGS drastically decrease the PSNR with degraded geometry and semantic accuracy.

[W1.3]. Training details.
We included training details in Sec. 4.1. We use the default configuration to train Feature-3DGS and NeRF-DFF. For pixelSplat, we use its pretrained checkpoint and code without any modifications to perform the evaluation.

[W2]:Detailed module design.
We will add a diagram to illustrate module interconnections in the revision. Additionally, we plan to release the source code for reproduction.

[W3]:Why use camera parameters in L117?
They are needed in training and evaluation to obtain the ground-truth point maps, and specify the target view. They are not required for inference.
[Q1]: Additionally, a more diverse set of results should be displayed.
Please refer to supplementary webpage for ten free-view videos. We also incorporate additional comparisons in the Fig.3 of the attached PDF.

[Q2]:Module Timing? Image resolution?
We study the computational cost (time) of each module by inferring 1,000 times with the model and obtaining the average inference cost for each model.

The tested resolution (256x256) is aligned with the generalizable 3D-GS method pixelSplat. pixelSplat is slightly faster than our framework 0.06s while it requires additional steps to run Structure-from-Motion (e.g., 20.52s for each scene on ScanNet on average) to obtain the camera parameters.

We thank reviewer 2 (fir7) for recognizing the contribution of our paper and offering insightful comments. Please find our response to the feedback below.

[W1] Discussion w/ SRT and RUST.
Scene Representation Transformer (SRT)[1] and RUST[3] have pioneered the exploration of representing multiple images as a "set latent scene representation" (OSRT[2] utilized Slot Scene Representation) and generating novel views even in the presence of flawed camera poses or without any pose information. In contrast, our method targets a holistic model for semantic 3D reconstruction from unposed images, using point-based representation for efficient rendering speed. We will include this discussion in the revision.

[W2] Table 4 suggests little impact on the results of NVS.
To formulate the well-represented radiance field, the Point-wise Aggregation module takes the pixel-aligned (x, y, z, r, g, b) as input (line 153) for regressing point-wise attributes, which already contain the complete appearance observation within the input views for interpolation. Thus, additional incorporation of feature maps does not provide visual cues to enrich the radiance field. However, the Point-wise Aggregation module design is important as it enables the lifting of features from inconsistent stereo image features to consistent feature fields, and the Multi-scale Fusion further improves the lifting accuracy.

[Q1] Novel view semantic segmentation with LSeg?
We notice that LSeg fails to produce view-consistent segmentation results but generates reasonably good per-view segmentation results. Our hypothesis is that LSeg is a well-trained 2D model for language-driven semantic segmentation, serving as a foundation for our method to understand 3D. Our method lifts the view-inconsistent multi-view feature maps from LSeg into a consistent 3D feature field in a zero-shot manner, utilizing the Point-wise Aggregation module. See the figures in the attached PDF (Fig.2) to see the consistency difference.

[Q2] High multiview depth accuracy induced by aligning scene scale?

Yes, the improved geometry accuracy stems from the alignment and combination of scenes, and our training data consists of dense depth annotations of various scenes for supervision.

[Q3] How to define novel cameras? Can they deviate from the reference one?
Thank you for acknowledging the good rendering quality. The rendered novel camera positions and orientations are determined by interpolation between the first and second reference (or support) images. Specifically, orientation interpolation is conducted using Spherical Linear Interpolation [4] between the two sets of quaternions, while position interpolation is conducted using linear interpolation.
We acknowledge that both our method and pixelSplat are reconstruction methods that build the radiance field using pixel-aligned Gaussians for interpolation, and thus cannot perform generation tasks that significantly deviate from the reference images.

Reference:
[1]. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations, CVPR 2022
[2]. Object Scene Representation Transformer, NeurIPS 2022
[3]. Rust: Latent neural scene representations from unposed imagery, CVPR 2023
[4]. Slerp from wikipedia page (we cannot provide the direct link in author response)

We thank reviewer 1 (K6pm) for recognizing the contribution of our paper and offering insightful comments. Please find our response to the feedback below.

[W1]: No significant new problem has arisen and novel solutions proposed?
We acknowledge that our work builds upon the contributions of many giants, such as DUSt3R [1] (end-to-end point map regression model), pixelNeRF [2] (one of the first generalizable NeRFs), pixelSplat [3] (one of the first generalizable 3D-GSs), and the concept of large-scale training. However, we respectfully disagree with the assertion that "no significant new problem has arisen and novel solutions proposed."
We target a very practical yet under-explored scenario: end-to-end semantic 3D reconstruction directly from images. While previous literature reduces this problem to “Computing Camera Parameters” followed by “Training 3D Representation”. We, for the first time, unify these problems into one differentiable vision system.
Technically, while there are many alternative design choices, our method successfully integrates these components into an end-to-end trainable system using a standard Transformer architecture, demonstrating the feasibility of future scalable 3D training.
We kindly suggest that the methods and ideas presented in this paper carry significant research value that can impact future 3D deep learning studies.

[W2]: Evaluation of generalizable methods on new datasets.
To avoid any possibility of overfitting with existing generalizable methods and considering the suggestion by Reviewer #MCB6, we adopt the Replica dataset [3], which is a photorealistic simulated 3D dataset containing accurate annotations of RGB, dense depth maps, and semantic maps for thorough evaluation. Specifically, the LSM generalizes well from the real-world ScanNet test set (25.44dB) to the simulated Replica test set (23.10dB). LSM also produces the best depth estimation metrics and is the only generalizable method that can enable 3D semantic segmentation. Splatter Image is an ultra-fast monocular 3D object reconstruction method using 3DGS, and we utilize the provided checkpoint for evaluation. However, while Splatter Image addresses object 3D reconstruction well when the background is masked out, it cannot handle scene-wise reconstruction well with complex backgrounds (please see the attached Fig.1).

[W3]: Improve the presentation.
We will highlight the input and output formats in the methodology section in a future version.
Although our method architecture exhibits some similarity to generalizable Gaussian models, we emphasize that the investigated problem is very different. pixelSplat focuses on the standard NVS setting, where calibrated images are assumed and the emphasis is solely on image synthesis. In contrast, our overarching goal is to develop a versatile 3D foundation model that unifies scene reconstruction and understanding with uncalibrated images, which is more aligned with real-world applications to deploy a single model. We will adjust the title to emphasize our unification of 3D tasks according to Reviewer 8Lhg's suggestions. Thank you for the feedback.

[W4] Missing citation.
Splatter Image [4] presents one of the first solutions for ultra-fast monocular 3D object reconstruction using 3D Gaussian Splatting as a representation. It takes a single or two object-wise images as input for reconstructing the 3D representation without test-time optimization, while demonstrating promising reconstruction accuracy. We will incorporate a detailed discussion in our revision.

[Q1] Replace encoder using monocular depth?
The “Dense Geometry Prediction” cannot be replaced with a monocular depth estimator. Our method takes unposed images as input to build global 3D point maps with versatile attributes for many 3D problems. It requires a stereo depth estimator to unify the point maps from two viewpoints into the same coordinate system.

[Q2] Can this model perform generation, like Free3D?
Our work reconstructs 3D scenes based on unposed images, similar to 3D-GS, but without the need for COLMAP pre-computation. The supervision comes from the new view interpolated between the training views, meaning it is not capable of extrapolating viewpoints (generation). We will clarify this point and discuss the potential to extend the proposed framework into a generative approach in the future work section.

[Q3] What’s “language-driven segmentation”?
"language-driven segmentation" means segmenting areas based on a set of language descriptions (e.g., “wall,” “floor”). This allows us to associate each Gaussian with text to render the 2D semantic map. Please refer to [5] for more detailed explanations.

Reference:
[1]. DUSt3R: Geometric 3D Vision Made Easy. CVPR 2024
[2]. pixelnerf: Neural radiance fields from one or few images. CVPR 2021
[3]. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. CVPR 2024
[4]. Splatter Image: Ultra-Fast Single-View 3D Reconstruction, CVPR 2024
[5]. Language-driven Semantic Segmentation, ICLR 2022