MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild

We thank all reviewers for their time and constructive feedback. We are encouraged by the generally positive comments, including:

• This paper addresses a valuable and challenging problem of sparse, in-the-wild reconstruction (PYHN).
• Our approach is technically sound (PYHN). Specifically, the semantic depth alignment is compelling (P8xk), and the multi-view geometry-guided supervision is interesting (B6Ss) and clever (P8xk, sq1N).
• The experiments are comprehensive (sq1N, P8xk, PYHN) and consistent with the claims (B6Ss), demonstrating the effectiveness of our approach with clear improvements (PYHN, sq1N, P8xk).
• Overall, the paper is well written and easy to follow (PYHN, B6Ss).

We will also fix the typo that the limitations were discussed in Section 5 instead of 6, as noted by reviewers P8xk and B6Ss.

Please find our responses to the reviewer' specific questions and comments below.

---

(1) Variance of improvements

We note that in Table 1, the pixel loss and feature loss each bring less improvement in LPIPS and DSIM than PSNR and SSIM. By adding additional constraints to multi-view optimization, we are trading off the ability to remove artifacts and the risk of smoothing the geometry. Since perceptual quality measures like LPIPS and DSIM are sensitive to high-frequency content, noisy/artifact images can get a boost in these metrics. We don't find significant variance in our experiments. Variance across three runs are 0.0045, 1.35e-5, 5.76e-5, 6.0e-6 for PSNR, SSIM, LPIPS, and DSIM respectively.

---

(2) Semantic scaling vs. single scaling

Please find the ablation study of semantic scaling vs. single image-level scaling in Section A.1.2 (Initialization Comparisons) in our appendix, which includes point cloud visualization, quantitative and qualitative comparisons. For convenience, a partial table is pasted below. The metrics are reported as average on the sparse MipNeRF 360 dataset.

---

(Q1) Semantic scaling validation

Thank you for your suggestion. We perform the recommended validation on the bonsai scene. Specifically, we use the Multi-View Stereo depth from dense views as GT depth estimates. A single per-frame estimate of scale results in a mean absolute error of 1.94, and our per‑semantic‑region scaling reduces this to 1.07. We also visualize the error map, which shows that the depth continuities around object boundaries exhibit higher error, while the depth inside semantic objects is more coherent and accurate after alignment. We attribute this to depth ambiguities from monocular depth that our piece-wise semantic alignment effectively mitigates. We will add this analysis and visualization to the revised manuscript.

---

(Q2) "Clarification about virtual view regularization"

Thank you for raising this point. In standard 3DGS optimization, training image pixels are only supervised along the rays that originate from their own camera center. With sparse views, this leaves large regions of the scene under-constrained. Introducing virtual cameras allows re-using the relatively accurate training views results and thereby enhancing multi-view consistency for constraint. For example, the color of a well-rendered training pixel is back-projected with its reasonably accurate depth and re-projected into the virtual view. If a "floater" Gaussian lies somewhere along the newly created ray at the virtual camera, it will alter the rendered color at the re-projected pixel, producing a photometric error. Minimizing this multi-view error as self-regularization forces the optimizer to suppress such floaters or inconsistencies.

We thank all reviewers for their time and constructive feedback. We are encouraged by the generally positive comments, including:

• This paper addresses a valuable and challenging problem of sparse, in-the-wild reconstruction (PYHN).
• Our approach is technically sound (PYHN). Specifically, the semantic depth alignment is compelling (P8xk), and the multi-view geometry-guided supervision is interesting (B6Ss) and clever (P8xk, sq1N).
• The experiments are comprehensive (sq1N, P8xk, PYHN) and consistent with the claims (B6Ss), demonstrating the effectiveness of our approach with clear improvements (PYHN, sq1N, P8xk).
• Overall, the paper is well written and easy to follow (PYHN, B6Ss).

We will also fix the typo that the limitations were discussed in Section 5 instead of 6, as noted by reviewers P8xk and B6Ss.

Please find our responses to the reviewer' specific questions and comments below.

---

(2) More discussion on limitation of MS-GS

Handling transient objects in 3D reconstruction under sparse-view conditions is severely under-constrained. While recent methods leverage uncertainty masks to remove transients and allow other observations to fill in the blank, oftentimes, no other observations exist under a sparse setting; therefore, the transient regions remain under-constrained regardless. In our experience, adding a masking mechanism to handle transients often leads to worse results.

---

(3), (Q2), (Q3) Non-Lambertian scenes and examples where MS-GS struggles

Generalizing to non‑Lambertian surfaces remains difficult even with dense views. Although we mitigate occlusion by comparing rendered and warped depths, the resulting mask can be imperfect, and specular highlights are smoothed or averaged out, as seen in our examples. It would be interesting to analyze how different techniques, such as explicitly modeling the light [1] and surface reconstruction [2], can be combined with our framework to further improve in‑the‑wild geometry reconstruction under sparse‑view conditions. More complex modeling requires more observations, which will further exacerbate the sparsity issue. This work mainly focuses on generalized appearance modeling.

In addition, because MS-GS favors more accurate semantic regions for alignment, pixels that lie outside the semantic masks or are missed by imperfect segmentations lack dense initialization. Examples show that the empty regions can manifest as artifacts without geometric support.

[1] Sun, Jia-Mu, et al. SOL-NeRF: Sunlight modeling for outdoor scene decomposition and relighting. SIGGRAPH Asia 2023 Conference Papers. 2023.

[2] Huang, Binbin, et al. 2d Gaussian splatting for geometrically accurate radiance fields. ACM SIGGRAPH 2024 conference papers. 2024.

---

(4) Clearer explanation of the coordinate alignment

Thank you for pointing this out. We agree that the notation around Eq. (9) was ambiguous and will revise it. We intend $P_{\text{cam}}\in\mathbb{R}^{4\times3}$ to be a stack of four 3-D points per camera (one camera center and three points sampled one scaled unit $s$ along the unit rotation-axis directions), where each row is a point. By incorporating these additional rotational points, we achieve more accurate camera alignment in both rotation and translation.

The goal of coordinate alignment / in-the-wild setup is to disentangle training and test views in registration. Conventionally, they are registered with one run of Structure-from-Motion (SfM) so that test images are involved in the calibration, which leads to unrealistically better registration and denser initialization. In our setup, train views are first registered in one run, resulting in the coordinate system $C_1$. A separate run of SfM on train and test views results in the coordinate system $C_2$ because SfM reconstructs the scene in a different coordinate system each time. We compute a transformation between the shared train views, which are used to transform the test view from $C_2$ to $C_1$. In this way, we simulate the real-world scenario, where camera poses and 3D points in $C_1$ are estimated from only train views while having the test camera poses in the same coordinate system for evaluation.

---

(5) Minor errors

Thank you for pointing this out. We will fix it and perform a more rigorous proofreading of the revised manuscript.

---

(Q1) Generalization to different depth models

In addition to Marigold [1], we have conducted an analysis of the semantic depth alignment vs. image-level alignment with Depth Anything v2 [2] and MiDaS 3.1 [3] on the sparse MipNeRF360 dataset. The metrics are reported as the average. As shown in the table below, our semantic depth alignment approach consistently improves the results, confirming its generalizability to different depth estimators.

[1] Ke, Bingxin, et al. Repurposing diffusion-based image generators for monocular depth estimation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

[2] Yang, Lihe, et al. Depth anything v2. Advances in Neural Information Processing Systems 37 (2024): 21875-21911.

[3] Ranftl, René, et al. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence 44.3 (2020): 1623-1637.

---

More visualization results

Thank you for your suggestion. We agree that more visualization results can assist in understanding of the method and its effectiveness. We have prepared more qualitative comparisons across scenes to demonstrate that MS-GS achieves better appearance rendering compared to baselines. We will add the results and a visual overview of the proposed dataset to the revised paper.

We thank all reviewers for their time and constructive feedback. We are encouraged by the generally positive comments, including:

• This paper addresses a valuable and challenging problem of sparse, in-the-wild reconstruction (PYHN).
• Our approach is technically sound (PYHN). Specifically, the semantic depth alignment is compelling (P8xk), and the multi-view geometry-guided supervision is interesting (B6Ss) and clever (P8xk, sq1N).
• The experiments are comprehensive (sq1N, P8xk, PYHN) and consistent with the claims (B6Ss), demonstrating the effectiveness of our approach with clear improvements (PYHN, sq1N, P8xk).
• Overall, the paper is well written and easy to follow (PYHN, B6Ss).

We will also fix the typo that the limitations were discussed in Section 5 instead of 6, as noted by reviewers P8xk and B6Ss.

Please find our responses to the reviewer' specific questions and comments below.

---

(1) Comparison with SPARS3R

We thank the reviewer for referencing SPARS3R, which is a recent concurrent work that only focuses on single-appearance sparse reconstruction. Compared to SPARS3R, we propose to use monocular depth vs. two-view point map for initialization. We note that 1. our approach is more scalable and enables high-resolution depth, whereas MAST3R [1] can go OOM due to its quadratic pairwise bottleneck and produces low-resolution depth. 2. Monocular depth estimation, e.g., from Marigold [2], is much sharper, while the prediction from MAST3R is coarse and smooth, as MAST3R needs to meet two-view constraints. 3. MS-GS also incorporates virtual view supervision to regularize the scene geometry and appearance under the sparse-view setting.

We also include the results of comparisons between MS-GS and SPARS3R below. The metrics are reported as the average on the sparse MipNeRF360 dataset.

We note that the result of MS-GS reported in Table 2 in our main manuscript is without virtual view regularization. Our dense initialization approach outperforms SPARS3R in PSNR and SSIM, demonstrating the advantage in using monocular depth estimation. Our virtual view regularization further improves the result.

[1] Leroy, Vincent, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[2] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492–9502, 2024.

---

(2) Sparser setting

We note that in-the-wild reconstruction with sparse inputs and various appearances is a very challenging problem. Compared to the single appearance setup, more data is needed to disentangle different appearances and reconstruct a holistic scene. As for the Phototourism dataset, we use 2.62%, 2.41%, and 1.18% of images compared to the original setting for "Brandenburg Gate", "Sacre Coeur", and "Trevi Fountain" scenes, respectively. The problem worsens as our newly introduced drone dataset represents 360-degree scenes, where the cameras are more divergent and have limited overlaps.

To demonstrate the effectiveness of our methods in sparser settings, we conduct the experiments in the 12-view setting for the sparse drone dataset, where each appearance has 3 images. The metrics are reported as the average over the scenes. As shown in the table below, MS-GS continues to outperform the SoTA methods, and the performance gap widens compared to the 20-view setting. We will add these experiments and results to the revised manuscript.

---

(Q1) In this setup, is pose estimation performed separately for the training and test views?

Yes, the train and test views are registered separately and aligned together. Conventionally, they are registered with one run of Structure-from-Motion (SfM) so that test images are involved in the calibration, which leads to unrealistically better registration and denser initialization. In our setup, train views are first registered in one run, resulting in the coordinate system $C_1$. A separate run of SfM on train and test views results in the coordinate system $C_2$ because SfM reconstructs the scene in a different coordinate system each time. We compute a transformation between the shared train views, which are used to transform the test view from $C_2$ to $C_1$. In this way, we simulate the real-world scenario, where camera poses and 3D points in $C_1$ are estimated from only train views while having the test camera poses in the same coordinate system for evaluation.

We thank all reviewers for their time and constructive feedback. We are encouraged by the generally positive comments, including:

• This paper addresses a valuable and challenging problem of sparse, in-the-wild reconstruction (PYHN).
• Our approach is technically sound (PYHN). Specifically, the semantic depth alignment is compelling (P8xk), and the multi-view geometry-guided supervision is interesting (B6Ss) and clever (P8xk, sq1N).
• The experiments are comprehensive (sq1N, P8xk, PYHN) and consistent with the claims (B6Ss), demonstrating the effectiveness of our approach with clear improvements (PYHN, sq1N, P8xk).
• Overall, the paper is well written and easy to follow (PYHN, B6Ss).

We will also fix the typo that the limitations were discussed in Section 5 instead of 6, as noted by reviewers P8xk and B6Ss.

Please find our responses to the reviewer' specific questions and comments below.

---

(Q1) More clarity on the newly introduced dataset

To obtain the appearance embeddings of a test image from the Phototourism [1] dataset, prior setups either optimize on half of the test image or use a network to extract features from the entire test image. Both of these approaches lead to the utilization of test images during evaluation. For our dataset, the multi-view, multi-appearance setup allows the correct appearance to be rendered based on metadata/timestamp, not pixel-wise information from the test image. For example, a snowy test image can be rendered with the appearance embedding of a snowy training image by relating their timestamps. This setup is also faster than optimizing half of the test image. Additionally, our dataset contains scenes with 360-degree coverage by perspective cameras, whereas Phototourism is only covered by face-forward images. We will include this clarification in the revised manuscript.

[1] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers, pages 835–846. 2006.

---

(Q2) Runtime and memory analysis

Thank you for your suggestion. We report the runtime and peak GPU memory usage of each scene in the table below.

The training time for 15,000 iterations is increased by 1.7 minutes on average with dense initialization compared to sparse initialization. While the optimization of more points is more costly, the pruning process can quickly remove most of redundant dense initialization. The peak GPU memory usage is increased by 1.2 GB on average. With our dense initialization, 3DGS converges faster to the final number of Gaussians and photometric loss.