Robust Neural Rendering in the Wild with Asymmetric Dual 3D Gaussian Splatting

Thank you for your thoughtful feedback and valuable suggestions! We are sorry that we cannot include images showing the stochastic artifacts according to new rebuttal rules, but we are committed to including more visual evidences in the next step.

---

W1: My major concern is that some statements in this paper require further visual analysis to support, especially the stochastic nature of artifacts, which is the core idea in this paper. For example, I wanna see more examples across various scenes and different runs to demonstrate the appearance of artifacts is stochastic. The visual comparison between applying the same mask or different masks on two parallel models is also necessary to show whether this design can avoid converging toward the same reconstruction errors. Does the above conclusion also work for EMA strategy? A visual comparison is also needed here.

We sincerely appreciate the reviewer’s insightful comments. Unfortunately, unlike NeurIPS last year, this venue does not provide a mechanism to submit a rebuttal PDF with visual evidences.

Regarding the stochastic nature of artifacts, we included visual examples in the main paper. As shown in Figure 1, different runs of 3DGS on the same scene (with only the view order randomized) result in different artifacts, particularly in uncertain regions. The mutual consistency loss helps suppress these artifacts in both models. On one hand, the shared static regions remain consistent and act as a strong regularizer. On the other hand, the differing artifacts in uncertain areas provide complementary supervision signals, allowing regions affected by artifacts in one model to be recovered by the other.

For effect of distinct masking strategies, Table 4 in the main paper presents a quantitative comparison. The performance degrades when both models use the same mask, for both GS-GS and EMA-GS settings. This supports our claim that using separate masks helps prevent convergence to the same erroneous reconstruction patterns.

We agree that linking quantitative results to visual evidence would strengthen our conclusions. Therefore, we are committed to including additional visual comparisons across different scenes, masking strategies, and framework settings in the supplementary materials.

---

Q1: For Eq.(7), will it still work in the case that the reconstructed 3D scene overfits to the distractors? If not, how does this method avoid overfitting to distractors with 3D Gaussians floating in the sky?

Yes, Eq.(7) remains effective even if the 3D reconstruction overfits to distractors.

When the model overfits to distractors, the rendered regions corresponding to distractors will exhibit noticeable deviations from the ground truth. These discrepancies result in low similarity scores, which serve as targets for the learnable mask as shown in Eq.(7). As a result, the mask is driven toward zero in those regions, effectively suppressing the influence of distractors.

For floating Gaussians in the sky, we adopt the pruning strategy from the original 3DGS implementation to remove transient or oversized primitives, preventing their accumulation in regions without valid geometry.

Thank you for the insightful review. We’re grateful for your acknowledgment of our key ideas and experimental efforts. We will thoroughly respond to the concerns below.

---

W1: However in practice, the way these reconstructions are kept different is by introducing more losses to perform the same task of transient mask estimation.

We acknowledge the reviewer’s observation. In our design, we use two distinct masking strategies for each branch of the dual 3DGS model. This explicitly enforces asymmetric supervision over dynamic regions, promoting diversity between the models.

At the same time, we apply a mutual consistency loss to encourage both models to agree in the static regions. This loss also enables each model to recover information that may be suppressed in the other due to masking, helping to mitigate confirmation bias arising from potentially inaccurate masking strategies. In other words, areas blocked in one branch can often be reconstructed using guidance from the other avoiding same masking error.

Overall, our approach maintains divergence in dynamic regions while enforcing consistency in static regions, enabling both robustness and completeness. In Eq. (8), the first two terms correspond to reconstruction under separate masks, while the middle terms encode cross-model consistency, supporting static region reinforcement and dynamic region recovery.

---

W2: The paper does not attempt to meaningfully explain how it differs from previous methods, and instead describes the entire literature of transient NeRF/3DGS as "heuristic strategies". This description is very vague and could be used to describe the proposed method too, which is instead called a "principled framework". For example, the estimation of multi-cue adaptive mask is very "heuristic".

We refer to existing methods as heuristic based on the design choices they make and the lack of formal principles guiding their generalization. For example, RobustNeRF introduces custom loss functions to suppress noisy training signals. However, this is achieved using several hand-crafted rules, for instance, treating residuals as inliers only if they fall below a certain percentile within a certain-size local neighborhood. While effective empirically, such rules are ad hoc and task-specific, potentially compromising stability when conditions vary.

In contrast, our Dual 3DGS framework is grounded in the principle of mutual consistency. It explicitly leverages the assumption that transient artifacts are random across training runs, guiding the training process toward stable and generalizable representations. This principle-driven formulation allows our method to generalize better to diverse scenes.

Nevertheless, we acknowledge that our approach still involves certain heuristic components. To improve clarity, we will revise the manuscript to replace vague terms like ''heuristic'' and ''principled'' with more specific and objective descriptions of the methodological differences.

---

W3: In fact, some of these works have a fairly similar idea, such as SimpleNeRF which also uses a simpler model to regularize the main one, or HybridGS which uses an additional 2DGS to separate moving elements. To be clear, these methods are not exactly identical, but the paper would benefit from discussing these differences in detail and why they matter.

Thank you for your advice on more discussions about related methods.

Our approach differs fundamentally from HybridGS in both design and training. HybridGS separates static and dynamic content using two models (3DGS for static, 2DGS for dynamic) and requires a staged training process with a learnable blending mask. In contrast, our method uses a dual 3DGS setup with mutual supervision to improve robustness against dynamic noise, all within the standard 3DGS training pipeline. While both methods use masking, HybridGS blends outputs based on 2DGS-derived uncertainty, whereas we apply two distinct masking strategies to reduce confirmation bias from a single, potentially inaccurate mask.

SimpleNeRF focuses on few-shot 3D reconstruction with sparse input views, using multiple augmented NeRF models (parallel and serial) to provide additional supervisory signals. In contrast, our work tackles robust in-the-wild reconstruction by employing two parallel 3DGS models with mutual consistency regularization, which reinforces static regions while suppressing dynamic artifacts through complementary representations. Additionally, SimpleNeRF requires about 22 hours of training per scene, whereas our asymmetric dual 3DGS completes training in under one hour.

We will elaborate more on these differences in the main paper.

---

Q1: It is a bit counter-intuitive that the dual-models would perform better than a single model combining both masking strategies (Single w/ Mh/s). What is the interpretation of this result?

Both the dual-model settings (lines 5 and 12 in Table 4) and the single model with alternating masks (line 1) utilize two distinct masks. However, a key difference lies in the mutual consistency loss used in the dual-model setup. This loss reinforces reliable signals from static regions and enables one model to help recover regions suppressed by the mask in the other.

When the mutual consistency loss is removed (lines 9 and 16 in Table 4), the performance of the dual-model framework drops and approaches that of the single model, effectively becoming a simple ensemble.

---

Q2: There are many other ways to keep the models different, and using two different losses for the same goal of masking transient objects is not the most intuitive. Have you experimented with other ideas, such as using different initializations or training on different subsets of images?

Yes, we have explored several strategies to introduce diversity between the models. For instance, we experimented with random dropout, where a subset of 3DGS primitives is randomly frozen during optimization, and random mixup, where the ground-truth image is blended with the rendered output for a given view. These methods are evaluated in lines 10 and 11 of Table 4 in the main paper. The results show that, although they introduce randomness, their effectiveness is limited compared to the alternating masking strategy.

We considered using different initializations or subsets of views but ultimately chose not to pursue these approaches. Because the original 3DGS paper demonstrated that deviating from COLMAP-based initialization leads to significant performance degradation. Similarly, training with only partial data adversely affects reconstruction quality.

---

Q3: The term "frame-independent" is used with and without dash. It also does not exactly convey the meaning intended.

Thank you for pointing this out. The use of "frame-independent" in Line 145 is a typographical error, and we intended to write "frame-dependent". In our work, the appearance modeling is actually view-dependent, as we assign a learnable global appearance vector to each input view and jointly optimize it with the rest of the 3DGS model. The term "frame" was intended to mean "view," but we agree that "view-dependent" is a more accurate and consistent term. We will revise the terminology throughout the paper accordingly.

---

Q4: In the EMA section, it is unclear at the beginning that the main model uses both masking strategies. It should be mentioned early.

In the EMA section, our primary focus was to highlight the computational efficiency introduced by the EMA proxy design, and the masking strategy was not emphasized upfront. Although the alternating masking strategy is described later in the subsection, we agree that providing a brief mention of it earlier in the section would give readers a clearer overview. We will revise the text to introduce the masking strategy at the beginning for better clarity.

Dear Reviewer b1d5,

Thank you for your thoughtful review and constructive feedback. As the discussion phase draws to a close, we’d like to check if our responses have fully addressed your concerns. If there are any remaining questions or points that need clarification, please feel free to let us know.

Thank you.

Thank you for the thoughtful and encouraging review. We truly appreciate your recognition of the paper’s clarity, technical depth, and contributions, as well as your constructive feedback on areas for improvement.

---

W1: Details are scant in the description of the Multi-Cue Adaptive Mask.

Due to space constraints in the main paper and our focus on presenting the dual-model framework and its efficient variant, we chose to include the detailed formulation of the Multi-Cue Adaptive Mask in the supplementary material. For better clarity, we also provide an expanded explanation in our response to Question 1 below.

---

W2: The authors observation that retrained models tend to disagree in areas of uncertainty is simply empirical and could have been better motivated.

We observe that randomness in the training process, particularly in the order in which views are fed into the 3DGS optimization, can lead to stochastic visual artifacts. While our paper does not provide a formal theoretical justification for this phenomenon, the underlying intuition is reasonable: if a single noisy view is introduced late in training, its influence may not be sufficiently corrected and could result in more prominent artifacts. Conversely, when introduced earlier, the model may better learn to ignore or compensate for the noise. These inconsistencies motivate our design of a dual 3DGS framework, where two models supervise each other. Since they are trained on the same scene, their shared static regions remain consistent, while the differing artifacts in uncertain areas provide complementary supervision signals, allowing regions affected by artifacts in one model to be recovered by the other.

---

Q1: Can you elaborate on the technique of the multi-cue adaptive mask? I'm a bit confused - are "static" regions always included? note that semantically static objects could still be considered as outliers due to difficult to model specular reflections. Are then only the "dynamic" regions considered for the union of the other cues?

(1) Multi-Cue Adaptive Mask

The Multi-Cue Adaptive Mask is a hard binary mask derived from four cues: semantic segmentation, stereo correspondence, pixel-level residuals, and feature-level residuals.

We first segment each image into semantic regions using Semantic-SAM [1], which serve as units for aggregating cues.

For stereo correspondence, we use SIFT matches from COLMAP [2]. A pixel with matches in more than three views is considered a reliable static point ($\mathbf{S}_{\ge 3}$ in Algorithm 1 of the supplementary material). Regions with many such points are unlikely to be distractors. We also compute pixel-level ($L_1$ loss) and feature-level (DINOv2 cosine distance) residuals between rendered and ground-truth images. High residuals indicate distractors.

Each region is classified as a distractor if: (1) Its average residual exceeds the global mean, and (2) Its static point density is below the global mean. Distractor regions are excluded by assigning a mask value of zero. We take the union of all such regions and subtract them from an all-one mask.

(2) Specular Reflections

Following the original 3DGS, we use spherical harmonics (Eq. (2) of our main paper) to model view-dependent appearance variations, including specular reflections. When specular effects are mild, our method remains effective. In rare cases where specular reflections are severe and cause abrupt appearance changes across nearby viewpoints, such regions may be treated as outliers by our masking mechanism. However, this scenario is uncommon when sufficient training views are available. We acknowledge this limitation and plan to explore more robust handling of such extreme cases in future work.

[1] Li et al. - 2025 - Segment and Recognize Anything at Any Granularity

[2] Schonberger and Frahm - 2016 - Structure-From-Motion Revisited

---

Q2: How is the self-supervised mask regularized against collapsing to zero?

The learnable mask is optimized to reflect the feature residual between the rendered and ground-truth images. This residual only becomes zero when the features are orthogonal, which is a rare occurrence. Therefore, the target generally remains non-zero throughout training, making mask collapse unlikely.

In addition, we include a warm-up stage where the 3D representation is sufficiently established. This ensures that in static regions, where the rendered and ground-truth features are highly similar, the residual provides a stable and meaningful supervision signal to guide mask learning.

Lastly, the mask is learned independently for each view. If a mask temporarily collapses in certain regions for one view, signals from other views can still contribute effectively, since multiple views are jointly used in the 3DGS optimization process.

While we did not observe mask collapse in our experiments, we believe including a regularization term is beneficial. We will try an explicit regularization loss that penalizes excessively small mask values (e.g., L2 loss targeting a moderate value), encouraging the mask to remain active and informative throughout training.

Dear Reviewer REjw,

We sincerely appreciate your detailed review and helpful suggestions. As the discussion period comes to an end, we want to confirm whether our rebuttal and follow-up responses have resolved your concerns. If there are any outstanding issues that still need clarification, we’d be glad to elaborate further.

Thank you.

Thank you for the detailed and constructive review. We greatly appreciate your recognition of the core ideas, practical improvements, and empirical rigor of our work, as well as your thoughtful suggestions for improving clarity, analysis, and comparisons. Your concerns are addressed below.

---

W1: The difference and comparison between HybridGS (another dual method) is not articulated well enough.

Our approach differs fundamentally from HybridGS in both design and training. HybridGS separates static and dynamic content using two models (3DGS for static, 2DGS for dynamic) and requires a staged training process with a learnable blending mask. In contrast, our method uses a dual 3DGS setup with mutual supervision to improve robustness against dynamic noise, all within the standard 3DGS training pipeline. While both methods use masking, HybridGS blends outputs based on 2DGS-derived uncertainty, whereas we apply two distinct masking strategies to reduce confirmation bias from a single, potentially inaccurate mask.

We will elaborate more on these differences in the main paper.

---

W2: Lack of hyper-parameter sensitivity analysis. Specifically, the method's performance hinges on the careful balancing of several loss terms and update rules, controlled by ($\lambda_m$ and $\lambda_{mask}$). For a method that claims "robustness," it is critical to demonstrate that it is not overly sensitive to its own internal settings.

Table A: Performance under varying weights of the mutual consistency loss.

Table B: Performance under varying weights of the learnable mask loss.

As shown in Table A and B, although the best performance is generally achieved at our default setting ($\lambda_m = 0.1$ and $\lambda_{mask} = 1.0$), the differences across settings are minimal (less than 0.1 dB). This indicates that the performance is not highly sensitive to the values of $\lambda_m$ and $\lambda_{mask}$. When $\lambda_m = 0$, mutual consistency regularization is disabled, resulting in a significant performance drop. Due to time constraints, all experiments above were conducted using the EMA-GS setting, and the PhotoTourism dataset was excluded.

---

W3: Computational cost of mask generation is unclear (was it included into the run-times?)

Table C: Average per-scene running time across different datasets (in hours).

No, it is not included in the overall runtime. Mask generation is a one-time preprocessing step performed offline, similar to COLMAP in existing 3DGS pipelines, which is also excluded from training time. We present the mask generation times in Table C. Notably, the PhotoTourism dataset requires significantly more time due to its higher image resolution and greater number of views.

---

W4: The limitations section (in the Supplementary) is quite brief.

We follow WildGaussians in using a per-view appearance embedding to control global appearance, and a per-Gaussian embedding shared across views to model the appearance of individual Gaussian primitives. While this strategy is effective for handling dynamic lighting variations across training frames, it may struggle to capture fine-grained effects such as object highlights. This can happen when the per-view and intrinsic appearances are not well decoupled, where the per-view embedding incorrectly attributes static intrinsic appearance to dynamic lighting, and the per-Gaussian embedding lacks the capacity to model high-frequency details. While our proposed mutual consistency loss helps improve this decoupling, it does not completely resolve the issue. This limitation is common in many existing works [1,2,3] and remains an important direction for future research.

[1] Fridovich-Keil et al. - 2023 - K-Planes: Explicit Radiance Fields in Space, Time, and Appearance

[2] Dahmani et al. - 2025 - SWAG: Splatting in the Wild Images with Appearance-Conditioned Gaussians

[3] Kulhanek et al. - 2024 - WildGaussians: 3D Gaussian Splatting In the Wild

---

Q1: Improve experimental clarity: clarify all ambiguous methodological and experimental details, including the full computational cost of the pipeline (with mask generation), the precise definition of the alternating masking strategy, and provide statistical significance for the main results (e.g., mean and standard deviation over multiple runs).

(1) The full computational cost of the pipeline is discussed in the response to Weakness 3.

(2) Clarification on the alternating masking strategy

In the GS-GS setting, we apply distinct masks to each branch of the dual 3DGS models as shown in Eq. (8). However, this is not feasible in the EMA-GS setting, which uses only a single 3DGS model and maintains an exponential moving average for consistency regularization. To encourage diverse supervision, we adopt an alternating masking strategy described in Eq. (11): one mask is used during a given optimization step, and the other is applied in the following step, alternating throughout training. This approach mitigates the bias of relying on a single mask.

(3) Statistical significance for the main results

Table D: Quantitative results on the NeRF On-the-go dataset. Each experiment is repeated five times, and we report the mean and standard deviation.

Table E: Quantitative results on the RobustNeRF dataset. Each experiment is repeated five times, and we report the mean and standard deviation.

We repeated the experiment for five time. We did not include results on the PhotoTourism dataset due to time constraints, but we will include it in the final version. Based on the results in Table D and E, our method shows statistically significant improvements.

---

Q2: It would be great to include sensitivity analysis for the key hyperparameters.

Please refer to Weakness 2, we will revise it accordingly.

---

Q3: Strengthen Related Work -- a more nuanced discussion of the related literature, explicitly contrasting the proposed conceptual framework with that of HybridGS would be quite beneficial.

Please refer to Weakness 1, we will revise it accordingly.

Dear Reviewer haRb,

Thank you for taking the time to review our work and engage in the discussion. As we approach the end of the discussion phase, we would like to ensure that all of your concerns have been adequately addressed. If anything remains unclear or requires further explanation, please don’t hesitate to let us know.

Thank you.