InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model

We thank reviewer 6csA for the valuable suggestions.

W1/P1: Limitation to sparse view settings

As acknowledged in our limitation section, InstaInpaint is indeed limited to a sparse-view setting and the performance degrades when the input view number increases.

However, the sparse-view setting also represents a feature for many real-world downstream applications where capturing dense multi-view data is impractical (e.g., mobile capture) or redundant (e.g., benchmarks such as SPIn-NeRF where the scenes are overly densely captured).

Regarding scaling to a larger number of input views and dense-view reconstruction tasks, one potential method to scale InstaInpaint to a larger number of input views is by applying the masking finetuning pipeline on dense-view feed-forward reconstruction model such as Long-LRM [1].

W2/P2: Lack of architectural improvements

We acknowledge that InstaInpaint does not introduce novel network architecture to GS-LRM model. However, our innovation centers on a flexible masking fine-tuning pipeline and its specific mask designs that effectively adapt reconstruction models to the 3D inpainting task. As a data-driven method, we believe that this pipeline's ability to leverage and guide existing powerful models is a significant and equally important contribution to the field.

Furthermore, while both InstaInpaint and MaskedLRM share architectural similarities with GS-LRM, we differ significantly in 3D representation and target application. MaskedLRM targets object-centric mesh editing and represents the 3D object in triplane-NeRF. InstaInpaint targets real-world scene editing, utilizing 3D Gaussian Splatting as its scene representation. The complexity of real-world scenes forces us to exploit more sophisticated masking strategies to adapt real-world multi-view datasets to 3D inpainting tasks.

W3/P3: Comparison with SimVS.

Thank you for bringing SimVS [2] to our attention and for asking about the differentiation.

The difference between SimVS and InstaInpaint highlights the benefit and the design philosophy of InstaInpaint. SimVS handles inherently 3D inconsistent multi-view images. On the other hand, InstaInpaint aims to lift 2D-generated content to 3D using 3D consistent context from the non-inpainting area. Our design naturally avoids resolving the challenging 3D inconsistency arising from multi-view inpainting models. We also show the benefit of our approach by comparing with "MVInpainter+LRM" (in Table 3), where the moderate 3D inconsistency of this approach leads to a more blurry appearance and less favorable performance.

Unfortunately, we are unable to compare to SimVS, as the model is not open-sourced. However, the "MVInpainter+LRM" baseline is highly relevant to the role of SimVS in the 3D inpainting problem: using a learned cross-view consistent diffusion model as reconstruction target, then performing feed-forward 3D reconstruction of the inpainted object. MVInpainter is our best effort in finding an open-sourced cross-view consistent generative model. However, MVInpainter produces very inconsistent results in high-frequency areas, leading to sub-optimal results compared to InstaInpaint, as is shown in Figure 3, Figure 7 and Table 2. This suggests a fundamental challenge in achieving perfect multi-view consistency solely through 2D generation for 3D reconstruction.

W4/P4: Direct comparision with multi-view 2D inpainting-based methods.

We would like to clarify that we have provided direct quantitative results with SPIn-NeRF, MALD-NeRF, GScream and InFusion in Table 1. We add comparison with InpaintNeRF360 in the table below.

W5/P5: Experiments with PSNR and SSIM metric

First, we would like to clarify that we do report PSNR, SSIM and LPIPS for both InstaInpaint and other reference-based 3D inpainting works (Including Infusion and MVInpainter+LRM) in Table 3. We further provide a comparison with GScream in the table below.

Below, we will explain why PNSR and SSIM can only be used to compare reference-based inpainting methods. PNSR and SSIM are low-level metrics that directly measure the resemblance between pixels.

For non-reference inpainting methods, the content within the inpainted area is non-deterministic. This means there can be multiple perceptually plausible and equally "correct" ways to fill a missing region, even if their pixel values differ significantly from the specific ground truth. In such scenarios, a low PSNR or SSIM score does not necessarily indicate poor perceptual quality, as the generated result might be reasonable despite differing from the exact ground truth pixels. For these methods, higher-level perceptual metrics (such as LPIPS and FID, which we include in Table 1 of the manuscript) are often more appropriate. We note that existing non-reference inpainting methods, such as SPIn-NeRF and MALD-NeRF, do not report PNSR and SSIM.

In the context of reference-based inpainting methods, the "inpainting" task is constrained by a reference. By providing the ground truth texture as a reference to these models, we effectively make the target content in the inpainted region more deterministic. For our evaluations in Table 3, we use the ground-truth image as this explicit reference. By providing this ground-truth content, the target for the inpainted region becomes deterministic, enabling PSNR and SSIM to accurately measure how closely the method reconstructs the specific ground-truth pixels at novel views.

[1] Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats. Chen, et al. ICCV 2025.

[2] SimVS: Simulating World Inconsistencies for Robust View Synthesis. Trevithick, et al. CVPR 2025.

We thank reviewer JFCW for the valuable suggestions.

W1/P1: More details and experiments on multi-region inpainting.

We mention InstaInpaint's multi-region inpainting to emphasize its flexibility and generalizability that various tasks can be supported without much specific design. However, we find that the number of training masks is relevant to model performance on multi-region inpainting. In our current training setup, the model processes input masks with 1 to 4 masking regions. We find that the number of masking regions combined during training influences the model's performance on multi-region inpainting tasks.

We add an ablation study on the number of masking regions during the training stage. We extend the SPIn-NeRF benchmark to support multi-region inpainting evaluation. Specifically, for each scene in the SPIn-NeRF benchmark, we randomly sample three oval masks on the reference view. We then warp the masks to other views the same way as the Geometric Masks in the paper (with metric depths predicted by LRM trained in the first stage). During Inference, we use the original image from the reference view as reference. We measure pixel-level metrics (PNSR, SSIM, LPIPS) on SPIn-NeRF's testing novel views. The results are presented in the table below:

W2/P2: Architectural innovations.

We acknowledge that InstaInpaint does not introduce a novel network architecture to GS-LRM [1] model. Our innovation centers on a flexible masking fine-tuning pipeline. The pipeline and its specific mask designs effectively adapt reconstruction models to the 3D inpainting task. As a data-driven method, this pipeline's ability to leverage and guide existing powerful models is a significant and equally important contribution to the field.

W3: Acceleration stems from LRM but not our designs

It is true that the 1000x acceleration is indeed powered by the underlying Large Reconstruction Model (LRM). But LRMs are not natively equipped for 3D inpainting. The main innovation is the masking framework designed to overcome the challenges involved in adapting powerful data-driven reconstruction models to the specialized domain of 3D inpainting, including a lack of inpainting dataset, object bias, etc. We are the first to adapt LRM to 3D inpainting field, unlocking their speed for this complex task, which is a problem not addressed by LRMs alone.

P3: More discussions on limitations.

While we are unable to provide visual examples in this rebuttal, we can elaborate on how dynamic scenes affect InstaInpaint. In dynamic regions, the reconstructions become blurry as the model averages conflicting information. Furthermore, if a dynamic object is occluded in the reference view but visible from other context views, the inferred content for the inpainted area may largely contradict the observations from those surrounding views, leading to huge visual inconsistencies. Addressing the challenge of 4D scene (space-time) reconstruction and inpainting for dynamic elements remains an active and complex research area. Future work will investigate integrating InstaInpaint with models specifically designed for dynamic 4D scene reconstruction or incorporating explicit temporal modeling capabilities to mitigate these issues.

We would like to clarify that InstaInpaint is designed to handle mask inconsistencies. The inconsistent random image masks help build robustness to slight inconsistencies introduced by inconsistent maskings.

[1] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. Zhang, et al. ECCV 2024.

Dear Reviewer,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions.

As we near the midpoint of our discussion period, we would like to confirm whether our responses have effectively addressed your concerns. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are happy to provide any further information you may need.

Thank you.

We thank reviewer nJGu for the valuable suggestions.

W1/P2: Scalability to 360 scenes.

We acknowledge that InstaInpaint is primarily designed for sparse-view inpainting and reconstruction, and full 360-degree scene reconstruction is indeed beyond its scope. But we have several promising solutions for extending our work to the challenge.

1. Finetuning on dense-view feed-forward reconstruction model (e.g. Long-LRM[1]). InstaInpaint's mask finetuning pipeline is very flexible and is suitable for feed-forward reconstruction models other than GS-LRM[2].
2. A two-stage 360-degree scene inpainting method. First, we can optimize a "partial" 3D-Gaussian scene with the inpainting area masked out. The inpainting area is left as a hole to be completed. Then we sample a sparse set of view images as inputs to InstaInpaint. Since Gaussian blobs predicted by InstaInpaint are strictly registered to the camera, we can store the predicted Gaussian blobs from the inpaint area at the original scene scale with a simple affine transformation and merge them into the optimized partial Gaussian Scene. To further enhance quality, especially at the merging boundaries, the combined Gaussian scene could undergo a few additional optimization steps.

W2: Details on input views selection in the experiments.

We select the 4 input views to ensure 1) reference view is posed close to the center and 2) the three other context views cover an area as large as possible.

1. For reference view, we calculate the mean of all training views (60 for SPIn-NeRF) and select the view whose position is closest to the mean as the reference view. The reference view is inpainted with SOTA 2D-diffusion inpainting models[3].
2. For the three other views, we aim to choose three views that cover as wide range as possible. We first regress a plane from all training views. Then we project all training view positions onto the plane. We then compute the convex hull of these projected 2D camera positions. The vertices of this convex hull represent the outermost camera viewpoints. From the vertices of the convex hull, we identify the three points that form a triangle with the largest area. Specifically, this is done by applying the rotating calipers algorithm on the convex hull's endpoints.

W3: Missing citations.

We will cite and discuss the suggested references in the revised paper.

P1: Per-scene Optimization Results

Yes, per-scene optimization does improve reconstruction quality compared with direct prediction from LRM. We experiment to test this.

This experiment is conducted following the setting mentioned in the paper from L241-L244. We optimize the LRM-predicted Gaussians on the 4 input views for 100 steps, which takes roughly 5 seconds. For reference view, we compute photometric loss on the entire image and all Gaussian parameters as learnable. For non-reference views, we compute photometric loss on the unmasked region and set Gaussians within the inpainted region as unlearnable.

[1] Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats. Chen, et al. ICCV 2025.

[2] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. Zhang, et al. ECCV 2024.

[3] FLUX. Black Forest Lab.

We thank reviewer hGee for the valuable suggestions.

W1: Typo in Figure 1.

We will fix the typo in the revised paper.

W2: Missing Related Works.

We will cite and discuss the suggested references in the revised paper.

W3: Clarification on L142-L145.

We revise and expand L142-L145 for better clarity.

We do not provide the inpaint region mask directly to the reference view. This means that when the reference image is fed into our pipeline, the binary mask image indicating the inpaint region is not given as an additional input alongside the reference RGB image itself, nor is the reference image's content explicitly altered (e.g., grayed out) in the masked region.

This design is inspired by a key observation: the inpainted region and the reconstruction regions should maintain a similar data distribution during training. To clarify the model's perception of different areas, we conceptualize three kinds of tokens within our pipeline:

(1) Unmasked regions with intact texture: These areas are fully visible and consistent across both reference and supervision views, providing clear multi-view stereo information vital for accurate 3D reconstruction.

(2) "Masked regions" in the reference view with original texture: This refers to the conceptual area within the reference view that corresponds to the region proposed for inpainting. The model "sees" the original, unaltered texture in this region. However, since other non-reference views do have these corresponding regions explicitly masked out, the model is implicitly tasked with inferring the geometry and fine details of this area within the reference view based solely on its surrounding unmasked context. This unique setup is vital for maintaining a similar data distribution between token types (1) and (2), enabling the model to learn to faithfully reconstruct both areas equally well, whether the mask is explicitly presented or not.

(3) Masked regions in non-reference views filled with gray pixels: These regions in non-reference views are explicitly altered (grayed out) with a provided mask and are provided with an explicit binary mask. This clear alteration forces the model to learn to ignore these areas or infer their content from other available views.

We aim for the model to faithfully reconstruct (1) using multi-view stereo vision, to accurately speculate the geometry of (2) based on neighboring regions in the reference view, and to mitigate negative effects from (3). This approach ensures a sufficient distinction between (2) and (3) to prevent feature contamination while promoting consistent learning across visible and conceptually "masked" areas.

W4: Generation details on random oval masks for geometric masks.

The random oval masks are generated on the fly during training. For each oval, we first randomly determine its center within the image boundaries. The major and minor axes of the ellipse are sampled with their sizes constrained to be within a certain proportion of the image dimensions to ensure masks are neither too small nor too large. A random angle between 0 and 360 degrees is selected to introduce rotational variability. We randomly sample 1 to 4 ovals and combine them to formulate the final geometric mask.

W5: Clarification on the motivation of random image masks.

We will include more design motivations in the updated paper. Here are two main motivations:

• Test-time 3D inconsistency. The testing benchmarks have their own annotation strategies different from our training strategy. The human annotation errors and mask dilation strategy often create substantial leftover masks bleeding through the actual object boundary, leading to 3D inconsistent masks in the background. Our random masks provide distinctive learning signals aside from the other two 3D consistent masking strategies.
• Data diversity. Similar to we address the object bias of object masks using randomized geometric masks. The geometric mask still follows certain biased data distribution and potentially introduces unattended biases. Our random masks provide challenging 3D inconsistent masks for the model to resolve the conflicts through finding 3D correspondences across images. We believe this strategy increases the data diversity and augments important learning tasks for the model to learn a more robust and generalizable representation.

P1: Potential application in autonomous driving.

InstaInpaint could be used for editing static elements in street scenes. For example, one could add or remove fixed objects like traffic signs, streetlights, or parked vehicles. Our method, trained on static scene data, can seamlessly integrate or remove these stationary items while maintaining 3D consistency.

However, applying InstaInpaint to dynamic road users (e.g., moving cars or pedestrians) would be challenging. InstaInpaint is trained for static 3D scene reconstruction and inpainting. Adapting our method for dynamic scene editing is an exciting area for future work.

P2: Related work from image editing area.

Masking training indeed is a common technique in image inpainting area, such as RePaint [1] and LaMa [2]. We will add representative works in the related work.

[1] RePaint: Inpainting using Denoising Diffusion Probabilistic Models. Lugmayr, et al. CVPR 2022

[2] Resolution-robust Large Mask Inpainting with Fourier Convolutions. Suvorov, et al. WACV 2022

Dear Reviewer,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions.

As we near the midpoint of our discussion period, we would like to confirm whether our responses have effectively addressed your concerns. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are happy to provide any further information you may need.

Thank you.

Dear reviewers,

The authors have responded to the original reviews. Could you read the rebuttal and share your thoughts? Does it address your original concerns? Are there any remaining questions for the authors? We still haven't heard from anyone for this paper and we would like to hear from everyone before the end of the discussion phase.

Best, AC