In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment

• 1. Quality of academic writing.

Thanks for pointing out these typos. In line 172, Omega(·) is a decoder which is introduced in line 151, and $\hat{D}_{\phi}$ is the depth estimated by NeRF. We hope these typos do not affect the clarity of the ideas presented in this paper.

• 2. Compromise of view-dependent effect in ELA.

Yes, we compromise the view-dependent effect in NeRF within the ELA module, using NeRF only as a geometry prior to propagate and aggregate initial latents. Due to the heuristic nature of diffusion models, incorporating view-dependent effects into their output remains elusive.

• 3. Lack of sensitivity analysis.

Thanks for the constructive feedback. We've conducted 4 sensitivity analysis regarding the base view selection, $\lambda_a$ in ILA, subset selection and $\lambda_\text{patch}$ for patch loss. Due to the computational burden, we conduct the sensitivity analysis on six out of ten scenes with higher inpainting variability from SPIn-NeRF dataset.

(a) Base View Selection:

To achieve better generalization, we propose to sample $n$ candidate views around the geometry centroid of the camera viewpoints and select the one with the highest similarity votes, automatically avoiding artifacts without human intervention. We used 5 candidate views, with similarity calculated using perceptual hashing.

We tested our results under different settings (candidate numbers): 3, 5, 7, and 9. The selection algorithm proved to be robust, with our algorithm typically yielding the same base frame. However, another factor influencing this step is the random seed. Setting different seeds causes the 2D inpainting model to produce different results, leading to different base frames being selected. We tested our method under five different seeds, with final scores reported in the table below and additional qualitative results in Figure 2 of the rebuttal PDF. While different seeds affect the appearance of the masked region in the final NeRF, the consistency of multi-view inpainting remains robust, resulting in minimal variance in evaluation scores. We will explicitly clarify this point in the revision.

Seed	LPIPS↓	MUSIQ↑	FID↓
1	0.46	46.61	264.91
2	0.44	48.04	255.29
3	0.44	46.47	262.09
4	0.44	45.72	261.04
5	0.46	48.65	258.50
Avg	0.45	47.10	260.37
Std	0.01	1.21	3.657

(b) $\lambda_a$ in ILA:

To examine the effect of the hyper-parameter $\lambda_a$ in ILA, we evaluated our method's rendering quality with varies $\lambda_a$ value. The metrics are reported in below table. The results are consistent across different $\lambda_a$ values, indicating a relatively small effect. This conclusion is supported by qualitative results in Figure 3 of the rebuttal PDF, where larger $\lambda_a$ values produce slight variations in small regions, but the global structure and semantics are preserved.

This stability is attributed to the significant role of the initial latent alignment in ELA, which effectively aligns the underlying inpainting structure, thereby maintaining low variability in appearance. Additionally, the self-attention layer, where cross-view attention is introduced, does not dominate the entire Stable Diffusion Unet. It is balanced by the presence of other (residual and linear) layers, ensuring cross-view attention does not override the signal during the denoising process. Hence we simply set $\lambda_a$ in our implementation. We will explicitly clarify this point in the revision.

$\lambda_a$	LPIPS↓	MUSIQ↑	FID↓
0.2	0.44	47.11	261.62
0.4	0.44	46.76	264.91
0.6	0.44	46.47	264.37
0.8	0.45	46.33	265.10
Avg	0.44	46.67	264.00
Std	0.01	*0.35 *	1.62

(c) Subset Selection:

We propose selecting a subset for stage 3 training based on the distribution of camera viewpoints. We evenly split the viewpoints into 12 groups based on the base view's camera space (evenly 2 on the x and y axes and 3 on the z axis) and select 50% views within each group according to perceptual hashing similarity to the base view. This approach avoids redundant views introducing supervision conflicts, while covering different viewpoints effectively.

We evaluated our method with varies selection percentages, as reported in the table below. The scores are close, indicating minimal differences for most scenes. For one complex scene with high frequencies, setting the percentage too low (0.2) yields artifacts due to insufficient viewpoint coverage, while setting it too high (0.8) introduces appearance conflicts due to variability in inpainted results. These results are visualized in Figure 4 of the rebuttal PDF.

Percentage	LPIPS↓	MUSIQ↑	FID↓
0.2	0.46	45.98	265.48
0.4	0.44	46.32	264.9
0.6	0.44	47.11	261.62
0.8	0.45	46.47	263.20
Avg	0.44	46.67	264.00
Std	0.01	0.35	1.62

(d) $\lambda_{patch}$ in patch loss:

To assess the sensitivity of the patch loss multiplier $\lambda_{patch}$ in Eq. 10 of the main paper, we evaluated the method's performance using varies $\lambda_{patch}$ values. The results in below table show similar performance across different settings, with low standard deviation. However, setting $\lambda_{patch}$ too low or too high adversely affects performance. $\lambda_{patch}$ is critical as it influences the extent of multi-view supervision on NeRF; insufficient supervision leads to inadequate training, while excessive supervision causes conflicts. We set $\lambda_{patch}$ at 0.01 in our implementation for optimal balance.

$\lambda_{patch}$	LPIPS↓	MUSIQ↑	FID↓
0.001	0.46	46.07	263.32
0.005	0.45	47.08	262.43
0.01	0.44	47.11	261.62
0.05	0.47	4.93	265.31
0.1	0.49	44.05	277.36
Avg	0.46	45.85	266.01
Std	0.02	1.35	6.49

• 1. Explanation of ILA

(a) The reviewer's intuition of ILA is entirely correct. ILA primarily contributes to appearance (color) consistency. Geometry consistency is achieved by ELA (the alignment of initial noise) since this element serves as a foundational structure (semantic) for the inpainting. This is verified in our ablation study in Figure 6 of the main paper. Removing ELA results in geometry mismatch (blurry boundaries), while dropping ILA achieves geometry alignment but causes color mismatch issues (blurry leaves). This is why we propose ELA and ILA together to ensure both geometry and appearance consistency.

(b) The rationale of replacing $KV$ with "prior" $p$, but not $Q$ is that the appearance information ($V$) of the prior image should be considered when inpainting the other views, with amount of information propagation is weighted by its attention key value ($K$). The attention query value comes from the current inpainting view $i$, $Q_i$ , representing the information the current inpainting for view $i$ is searching for. Together with $K_p$ , it decides how much attention the view $i$ inpainter should place on the prior view, and finally incorporates the prior view information $V_p$ into view $i$. We will elaborate on these points in the revised version of the paper.

• 2. Discussion of selecting base frame.

To achieve better generalization, we propose sampling the base frame based on the geometrical centroid of the training camera poses. However, Stable Diffusion occasionally inpaints strange artifacts in the masked region. To mitigate this, we sample $n$ candidate views around the centroid of the camera viewpoints and select the one with the highest similarity votes, automatically avoiding artifacts without human intervention. In our implementation, we used five candidate views, with similarity calculated using perceptual hashing.

We tested our results under different settings (candidate numbers): 3, 5, 7, and 9. The base frame selection algorithm proved to be robust, with our algorithm typically yielding the same base frame. However, another factor influencing this step is the random seed. Setting different seeds causes the 2D inpainting model to produce different results, leading to different base frames being selected. We tested our method under five different seeds, with final scores reported in the table below and additional qualitative results in Figure 2 of the rebuttal PDF. While different seeds affect the appearance of the masked region in the final NeRF, the consistency of multi-view inpainting remains robust, resulting in minimal variance in evaluation scores. We will explicitly clarify this point in the revision.

Seed	LPIPS ↓	MUSIQ ↑	FID ↓
1	0.46	46.61	264.91
2	0.44	48.04	255.29
3	0.44	46.47	262.09
4	0.44	45.72	261.04
5	0.46	48.65	258.50
Avg	0.45	47.10	260.37
Std	0.01	1.21	3.657

Table 1: Sensitivity analysis on the prior inpainting results and prior view selection. Results are evaluated on SPIn-NeRF dataset with different random seeds.

• 3. Eq (10) applied to Stage 3.

Yes, Eq. 10 is the final loss for Stage 3, designed to alleviate noise and inconsistency. Thank you for the reviewer's insight, and I fully understand the confusion. The prior loss $L_\text{prior}$ is an L2 loss first proposed in Stage 2. We propose keeping this term in Stage 3 because we believe the base view does not require patch-based optimization. As discussed in the limitations section, achieving perfect 3D consistency is challenging, so the remaining views may still have subtle inconsistencies. To preserve high-frequency details, we use patch loss instead of the exact-match L2 loss. We regard the base view as a baseline that does not exhibit inconsistency; thus, we do not apply the patch loss to it.

• 4. Discussion of Subset.

We found that for reconstruction tasks, more views can enhance quality; however, for generation tasks, using the entire set of images introduces unnecessary inconsistencies. Therefore, we propose selecting a subset based on the distribution of camera viewpoints.

We evenly split the viewpoints into 12 groups based on the base view's camera space (evenly 2 on the x and y axes and 3 on the z axis) and select 50 percent frames within each group according to perceptual hashing similarity to the base view. This approach avoids redundant views introducing supervision conflicts, while covering different viewpoints effectively.

We also evaluated our method with different selection percentages, as reported in the table below. The quantitative scores are quite close, indicating minimal differences for most scenes. For one complex scene with high frequencies, setting the percentage too low (0.2) yields artifacts due to insufficient viewpoint coverage, while setting it too high (0.8) introduces appearance conflicts due to variability in inpainted results. These results are visualized in Figure 4 of the rebuttal PDF.

Percentage	LPIPS ↓	MUSIQ ↑	FID ↓
0.2	0.46	45.98	265.48
0.4	0.44	46.32	264.91
0.6	0.44	47.11	261.62
0.8	0.45	46.47	263.20

Table 2 : Sensitivity analysis on proportion of images selected for the subset.

Overall, for most scenes, the subset selection algorithm is robust due to the consideration of viewpoints distribution. For extreme cases, careful selection of the percentage might be necessary. However, values between 0.5 and 0.7 remain a reliable choice. We will carefully revise the manuscript to include these details.

Thanks you very much for your feedback.

Regarding L169, the two components described are the input prompt $e$ and the masked image $I_p^{\prime}$ , as detailed in L170. Figure 2 illustrates the masked image $I_p^{\prime}$. The prompt $e$ is simply set as the description of the background we aim to inpaint, which is scene-dependent.

In L199, the prompt $e$ is defined same as In L169, we employ a unified prompt for each scene.

It's important to note that we did not involve any prompt engineering in our work, and we use the same prompt for each scene across all the methods in the experiments. We will provide further clarification on this point in the revised version.

We would like to thank the reviewer for providing these valuable comments.

• 1. Lacks a comparison with other baselines.

Thank you to the reviewer for pointing this out. Unfortunately, the method "Reference-guided Controllable Inpainting of Neural Radiance Fields" (ICCV 2023) is not open-sourced. However, this method, as stated in the introduction of our paper, operates similarly to the baseline method (InFusion), i.e., using a single reference view to serve the entire scene. These approaches assume that the geometry inferred by the radiance field (NeRF or Gaussian Splatting) is almost accurate, thus relying heavily on the geometric prior.

From the results of InFusion in Figure 4 of the main paper, we can observe that the accuracy of the geometry cannot be guaranteed from scene to scene. For instance, the InFusion excels in the first and third rows but exhibit some disconnection of the primitives in the second and fourth rows. Similarly evidence can be found in our new collected datasets as well, as in rebuttal PDF Figure 1.

These findings highlight the importance of multi-view supervision for 3D generation and inpainting tasks. Consistent multi-view supervision can mitigate the reliance on geometric priors and improve the robustness and accuracy of the inpainting results. By incorporating multi-view supervision, our method still works in the circumstances when such depth/geometry is not accurate, achieving promising results.

• 2. How to get the patches.

Thank you to the reviewer for the constructive feedback. We will revise the paper to include more complete details in the revision. The patches are uniformly sampled within the bounding box of the mask, and the patch size we used is 256×256. Therefore, only the content within the bounding box (mostly the inpainted area) is being optimized by the patch loss.

• 3. Eq.5 ground truth supervision.

Thank you to the reviewer for the comment. There is no ground truth supervision for the masked region. We used the official train-test split provided by SPIn-NeRF. In the training set, the masked region contains the unwanted object, while the test set contains the ground truth background in the masked region. We only optimize our radiance field on the training set, ensuring no information leakage. We will explicitly clarify this point in the revision.

• 4. Computational efficiency.

Thank you to the reviewers for the valuable feedback. We would like to provide a comparison of training time in below table. InFusion ranks first in efficiency due to the high rendering efficiency of Gaussian splatting and the simplicity of single-view optimization. The three NeRF-based multi-view methods (SPIn-NeRF, NeRFiller, and ours) exhibit similar optimization efficiency. Our method has a slight advantage in efficiency due to the sampled prior image and the relatively consistent multi-view images, which facilitate faster convergence.

Method	Supervision	Training Time (min)
SPIn-NeRF	Multi-view	71
NeRFiller	Multi-view	65
InFusion	Single-view	17
Ours	Multi-view	49

Table 1: A comparison of Training Time.

Dear Reviewer uTYn,

I hope this message finds you well. We are grateful for the time and attention you've already dedicated to reviewing our work. We have submitted a rebuttal addressing the concerns raised, and we would greatly appreciate any additional feedback or comments you might have. As the discussion period is drawing to a close soon, we are eager to know if our rebuttal addresses your concerns or if there is any additional feedback you might have. Thank you very much for your continued support and guidance.

Best regards,

Authors of Submission 4061

We appreciate the reviewer for the constructive feedback. Below we provide the answers to reviewer's concern:

• 1. Method over-simplified, and marginal difference compared to priors works.

We would like to argue that while our motivation may appear straightforward, the solution we propose is both non-trivial and novel.

The multi-view inpainter NeRFiller achieves consistent inpainting results by averaging noise predictions derived from multiple iterations. This approach tends to produce smooth results, as demonstrated in Figure 4 of the main paper and Figure 1 of the rebuttal PDF. In contrast, our method explicitly and implicitly aligns latents without sacrificing high-frequency details.

InFusion relies on a single reference view and its depth to represent the entire scene. This reliance becomes problematic when the assumption of accurate depth is violated. As shown in Figure 4 of the main paper, misaligned geometry would lead to disconnected primitives (second and fourth rows). Similar evidence can also be seen in Figure 1 of the rebuttal PDF. By incorporating consistent multi-view supervision, our method remains effective even when depth or geometry is inaccurate, achieving robust and promising results. This explains why our method shows little difference from InFusion when geometry is accurate (first and third rows of Figure 4 in the main paper), but excels when the depth is inaccurate.

We further collected nine scenes with manually annotated masks to evaluate the effectiveness of our method. This dataset includes four indoor scenes and five outdoor scenes. We conducted the experiment and evaluation under the same protocol as in the main paper, using the same inpainting pipeline for all methods as mentioned in the paper. The quantitative results are reported in table below and the qualitative results are shown in Figure 1 of the rebuttal PDF. We will include this evaluation with more details in the revised manuscript.

Method	LPIPS ↓	MUSIQ ↑	FID ↓
NeRFiller	0.68	19.43	399.19
InFusion	0.47	31.35	319.59
Ours	0.35	37.22	250.63

Table1: Quantitative Result on new collected dataset.

In summary, although our motivation is simple, i.e.,using consistently inpainted multi-view images to optimize the radiance field, our approach to achieve consistent results is both novel and effective without relying on any geometric assumptions.

• 2. ELA approach seems over-complicated.

Thank you to the reviewer for providing valuable comments. First, the reviewer's understanding of the three stages is totally correct. However, there are two key reasons why we propose fine-tuning the NeRF and using it as a geometric prior for ELA:

(a) After finetuning the NeRF, the geometric is represented by NeRF as a sharp (low variance) unimodal distribution on the ray. Consequently, the aggregated feature remains sharp, preserving the variations in the initial latents.

(b) As we discussed in the paper, we empirically found the depth prior inferred by the monocular depth estimator is not perfectly aligned for the NeRF. Fine-tuning the NeRF can also benefit this depth prior. Since NeRF learns relatively certain geometry in the known (unmasked) areas, this geometry constraint can improve the geometry of neighboring inpainted (masked) areas due to their geometric proximity.

We will explain these points in more detail in the revised version of the paper.

• 3. Limited scenario.

While prior works have been proposed for more general cases, our motivation focuses on a relatively smaller scope, specifically "object removal." As outlined in the paper, we have found that this task is extremely challenging due to the requirement for scene-level generation, which demands consistent multi-view supervision of a scene rather than just an object. Current approaches still exhibit some limitations in this regard.

In contrast, there are many works focusing on 3D object-level generation using multi-view diffusion models. Therefore, we believe that tasks such as object insertion and completion should follow this more promising direction, which we have decided to leave as future work.

However, in response to the reviewer's comments, we have further tested our method on three 3D object completion scenes, which are already included in the nine scenes mentioned in our first response (marginal difference compared to prior works). The results, shown in the rebuttal PDF file Figure 1 (first two rows), indicate that our method still performs promisingly on this complex task. Despite this, we continue to believe that inpainting tasks for objects might be more effectively addressed using multi-view diffusion or video diffusion techniques. We will elaborate on these points in the revised version of the paper.

• 4. Artifacts are obvious in the result video.

We have carefully examined our implementation of the method and identified that the artifacts are caused by the depth loss used. Initially, we borrowed the loss function proposed in "Depth-supervised NeRF: Fewer Views and Faster Training for Free (CVPR 2022)", which applies KL divergence to the rays' density distribution for sparse view reconstruction. We found that this loss is relatively sensitive and tends to produce more noise in under-constrained areas (with no multi-view supervision). To address this issue, we have replaced our depth loss with a more stable L2 loss, which has largely eliminated the artifacts. The revised implementation now produces significantly cleaner results.

Dear Reviewer S5oa,

I hope this message finds you well. We are grateful for the time and attention you've already dedicated to reviewing our work. We have submitted a rebuttal addressing the concerns raised, and we would greatly appreciate any additional feedback or comments you might have. As the discussion period is drawing to a close soon, we are eager to know if our rebuttal addresses your concerns or if there is any additional feedback you might have. Thank you very much for your continued support and guidance.

Best regards,

Authors of Submission 4061