MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing

We rearrange and classify similar questions together.

W1. No 3D representation? Tuning down "3D editing" in title.

Thanks. First, we would like to clarify that our paper has included a detailed discussion on the explicit 3D representation (3DGS) within our method in Section C of Appendix (Lines 727-762). Additionally, we have provided videos rendered using 3DGS in supplementary. Second, the primary contribution of our work is to 'bridge' the gap between 2D and 3D editing, rather than focusing exclusively on either 2D or 3D synthesis. This bridging is fundamentally about achieving multi-view consistent inpainting (Lines 45-46), which MVInpainter effectively addresses. Benefiting from the consistent inpainting, our method enjoys reliable 3D representation (Lines 298-299). While the key contribution of our work lies in multi-view inpainting, discussion on 3D representation holds a lower priority. Therefore, we provided them in the Appendix. We would further clarify this point in revision.

W2, W3. Limited novelty of LoRA, temporal transformers, and Ref-KV?

Thanks. While we understand your perspective, we respectfully disagree. The primary novelty of our paper lies not in the specific model design, but in the innovative task formulation that synergistically leverages video priors, optical flow, and the inpainting formulation to simplify the complex novel view synthesis (NVS) into a multi-view inpainting task (Lines 44-46). We believe that the novel model design is not the only way to advance AIGC development. An effective new formulation can significantly reduce task complexity (Lines 122-123), thereby also providing significant novelty and insights to the community. Besides, except for video inpainting modules, we also propose other novel and effective components, including flow grouping and mask adaption.

Q2. Clarifying for Line 186: "which always captures unmasked ... noise from another U-Net".

Thanks. Line 186 details the differences between our usage of Ref-KV and previous works [Hu L et al,CVPR2024;Ruoxi Shi et al,2023]. We apologize for the incorrect citation here; AnyDoor[Chen, Xi, et al,CVPR2024] should be AnimateAnyone[Hu L et al,CVPR2024]. Formally, AnimateAnyone used another diffusion U-Net to encode clear reference views (noise-free), while Zero123++ re-scaled the latent inputs by multiplying a constant to the reference latent (5x) to strengthen the effect of the reference view. In our work, no additional modules or adjustments are necessary to encode the reference latent. This is because the first reference frame in our inpainting model is completely mask-free, providing straightforward guidance when concatenated as the unmasked input (Lines 185-187). We will provide further clarification on these points and revise the citation.

W4. a) Setting details of tasks, datasets, and masks. Thanks. We have introduced the inference pipeline in Section 3.4 and Figure 3(a) of the main paper. The overall tasks of this paper are clear, i.e., multi-view object removal (MVInpainter-F) and object insertion (NVS by MVInpainter-O), while the replacement is just the combination of these two tasks as in Figure 3(a). We introduced all input-output pairs for NVS and object removal in Lines 142-156, Figure 2(a) of the paper, which is further detailed in Table 4 of Appendix. NVS is trained on object-centric data (MVInpainter-O), while object removal is trained on face-forwarding data (MVInpainter-F). Masks are discussed in Lines 163-169.

Ad-hoc pairs for object removal? No ad-hoc training pairs are needed for object removal. Based on the common conclusion of previous inpainting works, training on the scene-level data with random masks is sufficient to learn the object removal ability. We will further improve the task definition and data setting.

Q6. How to obtain training masks without object masks? Without the object-level tracking masks, we only use hybrid random inpainting masks (Lines 163-164) to train MVInpainter-O. Note that we have discussed that both MVInpainter-O and MVInpainter-O adopt random inpainting masks, while we additionally employed object masks to MVInpainter-O (Lines 164-165).

b) Various setups of experiments, Difference between Object Removal and Scene-Level Inpainting.

Thanks. All sections are used to evaluate MVInpainter-O and MVInpainter-F separately. Formally, Section 4.1 (Object-Centric Results) is dedicated to evaluating the NVS performance of MVInpainter-O trained on object-centric data; Section 4.2 (Forward-Facing Results) is dedicated to evaluating the performance of MVInpainter-F trained on forward-facing data; Section 4.3 (Real-World 3D Scene Editing) shows the result of the combination of object removal (MVInpainter-F), NVS (MVInpainter-O) and mask adaption (Lines 296-298). We further divide Section 4.2 into 'Object Removal' and 'Scene-Level Inpainting' to evaluate the abilities to remove objects with object masks (Line 279) and inpaint images with random masks (Line 291). So 'Object Removal' and 'Scene-Level Inpainting' are both tested with MVInpainter-F but with different mask types and test sets. We also detailed the settings of datasets (Lines 246-254, Lines 675-687) and metrics (Lines 263-267) in the paper.

c) Setup for quantitative and qualitative experiments, especially for object removal.

Thanks. All quantitative and qualitative experiments are conducted on the same dataset except for the object removal (Line 279-280), which were discussed in paper. Quantitative results of object removal are based on the test set of SPInNeRF without foregrounds which can be regarded as GT (Line 279), while qualitative results of object removal are evaluated on the train set of SPIn-NeRF with foregrounds (no GT, Line 280). More details are discussed in Lines 681-684 of Appendix.

Limited by the rebuttal context, other concerns from reviewer NjjA are answered in the global rebuttal (at the top of this page).

W1, Q2. Whether mask adaption is enough robust? How to deal with different masks and backgrounds?

Thanks for this good point. As long as the 'basic plane' where the object is placed can be approximated to a plane, the proposed mask adaption is robust enough with the theoretical basis of perspective transformation. We have shown the effectiveness of the mask adaption with various object shapes in Figures 12 and 13 of the Appendix. To further clarify the robustness, we provide more examples with complicated backgrounds in the wild shown in Figure 4 (rebuttal pdf), including textureless table, rough lawn, and pool with sunlight reflection. We find that the dense matching (RoMa[1]) used in our work enjoys good generalization on various backgrounds. Importantly, most backgrounds can be approximated to the 'basic plane' as claimed in Lines 231-233. We would add these results to the paper.

[1] Johan Edstedt, et al. RoMa: Robust Dense Feature Matching. CVPR2024.

Q1. Do low-quality flows obtained from masked images negatively impact the result?

Thanks. We have discussed this point in the Line 194 (footnote of page 5). We extract flows before masking because foregrounds largely benefit the flow quality. Note that we process flows with 5-pixel dilated masks to avoid any leakage. Given that optical flows are typically low-level local features, we have not observed any conflicts when using masked flows as guidance. Furthermore, the quantitative results of dense flow presented in Table 3(b) verify that no leakage occurs (dense flows did not improve the NVS results), ensuring the integrity of quantitative results. We would clarify about the detailed operation of flow masking (mask dilation) in the revision.

W1. Heuristic assumption of mask adaption?

Thanks for this point. We should clarify that the planar assumption of mask adaption in Line 232 is reasonable and straightforward to implement across diverse real-world cases. First, this strategy roughly decides the mask location rather than providing exact mask shapes, which would be further irregularly masked for better generalization (Line 241-242). Second, the mask adaptation proves to be robust for the 'bottom face' of targets with various shapes, as verified in Figures 12 and 13 of the Appendix. Almost all objects have explicit or approximate 'bottom faces' which makes this assumption widely applicable. For the `basic plane', the key ability is the dense matching method. We find that the dense matching (RoMa [1]) used in our work can be generalized to various wild backgrounds (textureless table, rough lawn, and pool with sunlight reflection) as verified in Figure 4 (rebuttal pdf), denoting the robustness of mask adaption.

[1] Johan Edstedt, et al. RoMa: Robust Dense Feature Matching. CVPR2024.

W2. Complicated figures and flow? Needing comprehensive content of figures.

Thanks for this good advice! Indeed, finishing a 3D editing is an inherently non-trivial task, including sophisticated pipelines as seen in many pioneering works. In this paper, we simplify the overall pipeline into the following steps:

(1) 2D inpainting → (2) multi-view removal (optional) → (3) multi-view synthesis/insertion → (4) 3D reconstruction (optional).

These steps are illustrated in Figure 3 and detailed in the inference pipeline in Section 3.4. Our work focuses on steps (2) and (3), which are addressed by MVInpainter-F and MVInpainter-O respectively (Figure 2a). We agree with the reviewer that providing a more comprehensive overview figure between Figures 2 and 3 could make readers understand our paper better. We design a new overview shown in Figure 5 (rebuttal pdf), which clearly indicates the steps that are the main focus of our paper and identify the models specifically designed for each step. We will re-organize these points, if accepted.

Q1. Generalizability for OOD cases.

Thanks. Benefiting from the T2I prior from StableDiffusion, our method enjoys the capacity to tackle OOD cases, which have been verified in our submission. Formally, 15 categories of MVImgNet are unseen test sets (Line 678). We further evaluate the OOD capability of MVInpainter in the zero-shot Omni3D (Table 1, Figures 4 and 9 of the paper). We show some qualitative NVS results of OOD categories in Figure 3 (rebuttal pdf). Moreover, our method could also be used for unseen toys generated from exemplar-based synthesis[2] as verified in Figure 10. Considering the difficulty of scene-level NVS, it is challenging to properly address some completely OOD cases, such as human bodies, which are limited by the capacity of SD and AnimateDiff, as well as the restricted in-the-wild multi-view datasets (CO3Dv2+MVImgNet). Despite these limitations, MVInpainter is, to the best of our knowledge, the most robust reference image-guided method that generalizes well to in-the-wild object NVS across various categories (Lines 253-254). We would discuss this in the revision, and consider scaling up our model with more diverse training data to develop a foundational model as interesting future work.

[2] Xi Chen, et al. Anydoor: Zero-shot object-level image customization. CVPR2024.

W1. How does the method perform with unordered multi-view inputs?

Thanks. Although our work mainly focuses on the sequential multi-view inputs to better leverage the video prior from video components, it can also achieve comparable performance with unordered inputs as exploratively verified in Table1 and Table2 (rebuttal pdf). Unordered inputs still enjoy competitive FID, DINO, and CLIP scores for object-centric novel view synthesis (NVS) and object removal, indicating good image quality and consistency. The proposed flow grouping and Ref-KV also strengthen the generated direction and appearance for unordered inputs, preserving stable PSNR. However, unordered NVS suffers from some structural distortions caused by irregular viewpoint changes as shown in Figure 1 of the rebuttal pdf. Thus, we would still clarify that using ordered images with video prior is a more efficient way to model the multi-view inpainting (Line 171-173). Maybe further fine-tuning our model with unordered images could alleviate this issue, which is beyond the scope of this paper.

Table1: Unordered generative results of object removal

Table2: Unordered generative results of object-centric NVS

W2. How are priors different from AnimateDiff and Flow Grouping? Why are they complementary?

Thanks. The priors from AnimateDiff and flow grouping serve different but complementary purposes in our method. The video prior from AnimateDiff enhances structure consistency in the generated outputs (Line 307-308 and Figure 6 of the paper). This helps maintain a coherent structure across frames. Flow Grouping improves pose controllability (Line 191-193 and shown in Figure 11 of the Appendix). This component ensures that the synthesized poses are accurate and aligned with the intended viewpoint changes from unmasked regions. For the NVS inpainting, objects are consistently masked, leading to potential ambiguities in some scenes. These ambiguities can cause errors in pose generation, but video models alone struggle to address them (Figure 11). Quantitative results further verify this point (Table 3).

W3. Why not compare to SPIn-NeRF-type methods? Because of the requirement of camera?

Thanks for this good point. Our contributions are orthogonal to NeRF editing-based manners (SPIn-NeRF[1]). MVInpainter focuses on tackling multi-view editing with a feed-forward model, while NeRF editing is devoted to reconstructing instance-level scenes with test-time optimization. Except for the exact camera poses, NeRF editing manners require costly test-time optimization (Line 109-112) for each instance (SPIn-NeRF needs about 1 hour for each scene). Moreover, NeRF editing manners fail to substitute for our method as evaluated in Figure 2 and Table 3 (rebuttal pdf). a) NeRF editing starts with inconsistent 2D-inpainting results, which leads to blurred results as shown in Figure 2 (rebuttal pdf), while our method could refer to a high-quality single-view reference without conflicts. b) Although both methods enjoy good consistency, rendering-based inpainting suffers from color difference when blended with the original images (the last row of Figure 2 (rebuttal pdf)). As shown in Table 3 (rebuttal pdf), our method is comparable to SPIn-NeRF in consistency (DINO-S, DINO-L) with better image quality (PSNR, LPIPS, FID) and fidelity in their official object removal test set. Note that our method is orthogonal to instance-level manners and can further boost their performance with much better inpainting initialization. For example, experiments in Section C of the Appendix show that our method could be easily integrated with 3DGS to achieve consistent 3D outputs. We would add these discussions to the paper, if accepted.

Table3: Object removal compared to SPIn-NeRF

[1] Ashkan Mirzaei, et al. Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields, CVPR2023.

W4. Limitation of complex background.

Thanks. It is important to note that all methods have certain constraints. Our approach is currently limited by the capabilities of StableDiffusion and AnimateDiff, making it challenging to inpaint 360° complex backgrounds that are out of reference view. However, our method is effective enough for most multi-view scene editing. We acknowledge that scaling up our model to a foundational video inpainting model would be interesting future work.

Thanks for the valuable feedback from all reviewers. We have carefully addressed all concerns in our rebuttal. Our models and codes will be open-released, including all details to reproduce our results. As the rebuttal deadline approaches, please feel free to discuss any remaining questions or concerns. We will try our best to answer your questions.