ReplaceAnything3D: Text-Guided Object Replacement in 3D Scenes with Compositional Scene Representations

We are grateful to reviewer ZFSz for their review! However, we are concerned that the reviewer has misunderstood our proposed method, as the summary contains multiple factual inaccuracies. In particular, in their summary, the reviewer states that we: “Fine-tune the existing NeRF model based on inpainted images” during the Erase Stage, in order to remove the target object, and then “Fine-tune the existing NeRF model using generated or customized images” during the Replace stage. Both of these are misunderstandings of our method. We do not fine tune an existing NeRF scene in our method; we introduce a novel Bubble-NeRF representation (Please see Fig 3, and lines 159 to 163 in main paper) which is trained from scratch, and which models a localised NeRF region corresponding to the masked pixels.

In Section 3.3 main paper, we explain how we first learn a background Bubble-NeRF during the Erase Stage, and in Section 3.4, we explain how we learn a foreground Bubble-NeRF which is composited with the background. In neither stage do we optimise these representations “using generated or customised images” as the reviewer claimed: directly training using inpainted images produces view-inconsistent results as we show in Fig 12, appendix, main paper. That is why we instead use a distillation-based training objective. However, given that the reviewer made no mention of our compositional structure, or localised Bubble-NeRF rendering in their summary (recognised as novelties by the other reviewers), we would be very grateful if they could carefully review our method (Section 3 main paper) before reconsidering their contribution rating.

We now address the reviewer’s specific concerns and questions: “The proposed method is quite obvious and too straightforward… most components inherit from previous works…” As mentioned above, we are concerned that these statements are based on an erroneous understanding of our method. The novel components of our method; localised Bubble-NeRF rendering, Halo region supervision, compositional rendering approach were not mentioned by the reviewer in their summary. We show that an Erase-and-Replace methodology can simplify the task of 3D localised scene editing; an interesting insight that we believe the community can benefit from.

“The problem itself is not that novel and the method does no open extra-functionality in the field of 3D editing” We showed how our model’s compositional structure allows adding multiple objects into the scene (Figure 6 main paper) as well as adding personalised objects using DreamBooth diffusion-finetuning in section C of the appendix, which is a practical novelty, as we are not aware of existing works with these capabilities. As far as we are aware, we are also the first to show high-quality object replacement results on 360 scenes such as the Garden scene, which constitutes a quality novelty.

“The way how existing methods use the prompt is different from the way how the authors used.” Please note that we compare our method with 5 localised editing papers in our main paper (Gaussian Editor, Blended NeRF, DreamEditor, Reference-Guided Controllable Inpainting of NeRF, Repaint-NeRF). All of these methods are conditioned by a spatial prompt, indicating the region which should be edited, in addition to the target text prompt. For example, GaussianEditor specifies the editing region using segmentation masks obtained from SegmentAnythingModel, exactly like ours. The instruction-style of prompts used by InstructNeRF2NeRF (and the similar works shown in the appendix figure 10) is due to their reliance on InstructPix2Pix, an image translation method which is not required in our method. In contrast, our model is conditioned by prompts which directly describe the desired new object, exactly like GaussianEditor (Object Adding functionality), Blended NeRF, DreamEditor, Reference-Guided Controllable Inpainting of NeRF, Repaint-NeRF. We would be grateful if the reviewer could elaborate on why this is a disadvantage. Please note that InstructNeRF2NeRF tackles the related but separate problem of scene-level editing. That is why we compare our method with state-of-the-art localised editing papers in the main text of our paper, and compare with Instruct-NeRF2NeRF only for completeness in the appendix.

“The quality improvement from the previous works is not that promising. Based on the visual results, it is quite difficult to say which one is showing the better results.” We have conducted a User Study which can be found in tables 3 and 4, rebuttal PDF, showing that our model’s results are preferred overall, compared with the existing works. We also show quantitative CLIP-based prompt-matching and temporal consistency results in Tables 1 and 2, which show that our model performs best overall.

“It is not fully clear how the authors perform only style changes, e.g the in Figure 5 (bottom row)...” This is explained in section G2 of our appendix, main paper. As mentioned on line 571, we use a Replace-only variant of our model for the Face scene, because “the torso region in which we aim to generate new content overlaps entirely with the mask region”. Therefore, fully removing the torso in a separate Erase stage is unnecessary. Note that our method is able to generate new geometric details on the clothes (collars and jacket lapels), whilst the baseline methods keep the original geometry fixed, only updating the texture (see Figures 5 and 9 main paper).

We are grateful to reviewer MnXs for their review! We are glad that they find our method “novel”, that it “enables localized 3D editing and obtains more realistic and detailed editing results”, that our experiments are “extensive”, and that our “ablation study validates the effective role of the key designs”.

We now address the reviewer’s specific concerns and questions: “The proposed approach is only suitable for the remove, add, replace editing… added objects should be within the pre-specified mask region”. This is an accurate limitation; the scope of our work is localised scene editing, on the same track as the works cited in lines 82 to 103 in the main paper.

“The training is time-consuming” Our training time is in line with other NeRF-based localised scene editing approaches, but we recognise that GaussianEditor provides much faster results due to its usage of the highly efficient 3D Gaussian Splatting representation. Nevertheless, there is a clear quality tradeoff in this case as we show in figure 5 of our main paper, table 1 main paper, and the user study results in tables 3,4 of the rebuttal PDF. As we also note in figure 14 of our main paper appendix, GaussianEditor requires manual adjustment of the new inserted object’s position, unlike our method which places objects automatically.

“Figure 5 shows that GaussianEditor makes less geometric modification compared to the proposed method. Is the main reason the Gaussian or NeRF representation?” We thank the reviewer for this interesting question! The main reason is the same as why Instruct-NeRF2NeRF also makes less geometric modification. Both of those methods use an iterative dataset update strategy to update the existing 3D scene (Gaussian in the case of GaussianEditor, and NeRF in the case of Instruct-NeRF2NeRF). In contrast, our method learns a localised 3D representation (Bubble-NeRF) for the edit region, from scratch. For this reason, our method is not biased towards the original scene geometry, and is able to make larger modifications.

“Is the proposed method able to replace the original object with a larger one?” We tried replacing the statue with a bus in the rebuttal PDF in figure 18b). We note that the model succeeds in adding a small bus onto the plinth. However, the significant size mismatch between the Erased and newly-added object seems to lead to degraded synthesis quality. “What if specifying a mask much larger than the original object?” We tried specifying a much larger mask than the original statue and generating “a super gigantic mechanical warrior robot” - results are in the rebuttal PDF, Figure 19b). We show the outline of the original statue object in red.

“What if erasing the object with the original property and replacing it with the object with the new property?” We tried replacing the statue with a gold statue, see results in rebuttal PDF, Figure 18a). Because our method generates a Bubble NeRF from scratch inside the edit region, the structural information regarding the original statue is lost, as stated in the limitations section of the main paper on line 654. Consequently, our model generates an entirely new statue, with gold appearance - we are happy to add this figure into the limitations section in the camera-ready. “It’s better to clarify the meaning of \theta_{bg} and \theta_{fg} in Section 3.2…” We thank the reviewer for this recommendation, and will be happy to fix this in the camera-ready version.

We are grateful to reviewer MzLe for their review! We are glad that they found our application “interesting and important”, that our Halo region method is a “nice solution”, and that the baselines we compare to are “sensible”. We now address the reviewer’s specific concerns and questions:

“Replacing the statue with something larger like a bus will not work…” We are grateful for this idea, and have generated new results for the statue scene, replacing the statue scene with a bus. As can be seen in Figure 18b) (attached Rebuttal PDF), the model adds a small bus onto the plinth. However, as the reviewer hypothesised, the significant size mismatch between the Erased and newly-added object seems to lead to degraded synthesis quality. We are happy to add this to the limitations section in the camera ready.

“In figure 3, the cookies look good but too gigantic in the scene…” This is true, and is a consequence of the generative prior from the inpainting model, which tends to generate objects that are scaled to fill up the masked region as much as possible, rather than a realistic scale relative to the scene. We would be happy to add a note on this to the limitations section.

“The Halo technique assumes that longer range interactions don’t exist … this would not be true for reflective surfaces.” This is true, and modelling reflective surfaces would be an interesting challenge for future work. We note that none of the existing localised scene editing papers can handle reflections either. In principle the Instruct-NeRF2NeRF-based global 3D editing papers may be able to handle reflective surfaces, but as far as we know this has not yet been demonstrated in practice.

“A table/quantitative results for Ablation studies would be more convincing. I'm especially curious about the impact of the depth reconstruction loss… a user study might make your claims stronger.” We thank the reviewer for these 2 great suggestions! We have now conducted a user study, and computed additional ablation results including quantitative results, shown in the rebuttal pdf Tables 5 and 6. The ablation results include a study on the impact of the depth loss, which is important for obtaining good quality results in the “Erase” stage. We also show SAM predictions (purple region) for segmentation masks in Figure 17, Rebuttal PDF, for the ablation model variants. Note that when the depth loss is removed, the Erased statue can still be detected by SAM, when applied to our model’s results. The same issue is observed when Halo loss is removed. However, SAM cannot detect the statue region when applied to our Full model’s Erase results, and instead correctly segments the hedge behind the statue.

For the user study, 15 participants were asked to compare our results with baseline models on Garden, Face and Fern scenes, using a variety of edit prompts. For each scene and prompt, participants were asked to indicate the best performing model in 2 categories. First, they were asked: “Which result most closely matches the given text prompt?”. And second, they were asked: “Which result shows the highest visual quality?” As shown in Tables 3,4 (Rebuttal PDF), our results were preferred overall in both categories.

“Do you trade speed/expressive power for quality when you use BubbleNeRFs?” Thank you for this interesting question! No, there is no quality tradeoff when using BubbleNeRFs. The tradeoff is that they only represent the localised mask region in the training dataset. As a consequence, our method enables us to dedicate almost the entire GPU memory capacity to querying the relevant rays corresponding to the edit region. This allows our method to work with higher final scene resolutions than InstructNeRF2NeRF (max resolution 512) or BlendedNeRF (max resolution 168); for example, we show results on the Fern scene at a final resolution of 2016x1512.

We cannot perform Novel-View-Synthesis of the entire scene using the BubbleNeRF, as it doesn’t model the scene background outside of the masked region. That is why we render our trained BubbleNeRF from every training set camera pose, and composite these renderings with the training images to obtain updated training images. These are then used to optimise a new NeRF or 3DGS scene (see Section 3.5, main paper), to obtain the final output.

“How were the language prompts chosen for objects to add?” We did not use a language model to generate prompts (as was done for example in InstructPix2Pix, as it requires a large training corpus of prompts), but rather simply wrote them out manually. Similarly to diffusion-based text-to-3D works, we experimented with adding real and mythical creatures (bunnies, corgis, dragons etc) as well as inanimate objects (including foods like pancakes and cookies) as can be seen in Figure 1 main paper.

We thank reviewer NhGj for their review! We are glad that they found that our ‘visualization results look good’, and that the pipeline is ‘novel and well-motivated’. We now address the reviewer’s specific concerns and questions:

“No ablation study was provided about HiFA” - we have now added this ablation study to the attached rebuttal PDF. Note that HiFA loss is highly related to SDS loss, and differs only by an RGB loss component (see eqn 4 in HiFA paper). In Figure 19a), (rebuttal PDF) we show Replace-stage results using simple SDS loss (with random diffusion timestep sampling) adding a corgi in the statue scene. The results show slightly less detailed texture synthesis as the RGB-space loss is removed. Nevertheless, note that our pipeline still synthesises the new object in the correct position, which is orthogonal to synthesis quality. We additionally provide quantitative ablation results in Table 6, which show that using simple SDS loss leads to only a slight drop in CLIP Text-Image Direction Similarity.

“If adding HiFA’s loss and design with Instruct-NeRF2NeRF or GaussianEditor, can they achieve comparable results?” - this is based on a misunderstanding; there is no SDS loss or other distillation loss in those methods. Instruct-NeRF2NeRF achieves scene editing using iterative dataset updates, which are obtained with a prior Instruct-Pix2Pix model. So it is not clear how HiFA loss could be incorporated into InstructNeRF2NeRF. Similarly, GaussianEditor employs the same training strategy for its Edit functionality. It differs from InstructNeRF2NeRF mainly in its choice of 3D representation (3DGS instead of NeRF). Both methods use an iterative dataset update strategy to update the existing 3D scene. In contrast, our method learns a localised 3D representation (Bubble-NeRF) for the edit region, from scratch. For this reason, our method is not biased towards the original scene geometry, which enables larger modifications.

Finally, regarding the GaussianEditor object Addition functionality, this makes use of an off-the-shelf image-to-3d method (Wonder3D, which also does not use distillation) to generate the new object separately (which does not guarantee harmonisation with the original scene), and leaves it to the user to manually adjust the new object’s depth (see section F appendix main paper, and Figure 14). As we note in the main paper (line 226) this results in visible artefacts at the boundary between the object and its surroundings, and significant quality issues (see Figure 5, and User Study Table 3 in the Rebuttal PDF). It is unclear how these issues could be addressed using HiFA’s loss and design.

“Some of the editing results are not good enough. For clothes editing, there is always an obvious cutting edge on the neck…” We only claim that our model outperforms the existing state-of-the-art on this task. We would be grateful if reviewer NhGj could indicate which published works can perform better, in terms of synthesising intricate localised texture/geometry patterns to match the edit prompt. We compared with the state of the art GaussianEditor in figure 5, and note that it fails to update the geometry of the clothes to match the edit prompt, whilst the texture synthesis is also inferior to our model. We would like to improve the generated geometry still further in future work, but still note that our model performs better than existing models on this challenging scene.

“The replacement task is easier than standard instruction-guided editing … this makes some comparisons unfair, eg Fig.9” - Please note that in the main paper, we compare with state-of-the-art localised editing methods; Reference Guided Inpainting, Blended-NeRF, GaussianEditor, as well as DreamEditor and RePaintNeRF in the appendix. Each of these uses a spatial prompt to indicate the edit region; for example, GaussianEditor uses a SegmentAnything-based mask to select the edit region, exactly like ours. In Figure 9 (appendix) we compare with Instruct-NeRF2NeRF (standard instruction-guided editing) only for completeness - nevertheless, all main paper comparisons are like-for-like. Note that GaussianEditor (Addition functionality), Reference-Guided Inpainting, Blended NeRF, Repaint-NeRF and DreamEditor all use edit-descriptive prompts as guidance exactly like ours rather than being “instruction-guided”. We note that our Erase-and-Replace strategy for scene editing obtains better results than the existing SOTA 3D localised methods. Reducing the task to Erase-and-Replace should therefore be considered part of the appeal of our method.

“The authors need to argue why this setting is reasonable compared with traditional instruction-guided editing…” Localised 3D scene editing is a well-established research track with multiple related publications (please see our Related Work section, in particular lines 82 to 103). These works may be seen as a 3D extension of 2D conditional inpainting research works such as RePaint and Blended Diffusion, and have myriad potential applications in Mixed Reality and Film Production. These applications require 3D object editing functionalities whilst keeping object surroundings consistent. Note that this is not possible for general scene-editing frameworks as they a) tend to modify the entire scene, (Fig. 9 and 10, main paper, and Fig. 3 in the DreamEditor paper) and b) struggle with object removal (Fig. 9 main paper). Therefore, the localised 3D scene editing track has emerged to address these shortcomings, and should not be considered ‘easier than standard instruction-guided editing’ as it is simply addressing a different task.

“The paper has multiple presentation defects, especially the math formulas…” We are grateful to the reviewer for bringing this to our attention! We will fix these defects for the camera-ready version.

I thank the authors for their rebuttal. Most of my concerns are addressed, but more concerns are raised.

---

“Some of the editing results are not good enough. For clothes editing, there is always an obvious cutting edge on the neck…” We only claim that our model outperforms the existing state-of-the-art on this task.

I double-checked Fig.10 in the paper after receiving this response, but more concerns are raised about the provided results: The artifacts in baseline EN2N and ViCA-NeRF's results are counter-intuitive, even unlikely to exist in a valid setting of these baselines.

• In Fig.10, EN2N generates high-quality Tartan jacket results but unreasonable results in the "checkered jacket" task. However,
  • In Fig.9, IN2N generates failed but reasonable results of the "checkered jacket" task, at least with a clear face.
  • Given EN2N is an improvement of IN2N with same diffusion model IP2P, I wonder why EN2N cannot produce at least the same face-clear results as IN2N.
• In Fig.10, ViCA-NeRF generates two images with lots of artifacts
  • I observed that the two images have very similarly structured artifacts, e.g., in the right images of both rows in ViCA-NeRF's results, the top-left part of the head has some hair-like artifacts.
  • Given that ViCA-NeRF is mainly a warp-based method (based on NeRF-predicted depth) while both editing tasks do not change any geometry of the head part, I wonder why similar artifacts appear in different edits - were these artifacts already existing in the original scene input to ViCA-NeRF?

I would like the authors to provide some analysis of these counter-intuitive artifacts. Given the code of EN2N and ViCA-NeRF is publicly available, I would also like the authors to provide the commands that they used to obtain the results in Fig.10, e.g., ns-train en2n ... or ns-train vica ..., so that I, along with other reviewers and AC, can reproduce these results on our side.

I am paying close attention to this newly-noticed concern, since it is related to whether the baselines are compared in a fair setting, e.g., at least with reasonable hyperparameters.

---

“If adding HiFA’s loss and design with Instruct-NeRF2NeRF or GaussianEditor, can they achieve comparable results?” - this is based on a misunderstanding; there is no SDS loss or other distillation loss in those methods.

In fact, the formulas (2)(3) in HiFA's paper have actually proven the equivalence between SDS loss and iterative DU's supervising rendered image with edited image. In (3) in HiFA's paper, the LHS is SDS loss, and RHS is a weighted term of $\|z - \hat{z}\|$, which is exactly the MSE loss between the latents of the original image and generated image (in one denoising step). Given that iterative DU's loss is between the original image and the generated image (in multiple denoising steps), they are actually equivalent.

As we are unable to discuss on new results, I would expect the authors to provide some high-level analysis of how the techniques in HiFA, e.g., formula (4)(5)(6)(7)(8)(9) improve the results. It will be better to provide some of these ablation studies in revision.

Dear AC,

Thank you for following up and acknowledging my efforts. Your understanding is correct - the degrading of baseline results from an unintentional misuse (in my opinion) of --downscale-factor 2, while the optimal command for a fair comparison is the default one without it.

I provide the results through this Anonymous Github Repo: https://anonymous.4open.science/r/neurips24_15922_review_NhGj-5684. As I do not have much time to render them as videos (also, some checkpoints are not saved due to crashing), I provide the results as Wandb validation images, where the left part is the original view and the right part is the edited view. These views are randomly sampled from all the validation views, so they might be different in different runs, but they should be sufficient for us to see the quality and artifact patterns. I apologize for the inconvenience here.

Here, I would like to provide some results in images from the same viewpoint. Unfortunately, OpenReview does not support image markdowns ![](), and I can only provide the links. All the results except for the "Tartan jacket" one are "checkered jacket".

• EN2N
  • EN2N Default
  • EN2N w/ Downscaling Run1
  • EN2N w/ Downscaling Run2
  • EN2N w/ Downscaling "Tartan Jacket"
• ViCA-NeRF
  • ViCA Default
  • ViCA w/ Downscaling

It can be seen that the pattern roughly matches the ones shown in Fig.10, and there is noticible difference between w/ and w/o downscaling. Even though the results w/ default (i.e. optimal) commands are still not better than the proposed method's (proposed method produces checkered patterns in common sense, but baselines can only produce scaly textures), it is still necessary for the authors to update their Fig.10 in their revision.

I hope these results could help the AC, other reviewers, and authors to understand the difference between the two settings. Thanks.

I sincerely thank the authors for the detailed follow-up discussion.

---

On the contrary, SDS loss and IDU are quite different, as discussed in the InstructNeRF2NeRF paper, Ablation Study section.

Again, I appreciate the authors for their detailed responses and arguments on this topic. Though I still do not fully agree with some arguments on the difference between SDS and IN2N - I also know and understand that the practical or empirical performance might be (even significantly) different, but I do not think there is a large difference in theoretical or semantic (like using different coefficients for L2 regularization may lead to success or failure in training but they do not have much difference) - I will not ask for more discussion on this point, the authors are correct that these are a little out of scope.

I also appreciate the authors for mentioning which parts of HiFA to adopt in their model: "We therefore adopt both HiFA loss (equation 4 in HiFA) and timestep annealing (equation 5 in HiFA). Nevertheless, we validate this choice with the new ablation studies in the rebuttal PDF. " However, I would like to point out that the point of "timestep annealing" is never mentioned in the main paper, while this, may contribute a lot to the generation quality. I understand that this method is widely used and is already a part of the model design, and I will not ask for an ablation study on this part. I would highly suggest the authors to add this information in the revision, to improve the reproducibility.

Also, for the revision, I would kindly suggest that the authors mention fewer "adopt HiFA," "combining HiFA," and "utilize HiFA" in their revision - these may mislead the audiences (like me) that "the pipeline is highly utilizing the whole HiFA." Instead, they may directly mention the name of the components, like "HiFA's loss," and "timestep annealing strategy inspired by HiFA" to focus more on the components they adopted from HiFA, instead of HiFA itself.

---

As for the baseline issues, in fact, these days I have also tried to reproduce the baseline results with all default settings:

# train a nerfacto model to start, with default settings
ns-train nerfacto --data <face dataset> 
# EN2N: default arguments of EN2N provided in their Github repo
ns-train en2n --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt "give him a checkered jacket" --pipeline.guidance-scale 7.5 --pipeline.image-guidance-scale 1.5
# ViCA-NeRF: default arguments of ViCA-NeRF
ns-train vica --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt "give him a checkered jacket"

And I obtained the results. As the policy about whether reviewers can post external links is not clear, I will instead describe the contents of images here. If AC permits, I will also post some images through an Anonymized Github's repo.

• EN2N: the appearance is like IN2N, but slightly more blurred. The jacket was modified to add some scaly texture (or checkered texture with relatively small grids). I did not observe the fully unreasonable results like in Fig.10.
• ViCA-NeRF: the appearance is that the jacket was changed to a suit with a scaly texture, while the collar part of the person has some blurred, checkered-like patterns. I did not observe any artifacts similar to those in Fig.10, either.

I noticed that the authors use a different command with an additional nerfstudio-data --downscale-factor 2. Perhaps these are some settings inherited from IN2N or some other baselines? I am not sure whether this is the reason why the results are worse and unreasonable.

In this case, I will not regard the credibility level of the results in this paper as unreliable at this point. However, I will try to reproduce the results with additional nerfstudio-data --downscale-factor 2 during reviewer-AC discussions, and then decide whether and how to adjust the rating.

I would also humbly encourage and request other reviewers that, if you have time and spare GPUs, you may also try to reproduce these results to check them from your side.

Finally, again, I sincerely thank the authors for the detailed follow-up feedback. I understand that my previous comment pointed out an issue not observed in the original review and was not sent in a very timely manner, and I sincerely apologize for this. I would also suggest the authors revise their Fig.10 with the results without nerfstudio-data --downscale-factor 2 in their revision, to achieve a fair comparison with the baselines - I agree that the proposed method is still better than the baselines in this version, and this will not change any rating about the quality.

# Nerfacto
ns-train nerfacto --data <face dataset> nerfstudio-data --downscale-factor 2
# EN2N
ns-train en2n --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt "give him a checkered jacket" --pipeline.guidance-scale 7.5 --pipeline.image-guidance-scale 1.5 nerfstudio-data --downscale-factor 2
# ViCA-Ne
ns-train vica --data <face dataset> --load-dir <nerfacto checkpoint> --pipeline.prompt "give him a checkered jacket" nerfstudio-data --downscale-factor 2