3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting

Q1: The paper does not demonstrate the editing procedure for practical use.

A1: Thanks for pointing this out. We have included an additional section to introduce the editing procedure in the Appendix. To ease and automate the procedure, we have developed an user interface for user to interact with our algorithm.

Q2: Missing details.

A2: We have included additional implementation details in the supplementary materials. Furthermore, we will release the code as soon as this work gets accepted, as we believe this will greatly assist the audience in using and reproducing our work. For camera configuration, $K, T$ are fixed during scene initialization stage.

Q3: it appears that editing primarily involves "text-guided" editing by querying specific object + object manipulation?

A3: Yes. We made an illustration (Figure A1-2) in the Appendix to show how the editing works.

Q4: Is "camera control" achieved always by changing camera instrincs and extrinsics parameters? But in Fig.1, it seems that text prompts are also used for control.

A4: Camera control is primarily achieved by adjusting the camera intrinsics and extrinsics. However, these factors do not need to be directly exposed to the user. Instead, we utilize the LLM to interpret users' text prompts and convert them into the necessary camera parameters.

Q5: In addition, there's no examples of bounding box usage in the paper.

A5: We provide examples of bounding boxes (user-created rough painting masks) in the Appendix.

Q6: Since the 3D-aware editing is based on explicit 3DGS, and the disenangled object manipulation does not involve whole image re-generation.

A6: 3D Gaussian Splatting does not explicitly model shadows and lighting; instead, it bakes them into the appearance. Consequently, our method, which uses 3DGS as the underlying representation, also lacks the ability to model these lighting factors. One potential solution is to consider regenerating the entire image, for instance, by applying SDS steps across the whole region. Another promising direction is to explore 3DGS with explicit lighting modeling.

Q7: What are the advantages of proposed semantics lifting schemes over existing language-semantics 2D-to-3D lifting works?

A7: Existing works such as LERF, LangSplat, and Gaussian Grouping begin with a well-optimized 3DGS and then lift 2D semantics to 3D. In contrast, our work, which focuses on image editing, starts from a single image. We show that 2D-to-3D semantics lifting can effectively lead to high-quality 3D-aware image editing. We hope that this could

As the rebuttal deadline approaches, we would like to know if you have any further concerns regarding our work. We would appreciate the opportunity to clarify any issues. If there are no additional concerns, could you please kindly consider updating the score accordingly? Thank you!

Q1: The implementation details are not clearly articulated.

A1: We have included additional implementation details in the supplementary materials. Furthermore, we will release the code as soon as this work gets accepted, as we believe this will greatly assist the audience in using and reproducing our work.

Thank you for your response~

Q1: I am afraid that the SAM mask is not accurate, the distill loss may cause some artifacts. Like some of the gaussians in the edge may not be able to match the image prompt.

A1: While it’s true that the SAM mask can be inaccurate, this does not lead to artifacts. With our proposed Layout Augmentation, any mismatched Gaussians at the edges will not move with the object, allowing them to be pruned during optimization. Figure 7 clearly demonstrates the ablation study for Layout Augmentation. Without this, the inaccurate mask can indeed cause floating artifacts, but Layout Augmentation effectively resolves this issue.

Q2: Current method cannot deal with shadow and lighting.

A2: Yes, this is true. 3D Gaussian Splatting does not explicitly model shadows and lighting; instead, it bakes them into the appearance. Consequently, our method, which uses 3DGS as the underlying representation, also lacks the ability to model these lighting factors. Although explicitly modeling lighting in 3DGS is a promising direction for future research, it is beyond the scope of our current work. We believe our approach could greatly benefit from advancements in this area.

Q3: How long does it take for training one scene? How long does it take for rendering one image?

A3: Training a single scene takes 10 minutes. We can achieve a rendering speed of 30 frames per second.

Q4: Is ther anything generative that enables this capability?

A4: As mentioned in lines 183-186, we utilize Stable Diffusion to inpaint the unseen regions, which may arise from novel views or occlusions. Specifically, we inpaint the occluded areas behind the object of interest with the generative prior. This is why the case of “removing the headscarf and then moving the camera” in the teaser can generate the woman's hair.

Q1: Limited novelty. The proposed method largely combines existing techniques.

A1: We agree with the point that the techniques used in our work originates from previous works. However, we would like to emphasize that these techniques were originally designed for different tasks—such as LucidDreamer for image-to-3D conversion without editing capabilities, and DreamFusion for text-to-3D generation—which differ significantly from the application and task that we focus. We believe the value of our work lies in demonstrating a viable approach to unify these techniques for 3D-aware image editing. Additionally, our work can continue to benefit from advancements in these areas in the future.

Q2: It is unclear how the proposed method outperforms prior techniques, such as LucidDreamer and AdaMPI.

A2: While prior works like LucidDreamer and AdaMPI can potentially generate novel view content through inpainting, they do not support 3D-aware object-level editing. In contrast, our work enables both camera control and object editing within a unified framework.

Q3: Since the method relies on depth maps to generate 3D Gaussian splats, it appears limited in its ability to handle significant changes in view angle.

A3: We use depth estimation only as a rough initialization, with the scene geometry refined during SDS optimization. Additionally, our method can be easily adapted to accommodate significant changes in view angle. We can progressively expand the azimuth range and inpaint novel views according to a schedule (for example, [−15,15] for the first 300 steps, followed by [−30,30] for the next 300 steps, and so on). In terms of image editing, we believe current method can meet the most demands of users and be further improved in future.

Q4: The method seems to only function with generated images rather than real-world inputs.

A4: Our work works well with real-world images (see the first two samples of Figure 4, all three samples of Figure 5, first two samples of Figure 6, and all the samples of Figure 7).

As the rebuttal deadline approaches, we would like to know if you have any further concerns regarding our work. We would appreciate the opportunity to clarify any issues. If there are no additional concerns, could you please kindly consider updating the score accordingly? Thank you!

Q1: The main weakness of this paper is that it mostly combines existing techniques in a pipeline to improve an application rather than introducing new technical insights.

A1: We agree with the point that the techniques used in our work originates from previous works. However, we would like to emphasize that these techniques were originally designed for different tasks—such as LucidDreamer for image-to-3D conversion without editing capabilities, and DreamFusion for text-to-3D generation—which differ significantly from the application and task that we focus. We believe the value of our work lies in demonstrating a viable approach to unify these techniques for 3D-aware image editing.

Q2: Evaluations are very limited.

A2: We acknowledge that the number of participants in our human study is small; however, we want to emphasize that the results are significant, which clearly demonstrate our performance improvement over previous methods. We also agree with your point about the absence of an objective metric for measuring image editing, as it is a complex and subjective task. We believe this is an interesting direction worth exploring in the future.

Q3: What is the runtime of the proposed method and other alternatives?

A3: Our method requires 10 minutes to process a scene, while most alternatives, such as AnyDoor and Object 3DIT, are feed-forward methods that take significantly less time (less than one minute). However, these alternatives do not achieve precise 3D-related editing. To reduce the time cost of our method, it would be beneficial to explore the use of amortization to incorporate per-scene optimization into a generalizable feed-forward model.

Q4: The teaser figure seems to suggest that this process is "hands-off”.

A4: Yes, the process is quite user-friendly. We illustrate the whole process in the Appendix. We explore two methods to achieve this: (1) using a large language model (LLM). We prompt the LLM to take the user's editing request as input and parse it into predefined components, such as camera movement angles, descriptions of the object of interest, and the transformation matrix for that object; (2) providing an interactive panel for refining the edits. We have implemented a web-based interactive panel that allows users to control the editing process. Draggable sliders facilitate various edits, including moving, removing, and rotating objects, so users do not need to interact with the code throughout the entire editing workflow.

Q5: To what extent does the method begin to fail if there are large baseline camera movements?

A5: Our method can be easily adapted to accommodate large baseline camera movements. We can progressively expand the azimuth range and inpaint novel views according to a schedule (for example, [−15,15] for the first 300 steps, followed by [−30,30] for the next 300 steps, and so on). However, since our primary focus is on image editing, we believe that significantly altering camera poses is outside the scope of our work, and therefore we do not emphasize this capability in the paper.

Q6: Are there other downstream tasks where this pipeline could be useful? Maybe other instances of image editing?

A6: Our method enables basic 3D editing capabilities, including translation, rotation, removal, and resizing, for image editing. Additionally, it can be integrated with other image-based or 3DGS-based editing techniques. For instance, we can combine our approach with GaussCtrl [1], ProEdit [2], or MvDrag3D [3] to alter the shape of an object within a given image.

[1] GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editing [2] ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing [3] MvDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Priors