3DGS-Drag: Dragging Gaussians for Intuitive Point-Based 3D Editing

Thanks for your informative comments. We are glad to address your concerns below. The additional experimental results discussed below are provided at the following anonymous link.

W1:it is important to acknowledge these limitations for quantitative evaluations.

We agree that quantitatively evaluating diffusion-based methods is challenging due to the potential biases, and we have included this acknowledgment in the revised limitation Sec. C.3. This is a common issue in this field, and we followed previous works (e.g., ViCA-NeRF, GaussianEditor) to conduct the user study, as well as using GPT evaluation to mitigate potential personal bias. However, such a problem cannot be fully addressed, and remains an important challenge for future research in the field

In addition, to help relieve such bias in the user study, we conducted a scaled user study with 99 participants. The results, shown below (also in Figure R5), still demonstrate that we significantly outperform the baselines. We are open to adopting any further suggestions to improve the quantitative evaluations.

W2: The qualitative figures do not clearly demonstrate the effectiveness of local editing.

Thanks for your suggestion to improve our visualization. We use a bounding box to highlight the changed region (Figure R6). Therefore, the reader can better notice which regions are changed and which are not. Such revision has also been made in Figure 5 of our revised paper. In Figure 7 of our paper, we also conducted an ablation study without using local editing, where the background was significantly changed, showing the importance of our local editing.

In addition, to better illustrate the effectiveness of local editing, we calculate the similarity between the edited result’s background and the originally rendered image in the unmasked pixels. As shown in the results below, all the metrics are much better when using local editing compared to not using it. Therefore, local editing is necessary to preserve the background.

Q1: the method’s ability to handle larger object edits within the scene, such as moving a table or a truck.

We are glad to provide the experiments you suggested in Figure R1. Specifically, we conduct experiments on large-scale object movements. Our tests include:

• move the flowerpot from the center of the table to its edge,
• move the large table,
• move the large truck.

As the results show, our method works well in both cases suggested. In addition, we can conduct long-range movements like moving the flowerpot to the side of the table. These experiments demonstrate the generalizability of our method.

Thanks for your comments. We are happy to address your concerns below. The additional experimental results discussed below are provided at the following anonymous link.

W1: Whether the proposed method is very similar to Instruct-NeRF2NeRF or not?

We would like to clarify that our method differs significantly from Instruct-NeRF2NeRF in various aspects, beyond just the use of NeRF or 3D Gaussian Splatting. The differences are:

• Different Setting: Instruct-NeRF2NeRF focuses on appearance and style editing, with minimal ability in geometry editing. In contrast, our method is capable of drag-based intuitive editing, which can achieve fine-grained and geometry-related editing. As shown in the comparison in Figure 6, Instruct-NeRF2NeRF fails in these cases.

• Different framework and techniques: Instruct-NeRF2NeRF directly uses Instruct-Pix2Pix to update the dataset and train the NeRF progressively. In contrast, we incorporate deformation guidance and diffusion guidance, together with a multi-step drag scheduling module. All these proposed components are crucial to successfully achieving challenging intuitive drag editing in our approach.

• Difference in the diffusion process: Instruct-NeRF2NeRF directly applies the Instruct-Pix2Pix model to obtain the edited image. However, instruct-Pix2Pix cannot consistently edit images with geometry changes, often preserving a layout similar to the original image. In contrast, our approach fine-tunes a Dreambooth model to the scene, adds noise to the deformed image, and generates the edited image from this noised version. The results show that our approach correctly maintains the geometry change from the deformed image while generating consistent content.

Q1: A more detailed explanation of Annealed Dataset Editing.

First, we would like to clarify that annealed dataset editing is one part of our contributions, not the center of the contribution. As mentioned in Lines 93-95, our primary contribution is proposing a framework for intuitive drag editing in 3D real scenes involving a deformation approach, an effective diffusion guidance technique, and a multi-step scheduling module. 3D geometry-related content editing is particularly challenging for pure diffusion-based methods like Instruct-NeRF2NeRF, whereas our method can achieve fine-grained control over these edits.

Second, we would like to elaborate on the difference with Instruct-NeRF2NeRF in terms of annealed dataset editing. A qualitative ablation of using different dataset update strategies is shown in Figure R2. In Instruct-NeRF2NeRF, they edit one view each time and iteratively update the dataset. As a result, they cannot change the geometry due to inconsistent constraints from other unedited views, leading to a degenerated result in the original scene. Instead, since our method allows for consistent edits from the start, benefiting from our deformation and diffusion process, it updates all views simultaneously. To further improve the details, we perform such updates several times with the annealed strength of the diffusion model. As shown in Figure R2, the annealed dataset editing is crucial to making the edit successful.

Q2: Writing questions.

• Line 197 (the definition of $d$): $d$ represents the dimension size for color features. Specifically, 3D Gaussian Splatting uses SH coefficients to represent color $c$, enabling view-dependent effects. In practice, $d$ is given by $d = 3 \times (\mathrm{degree}_{S} + 1)^2$, where the SH degree $\mathrm{degree}_S$ is set to 2, resulting in a dimension of 27 for $d$.
• Line 240-241 (different meaning of $k$) and the Minus symbol in Equations (1) and (2): Thanks for pointing these out, and we appreciate your corrections. We have revised them to $K$ in the current revision.
• Figure 4 is not referenced in the text: Thanks. Figure 4 illustrates the relocated object position during the multi-step drag operation. We have revised our paper to reference it in Sec. 3.5 in the revision.

Thanks for your encouraging comments, and we are happy to address your concerns below. The additional experimental results discussed below are provided at the following anonymous link.

W1.1: There is a noticeable color discrepancy between the original and the widened sections.

We conduct a study on this in Figure R4. Initially, we use a small radius for handle points, which leads to noticeable gray artifacts on the boundary of the wall. The color discrepancy happens because that gray part is recognized as a potential color shift from the diffusion model (Figure R4 (a)). Thus, although the edited result can correctly model the wall, the color is shifted. We also find that such a problem can be simply addressed by changing to a larger radius to avoid artifacts on the boundary (Figure R4 (b)). Since the wall can be correctly modeled in both cases, we believe that our approach remains robust and effective without introducing noticeable artifacts. The detailed texture is generated through diffusion guidance, while our intuitive dragging operation provides the flexibility to refine the results at a fine-grained level.

W1.2: The leaves near the edited boundary on the wall appear blurry.

The blurry appearance of leaves near edited boundaries stems from their high-frequency nature, which presents a significant challenge for existing 3D editing methods. As shown in Figure 6 of the main paper, our approach achieves notably better sharpness and clarity in these areas than baseline methods.

W2: Inaccurate claim when comparing baselines. Thanks for pointing this out, and we apologize for this inaccurate and confusing claim. We have modified our claim to “there is no directly comparable work on intuitive 3D drag operation in real scenes” in the revised paper (Line 467). We also revised our paper to reference these two related works in Lines 146-147.

Compared with the mentioned papers, we have the following important differences:

Different settings: Both Interactive3D and APAP focus on single object editing without background, but our work focuses on large real scenes, where we need to deal with challenges like photorealistic objects, complex backgrounds, lightning, and inaccurate geometric modeling. Thus, these methods cannot be applied in our setting, since they are designed for single objects with well-captured geometry.

Technical difference: Both Interactive3D and APAP follow the paradigm of using the Score Distillation Sampling (SDS) loss for editing. Such a loss is initially proposed for single object generation. In contrast, our approach introduces a deformation-and-correction paradigm to address the challenges of photorealistic editing in real scenes. In summary, these object-centric drag editing methods are not applicable for comparison in our setting.

Q1: Can this method reposition the entire flowerpot (as in Fig. 2) rather than only elongating it?

Thanks for your constructive question. Yes, our method can successfully reposition the entire flowerpot. As shown in Figure R1, we can reposition the flowerpot from the center of the table to its edge.

Thanks for reviewing our paper and the constructive comments. We improve our paper based on your suggestions and address your concerns below. The additional experimental results discussed below are provided at the following anonymous link.

W1.1: Will the method fail if we move a large object to a large range?

In most cases, our method will succeed. We conduct experiments on large-scale object movements in Figure R1. Our tests include:

• move the flowerpot from the center of the table to its edge,

• move the large table,

• move the large truck.

As the results show, our method works well in each case.

Meanwhile, we have identified limitations when moving objects from the background to the foreground (Figure R3 (a)). This occurs primarily because the background objects are initially partially observed and not sufficiently modeled in 3D. Thus, they do not have correct rendering results for other views when dragged to the center. We have also revised the paper to include this in our limitation section (Sec. C.1).

W1.2: Include more failure cases.

In addition to the background modeling, our discussion of failure cases also includes the boundary region issue in Figure R3 (b). When drag operations partially extend beyond most cameras' visible area, optimization becomes challenging due to limited edited image availability. In our original submission, the limitation section also discussed such limitations (now it is Appendix Sec. C.2 in the revised paper). We are happy to discuss this if you have additional comments.