MVDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Prior

Minor difference in Fig. 3. Thank you for your detailed observation. In the revised paper, we use yellow dashed boxes to highlight the regions with evident differences. Additionally, we compute the CLIP similarity between the reconstructed images and the original image to quantitatively demonstrate the difference. Specifically, the CLIP similarity between (a) and (d) is 88.59, whereas it improves to 93.59 between (a) and (e) by the DDIM inversion with random noise. we also included a new example in Fig. 10 of the Appendix to further illustrate the advantages of DDIM inversion with random noise.

Theoretically, getting an inverted noise that aligns with the Gaussian distribution is important for editing, as evidenced by numerous prior works (e.g. I2VEdit, DragDiffusion, etc.). However, the noise derived from the original DDIM inversion clearly deviates from a Gaussian distribution, as shown in Figs. 3 and 10 (b). By introducing a straightforward random noise processing step, the DDIM inversion noise becomes more consistent with a Gaussian distribution, as illustrated in Figs. 3 and 10 (c), thus leading to better editing results.

Similar results in simple cases. Thank you for your careful comment. Yes, for simpler cases without the need for significant structural changes or the generation of new content—such as the shoe dragged to be shorter in Fig 6—both our method and the baselines (e.g. APAP) perform well. However, we would like to emphasize that the key strength of our method lies in its ability to handle significant structural changes and generate new content during drag editing, which is a very challenging task even though the 3D objects are simple. Examples include a bird opening its wings (Fig. 1), a lion opening its mouth while generating teeth (Fig. 1), a flower blooming (Figs. 6 and 9), and a leg lifting in a more articulated manner (Fig. 6). In these cases, our method produces results that are accurate, multi-view consistent, and visually attractive. To the best of our knowledge, no existing method achieves comparable results. We will include more challenging cases.

Clarity of technical formulations. In the section on Gaussian appearance optimization, we have revised the technical description to make it clearer and easier to understand.

The “densification and pruning” process involves creating, removing, or adjusting geometry that is inaccurately positioned or sized. The pruning step removes Gaussians with opacities below a specified threshold, ensuring cleaner geometry. The densification stage is straightforward: it clones Gaussians to improve the coverage of small-scale geometry. For example, if a small-scale feature is represented by a single large splat, it is split into two smaller splats to provide a more detailed representation.

Since this is a standard step in 3D Gaussian Splatting [Kerbl et al., 2023], we adhered to the implementation provided in the original paper and its released official code. As such, we chose not to elaborate further on it in the main paper.

Regarding the “shoe” example in Fig. 5, we have replaced it with a new example. Additionally, we have included an ablation video in the supplementary materials, showcasing more examples to demonstrate the effectiveness of this step. Furthermore, since the evaluation of appearance is subjective to some extent, our use of GPTEval3D revealed an improvement in the Texture Details metric after SDS optimization, as shown in Table 2.

Human evaluation. Thank you for your constructive suggestions. We have conducted a questionnaire survey as shown in the appendix (Table 3 and Sec A.8).

View selection. The current four view choice is primarily motivated by two reasons: (1) These four views can already express the 3D drag trajectory relatively completely. Even in cases where the drag direction aligns with one of the four views, it will still be visible from another side view unless complex occlusion occurs. Our experiments have demonstrated that these four views perform well in most cases. (2) The multi-view generation and reconstruction models we employed exhibit a strong bias toward these specific views. For example, if we use the other four fixed orthogonal views, we observe a significant degradation in the reconstruction quality of LGM.

We also acknowledge that relying solely on orthogonal views may not be optimal and this highlights an intriguing future direction: exploring dragging on native 3D generation models (e.g., CLAY) or video-like multi-view diffusion models, like SV3D, which supports 21 views.

Some details lost by only 4-view rendering. For a more detailed discussion, please refer to our response to Weakness 2 from Reviewer xmM9.

Multi-view reconstruction. We understand that the term "multi-view reconstruction" in the comment refers to the traditional method of 3D reconstruction using multiple posed input images. We agree that this approach is theoretically possible. However, our setting, actually, presents a sparse-view 3D reconstruction problem, using only four views. Traditional reconstruction methods typically require at least dozens of posed views to achieve satisfactory results. Although recent advancements, such as ReconFusion, have shown promise, they often face challenges like quality degradation when the number of input views is rather limited.

In contrast, a key observation in our work is that pre-trained feed-forward multi-view reconstruction models (e.g., LGM, GRM, and Lara) have embedded strong geometric priors for object reconstruction and can offer significant advantages, including generally plausible quality, fast inference (in seconds), and ease of application. Therefore, we directly leverage these multi-view reconstruction models in our approach.

Directly applying image-conditioned SDS optimization. Yes, directly performing image-conditioned SDS optimization can work to some extent. However, this process often results in less visually pleasing results (see the new video demo). We analyzed the main reasons as follows: The 3D GS produced by LGM is under-reconstructed, exhibiting a certain degree of geometric misalignment and texture loss. The inherent ambiguity of the SDS loss makes it challenging to achieve an optimal result when simultaneously optimizing both the texture and geometry of the 3D GS, which has an excessive degree of freedom for optimization. To address these issues, we introduced a deformation network that serves as an effective regularization to eliminate low-frequency geometric mismatches. With this better-aligned 3D GS, we then apply SDS to fine-tune its appearance. This divide-and-conquer strategy leads to improved visual effects.

SDS produces better DAI on 3DGS than on meshes. The DAI value depends on the accuracy of multi-view image editing and the quality of 3D reconstruction (including initial reconstruction and two optimization stages). Regarding why SDS produces better DAI on 3D GS than on meshes, we analyze the reasons as follows:

Firstly, for the accuracy of multi-view image editing, the 3D GS models we selected are generated using LGM, which relies on MVDream first to generate four images and then reconstruct the 3D GS accordingly. As a result, the image distribution of LGM aligns more closely with the training distribution of MVDream, giving better DDIM inversion and more accurate multi-view editing. In contrast, the renderings of mesh models, even with the use of certain techniques, cannot ensure perfect inversion, causing some detail loss during editing (see Figs. 3 and 10). Secondly, during the SDS-based appearance optimization, we compute the LPIPS loss using the four edited images. This process may propagate errors originating from these edited images. These combined factors ultimately impact the DAI value, which is worse for mesh models compared to 3D Gaussian models.

Lastly, as analyzed above, we note that SDS-based appearance optimization cannot perfectly guarantee dragging accuracy. However, we incorporate this design to enhance the visual quality with sharper and clearer details. To illustrate the differences between using SDS and not using SDS, we have included additional video results in the revised video demo. This observation is further supported by the Texture Details metric evaluated using GPTEval3D (see Table 2 in the revised paper).

Final output when inputting mesh. The final output of our method is 3D GS for 3D meshes. To ensure a fair evaluation, we compute all quantitative results on the images rendered from these outputs (either 3D GS or meshes).

Low-quality 3D creations. Our method may slightly blur the original 3D content. This issue arises mainly due to three factors: (i) By rendering only 4 orthogonal views, some details (shape and texture) of the 3D object may be lost; (ii) The resolution of the multi-view diffusion model used (MVDream) is currently limited to 256×256. Consequently, even if high-quality images are rendered from the 3D objects, they must be downsampled to fit this resolution, leading to inevitable detail loss; and (iii) The reconstruction model (LGM) may introduce additional losses in both geometry and texture details.

Fortunately, our method is highly flexible and can incorporate advancements in 3D foundational models to address the artifacts. For example, Instant3D, with its 512×512 multi-view diffusion model, or Lara and GRM, which improve geometry reconstruction, can seamlessly replace MVDream and LGM in our pipeline. Moving forward, we plan to incorporate Lara and Instant3D into our method, which will offer stronger reconstruction priors and generation priors respectively.

Finally, we would like to emphasize that the key strength of our method lies in its ability to handle significant structural changes and generate new content during drag-based editing. For example, our approach enables edits such as a bird opening its wings (Fig. 1), an animal opening its mouth (Figs. 1 and 2), or lifting its leg in a more articulated manner (Figs. 1 and 6), and a flower in bloom (Figs. 6 and 9). These types of edits are not effectively supported by existing methods.

Failure cases. The intended edit in the owl case is to close the suit (Fig. 12 in the revised paper). While our result achieves this goal to some extent, it still fails to reach the final position perfectly and introduces an unintended style change in the tie. This limitation arises because accurately adjusting the suit’s position necessitates significant modifications to the tie area, where nearly half of the tie will be overlapped by the suit. Consequently, the gradient-guided editing mechanism modifies the latent noise in this region and completely relies on the diffusion prior to generate a semantically plausible in-distribution result. However, this process inherently entangles dragging accuracy (e.g., closing the suit), identity preservation (e.g., maintaining the tie’s style), and global visual plausibility, making it challenging to fully satisfy all these aspects simultaneously. This issue is also common in current drag-based image editing approaches (e.g., DiffEditor and DragonDiffusion) and video editing methods (e.g., DragNUWA) and remains a challenging problem to address. We discussed this limitation in the Appendix and gave a detailed analysis.

Thank you for your thoughtful, detailed feedback, and valuable literature recommendations. We appreciate your continued engagement with our work and the opportunity to address your concerns.

Differences and improvements with baselines. Thank you for your thoughtful comments. We acknowledge that, for simple local deformation cases that do not require significant structural changes or the generation of new content—such as shortening the shoe in Fig. 6—both our method and baseline approaches (e.g., APAP and PhysGaussian) perform well. Beyond these simpler cases, Fig. 13 provides additional comparisons on more challenging scenarios that highlight the limitations of the baselines and the strengths of our approach. Besides, we have summarized the main differences between the baseline methods and our proposed method. For further details, please refer to the common concern section.

Differences with Interactive3D. We acknowledge that Interactive3D can also generate new content thanks to the SDS loss. However, a key limitation of Interactive3D is that it heavily relies on manually selecting the 3D region to be explicitly dragged, due to the lack of a strong global-aware generative prior. Let's consider opening a bird's wings (see Fig. 1 or Fig. 13), where the wings are closely tucked against its body, with most 3D Gaussian points shared between these regions. Using Interactive3D to spread the wings requires precisely selecting the Gaussian points corresponding to the wing area and separating them from the body, which is extremely challenging and is prone to artifacts. In contrast, our approach, with a stronger generative prior offered by the multi-view diffusion model, can automatically discover where to split the wings from the body. We will include the comparison with Interactive3D and more discussions in the final version. We will also ensure a more precise description of this comparison in the paper.

Failure cases. First, we clarify that the failure case shown in Fig. 12 of the revised paper is not due to the SDS loss but primarily originates from incorrect multi-view editing results. To provide a detailed analysis, we included additional 2D editing methods and their corresponding 3D reconstruction results (see Fig. 12). All results were obtained using their official codes or Gradio demos.

In the last row of Fig. 12, we compare the 2D editing results of RegionDrag, SDE-Drag, DiffEditor, and our method. Both DiffEditor and our approach fail to precisely drag the suit to the target position and modify the tie’s style. SDE-Drag alters the tie’s style but introduces incorrect textures. RegionDrag requires careful selection of editing regions to avoid style changes, but distorts the overall shape and appearance. Due to these limitations in 2D editing, the corresponding reconstruction results for these methods also fail.

We believe that all methods fail due to the inherent challenge of balancing three interconnected aspects: dragging accuracy (e.g., closing the suit exactly to the tie), identity preservation (e.g., retaining the tie’s style), and global visual plausibility (e.g. avoiding distortion). Satisfying all these criteria simultaneously remains a significant hurdle.

We appreciate the suggestion to explore local editing techniques, such as those used in RegionDrag and GaussianEditor, as potential solutions. GaussianEditor allows masking Gaussian points to update only the areas requiring edits. However, this approach differs from the task we are addressing, as it primarily focuses on adding or deleting objects and altering their styles, while our work focuses on realistic deformations, structural changes, and new content generation.

Finally, we believe that region-based editing and point-based editing represent two types of techniques or tools, despite some overlap in the effects they can achieve (such as local deformation or region transformation). Adapting region-based techniques for creative 3D editing is an exciting avenue for future exploration.

Technical contribution. Thank you for your comment and we acknowledge that the proposed method in this paper is straightforward. However, our primary goal is to tackle a challenging and under-explored task: drag-based creative 3D editing involving significant structural changes or new content generation. Existing approaches, whether based on explicit spatial transformations or implicit latent optimization within limited-capacity 3D generative models, struggle to make substantial topology changes or generate new content across diverse object categories. For instance, APAP is unable to open the bird’s wing or the lion’s mouth (Fig. 1), PhysGaussian fails to generate complementary new content (Figs. 1 and 6), and DiffEditor struggles to achieve view-consistent 3D edits (Figs. 1 and 6).

To tackle this, we reformulated drag-based 3D editing as a multi-view editing and reconstruction problem. While we draw inspiration from image editing techniques, our approach is both innovative (as noted by Reviewers Dkc5 and bkUz) and effective (validated by all reviewers) for the 3D dragging task. Furthermore, we have proposed practical solutions to each sub-problem inherent in this framework, enabling our method to achieve state-of-the-art results.

We would like to emphasize that while the techniques we adopted may not be groundbreaking individually, each design was thoughtfully tailored to address a specific challenge. The effectiveness of our overall framework and the results it has achieved highlight the significance of our contributions.

Certain artifacts of the final output. Yes, the 3D GS generated by our method still has some limitations in detail quality. This issue arises mainly due to three factors: (i) By rendering only 4 orthogonal views, some details (shape and texture) of the 3D object may be lost; (ii) The resolution of the multi-view diffusion model used (MVDream) is currently limited to 256×256. Consequently, even if high-quality images are rendered from the 3D objects, they must be downsampled to fit this resolution, leading to inevitable detail loss; and (iii) The reconstruction model (LGM) may introduce additional losses in both geometry and texture details.

Fortunately, our method is highly flexible and can incorporate advancements in 3D foundational models to address the artifacts. For example, Instant3D, with its 512×512 multi-view diffusion model, or Lara and GRM, which improve geometry reconstruction, can seamlessly replace MVDream and LGM in our pipeline. Moving forward, we plan to incorporate Lara and Instant3D into our method, which will offer stronger reconstruction priors and generation priors respectively.

Finally, we would like to emphasize that the key strength of our method lies in its ability to handle significant structural changes and generate new content during drag-based editing. For example, our approach enables edits such as a bird opening its wings (Fig. 1), an animal opening its mouth (Figs. 1 and 2), or lifting its leg in a more articulated manner (Figs. 1 and 6), and a flower in bloom (Figs. 6 and 9). These types of edits are not effectively supported by existing methods.

Final output when editing a mesh. The final output of our method is a 3D GS representation for 3D meshes. If a mesh model is still required, it needs to be extracted from the 3D GS, which may result in some loss of detail. However, our primary focus is on handling significant structural changes and generating new content during drag-based editing. As noted in the comment, 2D GS could be a potential solution for improved mesh extraction. Additionally, we found that the open-source work Lara (ECCV 2024) could leverage four views to feed-forwardly regress a 2D GS model with a smoother surface, as illustrated in Fig. 11 of the Appendix. In the future, we plan to release a Lara-based version of our method.

Effect of deformation MLP and multi-view SDS loss optimization. Thank you for your thoughtful comment. In fact, we have observed that LGM is capable of reconstructing relatively good results. While there is some degree of texture loss and geometric misalignment, there are no significant geometric inconsistencies or noticeable texture artifacts. The worst case we have encountered was the crocodile example in Fig. 2, where the tail is misaligned. Here, our solution can correct this effectively. Additionally, we have provided a revised video demo in the supplementary material, showcasing additional results for the ablation study of the two optimization stages. This demonstrates the individual contributions of the deformation MLP and the multi-view SDS loss optimization.