MiraGe: Editable 2D Images using Gaussian Splatting

1. Lack of understanding of 3D structure: This paper elevates 2D image editing to 3D space, but it overlooks the importance of capturing the underlying 3D structure in the 2D image. As a result, the proposed method is limited in its ability to handle editing tasks that involve invisible parts or complex transformations. For example, while the method can achieve editing by dragging existing parts, it fails to handle editing tasks that require manipulating hidden parts, such as the hidden leg of a cat (as shown in Figure 2). Additionally, most of the manipulation effects are limited to affine transformations or local rigid editing, which can also be accomplished using standard 2D editing tools like Photoshop. Without a robust representation of the 3D structure, it is difficult to guarantee that the complex editing capabilities offered by the proposed method in 3D space will always translate to reasonable results when reflected in the 2D image.

2. The authors should provide a more detailed explanation of why the mirror camera is necessary for the proposed method and how it contributes to the overall performance of the method. Additionally, the comparison between the second row (Amorphous + mirror camera) and the fourth row (Graphic + mirror camera) in Figure 6 suggests that they achieve almost the same effects, which does not clearly illustrate the necessity of Graphite.

3. I think the paper is hard to follow for me, which take me a while. I suggest the authors to improve the readability of the paper further. The paper layout and formatting is a little messy, and the writing style is not very clear. The authors should consider revising the paper to make it more accessible to readers. And the experimental results are not very convincing and has insufficient experimental length. Concretely, this paper lack the visual comparison with baseline methods and 2D editing methods.

First of all, although not significant, the paper has been submitted in the camera-ready format rather than the under-review format. Therefore, I apologize for not being able to pinpoint the exact locations of each word and sentence.

It would be highly beneficial for the paper to provide more concrete reasons for why this low-level “manual editing” is an important issue, as manually editing Gaussians is quite trivial. Furthermore, there is an inconsistency regarding this matter. In the introduction, it states that GaussianImage lacks the capability for manual editing, but in the related works section, it contradicts this by claiming that Gaussians enable precise and flexible editing. More detailed explanations of the limitations of GaussianImage in manual editing, as well as a clearer highlighting of the strengths of this paper, would better clarify its intentions.

Additionally, more thorough and extensive experiments could be conducted, as Table 1 alone is insufficient to demonstrate how much each component affects performance. Specifically, an ablation study on GaMes parameterization and mirror loss concerning representation performance would clearly demonstrate the role of each component. A quantitative comparison between Amorphous, 2D, and Graphite on various metrics would help readers understand the benefits of each method more clearly. Moreover, quantitative and qualitative results on other manual editing methods would further enhance the paper.

(Minor) Additionally, there are some minor misused or misleading words and phrases that could be easily corrected without disrupting the main storyline. For example, “natural networks” in the Related Works section seems to refer to neural networks. The phrase “Training the Gaussian component is challenging” in Section 3 could be more clearly articulated. A clearer version might state like "the optimal number of Gaussians required to represent a given input (image) is not known a priori, and it is non-trivial to adjust the number of Gaussians.

*I did not see training/inference time and memory in the quantitative evaluation table

*The editing capability does not have a comparative evaluation. Because this is a claimed contribution, I think the authors should find baselines and demonstrate why the proposed approach is superior for interactive image editing. Some of the following references could be considered in the evaluation:

[1] Pan, Xingang, et al. "Drag your gan: Interactive point-based manipulation on the generative image manifold." ACM SIGGRAPH 2023 Conference Proceedings. 2023. [2] Shi, Yujun, et al. "Dragdiffusion: Harnessing diffusion models for interactive point-based image editing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [3] Jacobson, Alec, et al. "Bounded biharmonic weights for real-time deformation." ACM Trans. Graph. 30.4 (2011): 78. [4] Wang, Yu, et al. "Linear subspace design for real-time shape deformation." ACM Transactions on Graphics (TOG) 34.4 (2015): 1-11.

*Visualizations of the mesh are not shown when editing, and how the mesh edit propagates to the Gaussian splat. As an ablation, it would help to show why we can’t live with the edited (textured) mesh and need to map the edits to the Gaussian Splat.

*Evaluation of reconstruction quality is missing comparative qualitative examples. Because the PSNRs/SSIMs are generally quite high, it may be tough to appreciate these improvements qualitatively.

• Manipulating Gaussians directly to perform edits seems quite tedious and even challenging for users (compared to instruction based editing in InstructPix2Pix for example)
• While the method is promising, the edits performed have small delta with the original as it's mostly with scaling and shifting Gaussians. This limitation is a natural con of lacking a generative prior.
• The time taken to train the Gaussians on a single image is rather long 450~950 seconds.