ReGS: Reference-based Controllable Scene Stylization with Gaussian Splatting

Thank you for your constructive feedback and thoughtful comments! Please note our top-level comment and additional experimental results in the rebuttal PDF. Below we address your questions and concerns.

---

wrt to novelty. The core contribution of this paper is to enable stylizing 3DGS using an aligned reference image, which can benefit many real applications given its real-time rendering speed. For the first time, we illustrate that the bottleneck of stylization lies in the nature of 3D Gaussians rather than stylization techniques. Therefore, we made important contributions on this new 3D representation by (1) identifying the entangled appearance and geometry of 3DGS is the primary bottleneck, and (2) proposing a set of novel techniques to address the entangelement, resulting in (3) a complete framework, for the first time, enables real-time stylized view synthesis. These novel designs are crucial for obtaining high-quality renderings. We agree that we follow the style transfer techniques from Ref-NPR but they are perpendicular to our core contributions. We will make our contributions more clear in the revised paper.

---

wrt stylizing geometry. Following existing works [8, 9, 3, 10, 6], this paper focuses on stylizing the appearance, which remains an open problem for 3DGS. Our work aims to enable precise texture editing based on a content aligned reference. We agree that further stylizing the geometry will be very interesting, but would like to also mention that such task is significantly more challenging given current problem setup, i.e. using one reference image, due to the single-view shape ambiguity. Existing methods achieving high-fidelity geometry stylization often require a very different set of techniques such as shape prior$^1$, text guidance$^2$ and/or generative modeling$^{3,4,5}$ that is beyond the scope of this work. Combining our method with these techniques to enable both geometry and appearance stylization for 3DGS is an open and interesting future direction.

---

wrt video results on more complex scenes. Besides synthetic scenes, the demo video also contains two forward-facing scenes (1:20 and 1:33). We will include more complex scenes like Fern and Truck in the updated video. Unfortunately, according to the rebuttal instruction and further confirmed by the AC, we found that it is not possible to upload additional videos to openreview during the rebuttal phase. Instead, in Figure 6 in the rebuttal PDF, we render images from multiple viewpoints of requested scenes as a workaround. Our method can achieve consistent 3D stylization for these more complex scenes.

---

Reference:

1. Bao, Chong, et al. "Sine: Semantic-driven image-based nerf editing with prior-guided editing field."CVPR. 2023.
2. Wang, Can, et al. "Nerf-art: Text-driven neural radiance fields stylization." IEEE Transactions on Visualization and Computer Graphics (2023).
3. Haque, Ayaan, et al. "Instruct-nerf2nerf: Editing 3d scenes with instructions." ICCV. 2023.
4. Chen, Yiwen, et al. "Gaussianeditor: Swift and controllable 3d editing with gaussian splatting." CVPR. 2024.
5. Wang, Junjie, et al. "Gaussianeditor: Editing 3d gaussians delicately with text instructions." CVPR. 2024.

Thank you for your valuable comments and constructive feedback! Please note our top-level comment and additional experimental results in the rebuttal PDF. Below we address your questions and concerns.

---

wrt presentation of the materials. Thanks for your constructive suggestion. We have careful revised the manuscript and will keep revising it to improve the presentation. We have included the suggested pseudo code of our densification algorithm using Algorithm2E features in Algorithms 1&2 in the rebuttal PDF. Compared to the original algorithm: ReGS relies on 1. color gradient instead of positional gradient as the densification guidance, and 2. uses structured densification, i.e., replacing a parent Gaussian with a structured set of tiny Gaussians instead of two ''medium-sized'' ones in original 3DGS. These designs are crucial for addressing texture underfitting and verified by the experiments (see Sec. 4.2 and Appendix C, and Figure 4 in the rebuttal PDF).

---

wrt justification of the adopted loss functions. We would like to first clarify that only TCM and color matching loss are adopted from Ref-NPR. Depth regularization and pseudo-view loss are proposed in this work and have already been studied and ablated in Figure 4 (b)&(c) in the paper. While TCM and color matching loss are not our main contributions, we agree that re-assess them is necessary and helpful. We report their ablation results in Figure 1 of the rebuttal PDF. As shown, using color matching loss reduces color mismatch and using TCM loss removes artifacts in the occluded areas. These findings are similar to the observations in Ref-NPR. These results suggest that the adopted losses indeed work for our 3DGS-based model. We will include this experiment in the revised paper.

---

wrt quantitative comparison table of each scene. Thanks for your suggestion. Following Table B.1 of Ref-NPR, we report Ref-LPIPS score of each scene in the table below. Our method consistently outperforms baselines. We will include this detailed table in the revised manuscript.

---

wrt results for different style images acting on a single scene. Thanks for your suggestion. We have included results of multiple style references acting on a single baked scene in Figure 2 in the rebuttal PDF. We temporarily omit the references and original content views here due to limited space. We will include more visual results in the revised manuscript.

---

wrt balancing parameters of equation (8). We perform a simple grid search for these hyperparameters on Blender scenes, and empirically found these values generalize well on other scenes and across difference style references. We will include these details in the revised paper.

---

wrt number of Gaussians changed during stylization. Our method maintains the original opacity-based pruning strategy during training, therefore do prunes Gaussians with low opacity. By design, new Gaussians are adaptively created in the texture underfitting areas. The number fully depends on the complexity of the reference texture. Therefore, the resulting scene is not necessarily heavier than the original. If the overall reference texture is simpler than the original scene, there will be less Gaussians compared to the original scene. For example, the wood-drums case shown in the Figure 4 in the paper has less Gaussians (0.29M) after stylization than the original scene (0.41M). If the reference texture is more complex, then the resulting scene can be heavier than the original scene. For example, the mic scene in Figure 1 in the paper has 0.25M more Gaussians after stylization. The quantitative scores will increase as new Gaussians at underfitting areas are created to fill the missing texture, but will remain the static after convergence (i.e. no texture underfitting remains). We will include this discussion in the revised paper.

Thank you for your constructive feedback and thoughtful comments! Please note our top-level comment and additional experimental results in the rebuttal PDF. Below we address your questions and concerns.

---

wrt inspired by Ref-NPR. Thanks for pointing this out. We will include a more detailed description of Ref-NPR in the updated Appendix to help readers understand the motivation behind Sec. 3.4.

---

wrt superscript $(x,y)$ in Eq. 5. $(x, y)$ denotes the $xy$ coordinates on a feature map. Therefore, Eq. 5 refers to find the location $(x^{\*}, y^{\*})$ on $F_{I_{R}}$ that has closest feature-wise distance to the given location $(x,y)$ on $F_{I_{i}}$. $(x^{\*}, y^{\*})$ is then used to reconstruct a guidance feature $F_{G_{i}}$ for TCM (Line 218). We will clarify this in the revised manuscript.

---

wrt minor indentation issues. Thanks for your careful reading. We will fix these issues in the revised paper.

---

wrt ablation on TCM loss. Thanks for your suggestions. As stated in Line 214, TCM loss is used to spread stylized appearance to the occluded areas, so that the entire scene is stylized. We have included an ablation on TCM in Figure 1 (b) in the rebuttal PDF. Without TCM, the model cannot properly stylize the unseen areas and produces artifacts. Applications like appearance editing (Appendix D), that do not require stylizing entire scene, do not need TCM loss. We will include this ablation in the revised paper.

---

wrt multi-view texture references. Yes, our method can be naturally extended to take multi-view texture references. We leave such exploration for future work.

Thank you for your detailed review and constructive suggestions! Please note our top-level comment and additional experimental results in the rebuttal PDF. Below we address your questions and concerns.

---

wrt geometric stylization. Thanks for your suggestion. We agree that also editting the geometry will be very interesting. We believe our method do able to handle minor shape changes, for example, by relaxing the depth supervision. However, we would like to also mention that precise geometry editting based on a reference image is inherently more challenging due to single-view shape ambiguity. To achieve high-quality geometry stylization, existing methods often adopt a very different set of techniques such as shape prior$^1$, text guidance$^2$ and/or generative modeling$^{3,4,5}$ to hallucinate missing geometry, which are beyond the scope of this work. Combining our method with these techniques for joint geometry and appearance editing is an open and interesting future direction. We will include more discussions in the revised paper.

---

wrt the selection of the number 9 in densification. Thanks for your suggestion. We actually conducted a similar experiment when developing this method, and we found the current splitting strategy provides the best overall performance. Here we conducted the suggested ablation on the number of Gaussians and present the results in Figure 3 in the rebuttal PDF. We plot the PSNR value between the style reference and the corresponding stylized view to quantitively show the texture fitting capability using Blender scenes. As shown, when the number is small, the model fails to capture the target texture details. As this number grows, the performance becomes saturated. When the number equals 9, the model can achieve peak performance but also without inducing many excessive Gaussians that might slow down rendering. We will include this experiment in the revised manuscript.

---

wrt the Default setting for Appendix C. The Default setting is based on the color-gradient-guided density control (same as Ours) and therefore we can fairly ablate structured densification. In the experiment, we showcase the effectiveness of the proposed structured densification by studying how the performance will be affected when we remove such design (i.e. by switching to the default strategy) from the full model. We will clarify this in the revised paper.

---

wrt ablation on texture guidance. Thanks for your suggestion. Here we conduct the suggested ablation study on texture guidance, where we construct the baseline by removing texture guidance from the full model (i.e., switching to the default positional-gradient guidance). We report the comparison results in Figure 4 in the rebuttal PDF. As shown, without texture guidance, the model fails to capture tiny texture details in the reference. We will include this experiment in the revised paper.

---

wrt the importance between Texture Guidance and Structured Densification. From the Appendix C and Figure 4 in the rebuttal PDF, one can see that both strategies are equally important. Removing either of them will reduce stylization quality.

---

wrt 3DGS version of baseline SNeRF. Thank you for the suggestion. We will reproduce an 3DGS version of SNeRF and use it as an additional baseline the revised paper.

---

wrt related work. Thanks for your suggestion. We will revise the related work section accordingly.

---

wrt other gradient hints. Yes, we have tried to use other gradient hints but none of them achieves similar level of details and fidelity as color gradient. This is because other variables (e.g. scales, rotations, and opacity) are not directly/strongly related to final appearance and thus are less sensitive to texture underfitting, especially in the high-frequency areas.

---

wrt supports for other tasks. Our method can potentially benefit scene reconstruction and other downstream tasks, especially when the scene texture is complex. For example, one can simply use our method as an additional training stage to further refine texture details that are missing in the initial reconstruction. Such exploration might be an interesting direction for future work.

---

Reference:

1. Bao, Chong, et al. "Sine: Semantic-driven image-based nerf editing with prior-guided editing field."CVPR. 2023.
2. Wang, Can, et al. "Nerf-art: Text-driven neural radiance fields stylization." IEEE Transactions on Visualization and Computer Graphics (2023).
3. Haque, Ayaan, et al. "Instruct-nerf2nerf: Editing 3d scenes with instructions." ICCV. 2023.
4. Chen, Yiwen, et al. "Gaussianeditor: Swift and controllable 3d editing with gaussian splatting." CVPR. 2024.
5. Wang, Junjie, et al. "Gaussianeditor: Editing 3d gaussians delicately with text instructions." CVPR. 2024.

Thank you for your detailed review and constructive comments! Please note our top-level comment with additional experimental results in the rebuttal PDF. Below we address your questions and concerns.

---

wrt novelty on style transfer techniques. The core contribution of this paper is to enable precise stylization of 3DGS using an aligned reference image, which can benefit many real applications given its real-time rendering speed. For the first time, we illustrate that the bottleneck of such stylization lies in the nature of 3D Gaussians rather than the well-established style transfer techniques. Therefore, we made important contributions on improving this new 3D representation by (1) identifying the entangled geometry and appearance is the primiary bottleneck for stylization, and proposing (2) a set of novel techniques (Sec. 3.2&3.3) to tackle this entanglement, resulting in (3) a complete method, for the first time, that enables real-time stylized view sythesis. These novel designs are verfied by extensive experiments (Sec. 4.2&4.3). Style transfer techniques from Ref-NPR are perpendicular to our core contributions. But we agree that these techniques might not be perfect for 3DGS and further improving them is an interesting future direction.

---

wrt comparison with ARF and SNeRF on aesthetic quality. Thanks for your advice. We simply follow Ref-NPR to include ARF and SNeRF as additional baselines. We do aware such distinction and agree that evaluating aesthetic quality using the original style image for them is indeed a fairer comparison. In Figure 5 in the rebuttal PDF, we report the suggested qualitative comparison with SNeRF. The original style images are acquired from Ref-NPR authors. One can see that SNeRF produces results mimicking the abstract style of the original art, whereas our method follows the extract stylized texture in the reference image by design. We will include more visual comparisons in the revised manuscript.