DisC-GS: Discontinuity-aware Gaussian Splatting

Q1: Qualitative comparisons and quantitative metrics.

A1: (1) Qualitative comparisons. In the PDF uploaded during rebuttal (at the bottom of the "Author Rebuttal by Authors" comment), besides in the 3D scenes in Figs 3 and 4 in paper, we also provide qualitative comparisons in more 3D scenes. Moreover, Scaffold-GS is also added in the comparisons. As shown, across different 3D scenes, our framework consistently achieves better visual effects than existing methods such as Scaffold-GS, further showing the efficacy of our framework. Due to the space limitation of the PDF, we'll also add qualitative comparisons in more 3D scenes to paper.

(2) Quantitative metrics. Moreover, for the mentioned mismatch between the difference in rendering metrics and the difference in visual effects, this can be because, different testing views are of different complexities (i.e., containing different numbers and scales of discontinuities). During measuring the rendering metrics, all different testing views (including those very complex ones) are utilized and they all contribute to the measurement. Yet, in Figs 3 and 4 in paper which serve as the qualitative results of our method in general, we did not dedicatedly show all those very complex testing views that suffer from very severe boundary modeling issue.

As shown in the PDF uploaded during rebuttal, while good visual effects can be consistently achieved by our framework across different viewpoints and different 3D scenes, especially over those testing views that are more complex (richer in boundaries), the difference in visual effects between our method and existing methods such as Scaffold-GS tends to be more significant. These complex testing views, during quantitative evaluation, also tends to more significantly contribute to the superior performance of our method in metrics such as SSIM and LPIPS. We will also add more qualitative comparisons over those complex testing views across different 3D scenes to paper.

(3) Questions w.r.t. evaluation details. (3.1) Yes. We use the same data as vanilla GS without rerunning COLMAP. (3.2) Following the vanilla GS, during calculating the rendering metrics, for images with width exceeding 1600, we would rescale them for their width to be 1600. For rest images we would use their original resolution. (3.3) Yes, SSIM and LPIPS calculations are both done in the same manner as vanilla GS.

Q2: Synthetic scenes.

A2: As suggested, we also evaluate our method on the synthetic NSVF and NeRF datasets, on an RTX 3090 GPU.

For rendering metrics, below, we report the SSIM, PSNR, and LPIPS metrics. Moreover, we also show images rendered over these two datasets in the PDF uploaded during rebuttal.

As shown above, on synthetic scenes, our framework, when applied on both 2D and 3D Gaussian Splattings, can consistently achieve performance improvements. Moreover, as shown in the PDF, on synthetic scenes, our framework can also render images with high quality. This further shows our framework's efficacy.

Q3: Discussion on the number of Gaussians, training time and storage.

A3: Below, we show the number of Gaussians, storage, training time, and inference time of our framework, on the Tanks&Temples dataset on an RTX 3090 GPU.

As shown, though our framework achieves obviously better performance, it uses less number of Gaussians, does not increase the memory storage, only brings a relatively small increase of training time, and can perform rendering efficiently during inference. We'll discuss the above in paper.

Q4: Optimization of DisC-GS.

A4: (1) Computations. No, we do not use special optimization strategies. Instead, we would like to highlight that computations introduced by DisC-GS are not mathematically complex. Specifically, in the forward rendering process, as mentioned in lines 261-265 in paper, for each Bézier curve, the binary indicator function that DisC-GS builds for it is with $O(1)$ time complexity. Meanwhile, in the gradient backpropagation process, the cubic functions that DisC-GS introduces (Eq. 11 in paper) also have closed-form solutions, and can thus be very efficiently solved also in $O(1)$ time complexity. Thus, overall, as shown above in A3, our framework would only bring a small increase of training time. Meanwhile, during inference, our framework can also perform rendering efficiently.

(2) Local optima. As mentioned above, the computations our framework introduce are not complex. It thus would not add significant complexity to the typical Gaussian Splatting technique. Specifically, across different 3D scenes among different datasets, our framework consistently outperforms both 2D and 3D Gaussian Splattings leveraging the typical Gaussian Splatting technique. This also implies that our framework would not easily get stuck in local optima. Meanwhile, in the PDF uploaded during rebuttal, we also provide loss curves over the 3D scene Train in the Tanks&Temples dataset. As shown, no matter when applied to 2D or 3D Gaussian Splattings, our framework consistently achieves a similar loss reduction trend. This further implies that our framework would not raise the risk of easily getting stuck in local optima.

Q1: Evaluation of the storage.

A1: Below, we show the storage of our framework on the Tanks&Temples dataset on an RTX 3090 GPU. As shown, either applied on 2D or 3D Gaussian Splattings, our framework does not increase the storage size.

This can be because, admittedly, for each Gaussian representing the 3D scene, our framework equips it with a fifth attribute to represent the Bézier curves and thus makes it have slightly more parameters. Yet, on the other hand, with this additional attribute incorporated, our framework also enables each Gaussian representing the 3D scene to be aware of the discontinuity of shapes and thus enhance its representation ability. As shown below, this allows our framework to ultimately represent the 3D scene with much fewer Gaussians, thus not incurring the increase of memory storage. We'll discuss this more in paper.

Q2: Additional qualitative results.

A2: (1) As suggested, we have rendered additional qualitative results (i.e., videos over different 3D scenes each with continuous camera movements). Among these videos, we observe that DisC-GS can consistently preserve multi-view consistency. Meanwhile, in these videos, artifacts such as flickering are also not observed. This can be because, while DisC-GS produces hard boundaries, the boundaries produced over different viewpoints are all based on the projection of the same sets of control points stored in the Gaussians representing the 3D scene. Thus, (multi-view) consistency of boundaries across different views can be kept. Meanwhile, with boundaries rendered in a multi-view consistent manner, artifacts such as flickering are also not observed. (2) During rebuttal, due to its format and space constraint, we provide a sample of such video qualitative results in GIF format in the PDF at the bottom of the "Author Rebuttal by Authors" comment. To view those results in animation mode, please use a computer and Adobe Acrobat Reader Version 2024.002 (downloadable from Adobe Reader's official website). We'll also include more results in paper.

Q3: Comparison with mip-splatting.

A3: Below, we also compare our method with mip-splatting. As shown, on all three metrics and across various datasets, our framework consistently outperforms mip-splatting. This further shows the efficacy of our framework. We'll include mip-splatting in Tab. 1 in paper.

Q4: More comparisons w.r.t. boundary issues.

A4: In Figs 3 and 4 in paper, we demonstrate that, compared to both 2D and 3D Gaussian Splattings, our method can mitigate aliasing artifacts and achieve better rendering quality, especially in image regions containing numerous boundaries. To more clearly show that such advantages of our framework are due to its ability in handling boundary issues, we also perform quantitative comparisons.

In Tab. 4 in paper (re-shown below), we quantitatively test our framework, particularly over the boundary-rich areas of the testing images. Specifically, for each testing image, to acquire its boundary-rich areas, we first use the Canny algorithm [9] to detect its boundaries. We then define its areas that involve or surround its Canny-detected boundaries as its boundary-rich areas, and define its rest areas as its boundary-sparse areas. To enable evaluation over image sub-areas, following [40], we use the MaskedSSIM metric (the larger the better). As shown, especially in the boundary-rich areas of the testing images, our framework can achieve a significant performance improvement. This shows our framework's ability to handle boundary issues from one perspective.

Meanwhile, in Tab. 6 in paper (re-shown below), to evaluate if our framework can render sharp boundaries accurately instead of rendering them with blurriness, we also test our framework from the image sharpness perspective. Following [15], we measure image sharpness using the energy gradient function. As shown, our framework can increase the sharpness of its rendered images. This also from another perspective implies our framework's ability in accurately rendering the sharp boundaries in the image.

We will further clarify the above in paper.

Q5: Bézier curves in a 2D subspace.

A5: (1) In DisC-GS, we can define the Bézier curves in a 2D subspace, as well as directly in a three-dimensional space (3D space). (2) In the method section, we define the Bézier curves in a 2D subspace since we there take 2D Gaussian Splatting as an example and explain how we apply our framework on 2D Gaussian Splatting (as mentioned in lines 168-169 in paper). (3) We can also similarly define the Bézier curves in a 3D space for 3D Gaussian Splatting. As mentioned in lines 388-392 in paper, to achieve this, we just need to modify a single place of our framework (i.e., defining the control points of the Bézier curves for each Gaussian in the 3D space coordinate system instead). (4) As shown in Tab. 2 in paper, no matter when defining the Bézier curves in the 2D subspace or the 3D space, our framework can consistently achieve performance improvements. We will further clarify this in paper.

Q1: 2D parameters to control the position of the control points.

A1: As mentioned in lines 121-125 in the paper, in existing works, both 2D and 3D Gaussian Splattings have been utilized to represent the 3D scene. In this work, we proposed a framework that can be applied to both 2D and 3D Gaussian Splattings. Specifically, as mentioned in lines 218-220 and lines 388-392 in the paper, when applying our framework on 2D Gaussian Splatting, we use 2D parameters to control the positions of the control points, while when applying our framework on 3D Gaussian Splatting, we instead use 3D parameters. As shown in Tab. 2 in the paper, no matter when applied to 2D or 3D Gaussian Splattings, our framework can consistently achieve performance improvements. Meanwhile, in both cases, we do not observe rendering problems in areas significantly affected by large viewpoint changes. We will further clarify the above in the paper to avoid confusion.

Q2: When additional properties such as control points and curve rendering are introduced, what impact does this have on training efficiency? [...] impact on training time and convergence(number of iterations).

A2: (1) Training time. Below, similarly to the rendering time shown in the Appendix, we also show the training time of our framework, on the Tanks&Temples dataset on an RTX 3090 GPU.

As shown, though our framework achieves obviously better performance, no matter when applied to 2D or 3D Gaussian Splattings, our framework only brings a relatively small increase of training time.

(2) Convergence (Number of iterations). Meanwhile, in the PDF uploaded during rebuttal (at the bottom of the "Author Rebuttal by Authors" comment), we also provide loss curves over the 3D scene Train in the Tanks&Temples dataset. As shown, our framework consistently achieves a similar convergence rate compared to the baseline without parameters such as the control points added. Moreover, for a fair comparison, in the experiments in the paper, we train our framework for the same number of iterations (i.e., 30k iterations) as the existing Gaussian Splatting works.

We will add the above discussion on training time and convergence to the paper.

Q3: Affects on the distribution of Gaussians in edge and surface regions. [...] whether the number of Gaussians in the scene will differ significantly from the original 3DGS

A3: Below, we aim to compare the total number of Gaussians in the whole 3D scene, as well as the number (distribution) of Gaussians in the edge and plane regions of the 3D scene, of our framework with the original 3DGS. Yet, both the edge regions and the plane regions of the 3D scene are not annotated in the dataset. Thus, to enable the latter of the above comparisons, below, we first describe how we estimate the edge regions and the plane regions of each 3D scene.

Specifically, given a 3D scene, to estimate its edge regions, we here conduct the following 4 steps: (E.1) for each testing image of the scene, we first pass the image over the Canny algorithm [9] for detecting the pixels that are on the edge regions of the image. (E.2) After that, each testing image is further passed over the Depth Anything model [a] to acquire the depth values of its edge-region pixels detected in (E.1). (E.3) Then for each edge-region pixel detected in (E.1), leveraging its 2D coordinate, its depth value, and the viewpoint of its corresponding testing image, we can map this pixel back to the 3D space of the 3D scene. (E.4) Finally, we can define the union of the 3D positions of all the edge-region pixels as the estimated edge regions of the given 3D scene.

Meanwhile, to estimate the plane regions of a 3D scene, we here conduct the following 3 steps: (P.1) for each testing image of the scene, we first pass it over the Depth Anything model [a] to acquire its depth map. (P.2) Given the depth map of a testing image, following [b], using Hough Transform, we can detect the pixels that are on the plane regions of the image. (P.3) Then similar to (E.3-E.4) above, we can transform all the plane-region pixels to the space of the 3D scene, and we can then finally define the union of the 3D positions of all the plane-region pixels as the estimated plane regions of the 3D scene.

With the edges and the plane regions of the 3D scenes estimated in the above way, we can then estimate the number of Gaussians that cover (overlap with) these regions and thus perform comparisons over the number of Gaussians in the edge and plane regions of the 3D scene. Below, we compare the total number of Gaussians in the whole 3D scene, as well as the number of Gaussians in the edge and plane regions of the 3D scene, between our framework and the original 3DGS, on the Tanks&Temples dataset.

As shown, compared to the original 3DGS, our framework can represent the 3D scene with much fewer Gaussians, especially over its edge and plane regions. This further demonstrates our framework's ability in performing discontinuity-aware rendering and handling boundary issues. We will also discuss the above more in paper.

[a] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data.

[b] Hough Transform for real-time plane detection in depth images.

Q4: Description of the challenges.

A4: Thanks for your suggestion. We agree that the Gaussian Splatting technique still faces challenges. As suggested, we will provide a detailed description of the mentioned challenges in the paper.

Thank you to the author for providing additional explanation.

So, if we assume the presence of 4M control points, are these points situated within a 3D space and projected onto a 2D image plane? If this is the case, it appears that the impact on 2DGS might be minimal. However, for 3DGS, it seems that the method might inadvertently reduce 3DGS to a quasi-2D state.

Additionally, the MaskedSSIM comparison offers a clearer measure of improvement in the edge areas.

Q1: Defines numerous labels. [...] It would be better to include a list of labels and their explanations. [...] label explanations under each equation.

A1: Thanks for your suggestion. Following it, (1) below, we formulate a list of labels (symbols) and their explanations. (2) Meanwhile, under each equation in our paper, we will fully explain all the labels used in that equation. For example, we will re-explain $\mu$, $r_1$, and $r_2$ under Eq. 4, and re-explain $P$ and $W$ under Eq. 5. We will also include a list of all labels and their explanations in the Appendix of our paper.

Q2: Analysis of the computational complexity of the proposed method.

A2: Below, we show both the training time and the rendering time (inference time) of our proposed method, on the Tanks&Temples dataset on an RTX 3090 GPU.

As shown, though our framework achieves obviously better performance, it only brings a relatively small increase of training time. Besides, during inference, our framework can also achieve a competitive rendering time (speed) compared to the existing methods leveraging the conventional Gaussian Splatting technique and satisfy most real-time requirements. We will extend Tab. 11 in the paper for the table to also include the analysis of training time (besides the analysis of rendering time).

Q3: How the curves affect the densification procedure. For example, will the new cloned/splitted points be assigned the same curve attribute?

A3: Yes. In the densification procedure of our framework, when a point (Gaussian) is cloned/splitted into two new points (Gaussians), similar to how other attributes such as the spherical harmonic coefficients and the opacity are assigned to the new points, we simply assign both the new points with the same curve attribute as the original point. We will discuss this in the paper.

Q4: Is there any penalty to optimize/adjust the positions of four control points in a curve?

A4: Yes. In our framework for which we have maintained its differentiability, similar to the other attributes of the Gaussian, the positions of the four control points stored in the curve attribute are also learnable parameters. This means that, during training, based on the loss function as the penalty, the gradients would first backpropagate from the loss function to the positions of control points stored in the curve attributes. Leveraging such backpropagated gradients, the positions of control points would then be correspondingly optimized/adjusted to facilitate the accurate representation of the 3D scene.

Q5: Will this affect the opacity of the whole point, and whether it will help the point escape pruning.

A5: (1) In Gaussian Splatting, the opacity of the whole point (Gaussian) that would be used during point (Gaussian) pruning is the opacity attribute $\alpha$. In our framework, to perform "scissoring out", we add a binary indicator function $g$ to the color blending function and this does not edit the opacity attribute $\alpha$ (as can be seen in Eq. 6 in the paper). (2) In other words, as long as "opacity $\alpha$ < threshold", the pruning process of a point (Gaussian) would always still occur in our framework, and our framework would not interfere this process. Thus, our "scissoring out" process would not help the point escape pruning.