Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting

We thank you for the positive feedback and constructive suggestions. Our response to the your concerns are incorporated below:

Q1: Training time of our method.

This is a great question regarding the scalability of our Spec-Gaussian model. The training time of Spec-Gaussian does not significantly increase compared to the baselines. We present the comparison on NeRF-Synthetic dataset in the table below:

Q2: Spec-Gaussian may not perform well in texture-less scenes.

It is worth noting that our approach is generic to be incorporated into 3DGS and Scaffold-GS: both the Ours and Ours-light versions are based on 3D-GS; and Ours-w/ anchor is based on Scaffold-GS. The anchor-based design is only used for efficiency improvements to balance the additional overhead introduced by ASG. As shown in Tab. 4, Ours-w/ anchor still outperforms Scaffold-GS by a large margin, demonstrating its superior modeling ability over Scaffold-GS. Also, we didn’t observe significant degradation in texture-less areas by using anchors in NSVF dataset.

Q3: Use RGB color as the diffuse color.

Great thought. We also experimented with using RGB color as the diffuse color to reduce the computational and storage overhead introduced by spherical harmonics (SH). Experiments on the NeRF-Synthetic dataset showed that using RGB color slightly decreases the rendering metrics. Considering that Spec-Gaussian aims to explore the upper limits of 3D-GS rendering quality, we chose to filter the diffuse color with third-order SH. We will include a discussion on diffuse color modeling in the paper.

Dear Reviewer,

I wanted to gently remind you to please review the rebuttal provided by the authors. Your feedback is invaluable to the decision-making process, and if you feel that the rebuttal addresses any of your concerns, please consider updating your score accordingly.

Thank you for your continued dedication to ensuring a fair and thorough review process!

Best, Your AC

We thank the reviewer for the positive and detailed review as well as the suggestions for improvement. We will revise the mathematical notations in the paper based on these insightful suggestions. Our response to the reviewer’s comments is below:

Q1: Color Separation.

Great question. It's worth noting that in Fig. 6, we've already explored the model without separation. This is because Scaffold-GS inherently uses an MLP to represent color without distinguishing between diffuse and specular components. We conducted further experiments on the teapot scene in Fig. 6, as shown in the table below:

It can be observed that color separation significantly improves the rendering quality of scenes with specular highlights, while using ASG to model diffuse color results in a decline in rendering quality. This experimental result reveals that clean separation of diffuse and specular components can aid in the learning of each part, thereby enhancing the overall color quality. Compared to ASG, SH serves as a better low-frequency filter. The inherent limitations in SH's fitting capability actually contribute to a cleaner diffuse component, allowing ASG to focus more effectively on learning the specular part.

Q2: Related work.

We will add and discuss them in the related work. GaussianShader cannot effectively address scenes with specular highlights; it primarily attempts to enhance the capability of 3D-GS in modeling reflective scenes.

We are sorry for missing Spec-NeRF. Although we both aim to address specular highlights, the approaches to modeling specular highlights differ from each other: Spec-NeRF uses Gaussian directional encoding, while we employ anisotropic spherical Gaussian (ASG).

Q3: Coarse-to-fine training.

Thank you very much for the reminder. We have taken note of the outstanding work in EAGLES, and we will cite this paper in Spec-Gaussian. It is important to note that, unlike EAGLES, our coarse-to-fine approach includes two components: 1) L1-normed gradients for GS densification, and 2) progressively training from low to high resolution. This design aims to prevent GS from becoming overly densified in the early stages of training (due to the L1 norm), which significantly reduces the number of GS.

Q4: Explanation of other questions.

Line 144: Is $\xi$ really $\mathbb R^2$ or is this a typo of $\mathbb R$?

• $\xi$ is $\mathbb R^2$, which means it is a 2-dimensional real vector.

Lines 185: Could you specifically state what is decoupled through $\Psi$.

• Thank you for raising this question. We now believe that decode better conveys the meaning of $\Psi$ compared to decouple. Specifically, it involves decoding the ASG-encoded features to obtain the specular color.

Line 191: There are no equations for pure ASG nor pure MLP. Could you state the equations or clarify how they work?

• The ablation of pure ASG and pure MLP aims to demonstrate that both ASG and decode MLP are crucial. In the case of pure ASG, we directly employ ASG to obtain the color through $c_s = \bigoplus_{i=1}^{N} ASG(\omega_r \: | \: [\mathbf{x}, \mathbf{y}, \mathbf{z}], [\lambda_i, \mu_i], \xi_i), \text{where}\ \xi_i \in \mathbb R^3$. While for the pure MLP, we need to input features that have not been encoded by ASG to ensure a fair comparison. Therefore, the formula is shown below: $\Psi (\kappa, \gamma(\mathbf{d}), \langle n, -\mathbf{d} \rangle) \rightarrow c_s, \kappa = \bigoplus_{i=1}^{N} [\lambda_i, \mu_i, \xi_i].$

Lines 232-234: This seems contradictory to lines 178-179. Could you clarify which one was actually used during the experiments.

• As previously mentioned, our method has three versions. Ours and Ours-light are based on 3D-GS, while Ours-w/ anchor is based on Scaffold-GS. For the versions based on 3D-GS, we used the approach described in Lines 178-179, and for the version based on Scaffold-GS, we adopted the color model described in Lines 232-234.

Lines 242-244 and Tabs. 3 and 4: Based on the explanation, Ours-light refers to 3DGS + ASG, while Ours-w/ anchor refers to Scaffold-GS + ASG. However, it is unclear what the performance version (Ours) refers to.

• Thank you for pointing out this issue. Clarifying this in the experimental section is very important. We will include the previous explanation in the paper.

Fig 6: Lines 181-183 and 277 state that directly using ASG leads to an inability to represent specular and anisotropic components. The meaning of “MLP” in “Ours w/o MLP” is ambiguous.

• As mentioned in Lines 181-183, directly using ASG can result in failure to model specular color. Therefore, in the ablation study of Fig. 6, we included Ours-w/o MLP to support this statement. Here, MLP refers only to the decoding MLP $\Psi$ in Eq. (10). This is an excellent suggestion, and we will include a more detailed explanation of this in the paper.

Finally, we would like to thank the reviewer once again. Many of these points were things we had not noticed before, and these suggestions will significantly improve the readability of the paper.

Thank you for the prompt response and your suggestions.

As mentioned in the global response, our method has three variants: the 3D-GS-based Ours and Ours-light, and the Scaffold-GS-based Ours-w/ anchor.

In Section 3,

• Sec 3.1 is the Preliminaries, where 3D Gaussian splatting and Anchor-based Gaussian splatting are introduced as explanations of 3D-GS and Scaffold-GS, respectively. The Anisotropic Spherical Gaussian is introduced to explain the ASG formula, which will be applied to each of our variants to encode specular features.
• Sec 3.2 is the modeling of view-dependent appearance for Ours and Ours-light.
• Sec 3.4 is the modeling of view-dependent appearance for Ours-w/ anchor, which is slightly different from Sec 3.2.
• Sec 3.3 is a general part used for all three variants of our method, aimed at removing floaters in real-world scenes.

We hope our explanation can resolve your confusion about Section 3. And we fully agree that clarifying these details will strengthen the readability of the paper. We will include more detailed explanations for Section 3 in the paper.

We thank the reviewer for the positive review as well as the insightful suggestions for improvement. Our response to the reviewer’s comments is below:

Q1: Ablation on the Number of ASGs.

That's an excellent question. During the implementation of our code, we explored the number of ASGs. In the table below, we present the ablation results on the teapot scene. ASG=32 achieves the highest overall rendering metrics without causing a significant increase in training time or a decrease in FPS. Further increasing the numbers will not result in increase of rendering qualities but decreasing the rendering speed.

Q2: Comparison with inverse rendering GS.

Although theoretically, it is reasonable to incorporate inverse rendering to improve rendering quality, this suffers difficulties and may cause burdens in practice. This is because decoupling and learning the required information for physically based rendering (PBR) from multiview images is a highly ill-posed problem. The errors will, in turn, have negative impacts on rendering. The table below compares our method with Relightable-GS on the NeRF-Synthetic dataset, and Spec-Gaussian outperforms Relightable-GS by a large margin.

Q3: Missing comparison and citation about works on reflection.

Thanks for mentioning this. Although reflection and specular highlights (ours) focus on two different properties of objects and environments for causing view-dependent appearances as clarified previously, we will cite these works and discuss their differences in the related work. Although our work is not directly related to addressing reflection, we have still provided some experimental results for comparison. The qualitative experimental results can be seen in the submitted rebuttal PDF. The table below shows the comparison on real-world scenes from ref-nerf.

We also provided a comparison between Spec-Gaussian and GaussianShader on the Ref-NeRF Synthetic scenes (Shiny-Blender):

We would like to emphasize once again that specular highlights and reflections are two distinct shading effects, each requiring different technical approaches to address. Improvement in one shading effect does not necessarily translate to improvement in the other. Generally speaking, enhancing the modeling of specular highlights can also improve the modeling capability for general scenes. However, improving reflections mainly enhances the rendering quality of the reflective parts, and it may lead to negative optimization for the non-reflective parts, like NeRO and GS-DR.

Q4: Missing references.

Thank you for pointing out these awesome works. We will add these citations to our paper.

Q5: About the insights of the L1 norm of gradients.

During the optimization process, 3D-GS accumulates the gradient of each pixel ( $\frac{d L}{d \mathbf{x}}=\sum \frac{d L}{d \mathbf p_i} \frac{d \mathbf p_i}{d \mathbf{x}}$) for every GS. When the accumulated value exceeds a threshold $\tau_g$, the GS will densify. It is important to note that this value is not the gradient of each GS position but rather the accumulated gradient sum used for densification. Our insight is that gradients can be both positive and negative, and summing them for accumulation is not reasonable because large negative gradients can decrease the accumulated value, preventing GS that should densify from doing so. While negative gradients are meaningful for position optimization, they are clearly not reasonable for accumulation to determine whether densification should occur, as large negative gradients indicate that the GS requires more refined optimization. Therefore, we decided to apply the L1 norm only to the gradients used for accumulation to determine whether densification should occur ( $\frac{d L}{d \mathbf{x}}=\sum \Vert \frac{d L}{d \mathbf p_i} \frac{d \mathbf p_i}{d \mathbf{x}} \Vert_1$).

Q6: Time breakdown.

We have provided the impact of different components on FPS in the paper. It can be found in Tabs. 5-6.

Dear Reviewer,

I wanted to gently remind you to please review the rebuttal provided by the authors. Your feedback is invaluable to the decision-making process, and if you feel that the rebuttal addresses any of your concerns, please consider updating your score accordingly.

Thank you for your continued dedication to ensuring a fair and thorough review process!

Best, Your AC

We are glad and appreciate that you recognizes that the results of Spec-Gaussian are comprehensive and compelling. Our response to your valuable comments is below:

Q1: What makes Spec-Gaussian work: Evaluation of the different components.

The key components that make Spec-Gaussian work includes: the ASG appearance field and the coarse-to-fine training mechanism. These components are extensively studied both quantitatively (Tables 5-6) in supplementary file and qualitatively in Figures 6-8.

First, the ASG appearance field works by using ASG and decoding MLP to augment Gaussians capabilities. (a) With the ASG appearance field, the performance improves by ~4dB in our anisotropic synthetic dataset. Our extensive ablation study of the ASG component also demonstrates that the improvements are not from the using of MLP like in Scaffold-GS (see Figure 6 and Tables 1-4), but more fundamentally from the introduction of ASG to effectively capture the high-frequency specular colors. (b) Moreover, our empirical experiments also show that using SH or MLP to fit the entire color spectrum independently is not ideal, as color contains both high and low-frequency signals, making it difficult to fit accurately. By filtering diffuse with SH and modeling the remaining specular components with ASG, each part can operate within its fitting capability, thereby improving the overall rendering quality.

Second, the coarse-to-fine strategy is effective in resolving the floaters in real-world scenes. The reason might be that by fitting the model on coarse images, the model is encouraged to capture the low-frequency geometry instead of the high-frequency noisy details, which reduces the chance for overfitting to noisy details that are harmful for generalization causing floaters. We also introduced L1-normed gradients in the accumulation process used for densification, making the GS densification more reasonable.

Q2: Real-world comparison with Scaffold-GS.

In this rebuttal, we have submitted more comparisons incorporating scaffold-GS for comparisons and a zoomed-in version of Fig. 8 in the PDF. We will incorporate more comparisons in the final version.

Q3: About the term Anisotropic.

In this paper, the term anisotropic often appears alongside specular. Take Figure 1's CD as an example—the anisotropic part refers to the CD's surface, which shows different specular colors when viewed from different angles (resulting in the rainbow effect). While SH is indeed an anisotropic spherical function, using low-order SH (e.g., first 3 orders in 3D-GS) struggles to model complex shading effects. In Figure 6, even the first 6 orders of SH falls far short of properly modeling specular scenes (perhaps using more than 100 orders would make a difference, but the computational cost would be far greater than that of ASG).

Dear Reviewer,

I wanted to gently remind you to please review the rebuttal provided by the authors. Your feedback is invaluable to the decision-making process, and if you feel that the rebuttal addresses any of your concerns, please consider updating your score accordingly.

Thank you for your continued dedication to ensuring a fair and thorough review process!

Best, Your AC