QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos

Thank you for your thoughtful feedback. Please refer to our shared rebuttal (texts and PDF) for additional discussion and results. We address the specific questions below.

Q: Details and ablation studies regarding the quantization process.

Thank you for the suggestion. We will add a detailed explanation on the quantization process in the revision as preliminary.

• (a) We provide a more detailed explanation on entropy coding (under “Q: No information on how the entropy coding works” for Reviewer dVjq). We answer questions about the quantization approach (under “Q: Experimental settings” for Reviewer jnXV). Please refer to corresponding sections for the details. We will clarify in the revision.

• (b) We provide further explanation and ablations analyzing the accuracy-memory tradeoffs. In the PDF, we show results varying loss coefficients for the quantization and sparsity modules in Fig. R1. We also ablate the effect of varying latent dimension in Tab. R1. We discuss these results in the shared responses (under “Q: Accuracy-memory trade-off and ablation study on quantization setup”).

Q: Highlighting specific technical contributions.

We agree that the ideas behind the individual modules for quantization and sparsity have been used for other applications such as static scene reconstruction and deep network compression. Our contribution is that we propose an approach for modeling dynamic Gaussians as residuals combined with the optimization of their storage size via our quantization-sparsity framework. This has not been explored previously. We will clarify our contribution in the revision.

Q: Code release.

Yes, we will release the code for research purposes upon acceptance.

Q: The ablation results presented in Tab. 2 are confusing and lack crucial analysis.

Thank you for pointing it out. We will address this further in the paper. Please refer to the shared responses for all reviewers (under "Q: Evaluation analysis and confusing Tab. 2").

Thank you for initiating discussions, reviewers and AC. Thank you for the insightful comment. Rendering speed for the Gaussian splatting depends on a wide variety of factors involving total number of Gaussians, their characteristics in terms of position, rotation, scale, and opacity as the rasterization for each tile varies depending on these attributes. Hardware configurations (I/O, OS scheduling, GPU clock fluctuation) can also cause a difference. For a similar number of Gaussians, it is possible for FPS to vary drastically with different attribute distributions which is indeed what we observe for N3DV (Table E-1(b)) based on the different scaling attribute distribution (Figure R2).

Table E-1: Further analysis on rendering speed. We provide more comparison between the baseline and QUEEN (+Attribute Quantization) on (a) one sequence “flame” from the google immersive datasets and (b) the N3DV datasets. All time measures are in the unit of milliseconds. Storage size in MegaBytes (MB).

(a) Sequence “flame” from the google immersive datasets

(b) N3DV datasets

Table E-1 shows a more extensive analysis on rendering efficiency and other performance metrics of the baseline and with attribute quantization, for the sequence “flames” in Immersive as well as on the N3DV dataset. On the sequence “flames” of the immersive datasets, we obtain 10.6% more Gaussians (+23k) with the densification process compared to the baseline. This leads to higher rendering time (or lower FPS) (+8.1%) which is increased further due to the additional decoding time (0.43 ms).

Our attribute quantization of the Gaussians during training affects the distribution of the attributes, leading to different cloning and splitting during densification (based on viewspace gradients and Gaussian scale attribute) compared to the baseline. This also varies with different scenes depending on the scene changes (e.g., motion and appearance/disappearance of objects).

We highlight that the main goal of our paper is to reduce the storage and training time of online free-viewpoint videos (FVVs) while maintaining the reconstruction quality and the efficient real-time rendering speeds brought by Gaussian splatting. Although it is interesting to pursue higher rendering efficiency or controllability, those are not our goal. Our approach does not explicitly control the distribution of the Gaussian attributes or the number of Gaussians, which are the main factors behind rendering speeds. We believe this can be an interesting direction for future work to pursue and we will add discussion in the revision.

Furthermore, our method addresses a joint problem of reconstruction and quantization for online FVV, in contrast to a two-step formulation of first reconstruction and then quantization. In the latter case, it is more likely for the system to pay for additional computational cost and some loss of quality. In our case (the former case), a joint formulation shows more potential to achieve a better tradeoff between quality and efficiency, as shown in Tab. D1 as well as recent methods such as [84]. We will add discussion in the revision to better explain our contributions.

We appreciate the discussion and we will include it in the revision for future audiences. Furthermore, we will release the codebase to the community to experiment with various types of scenes and discover new insights into the effect of quantization on rendering speeds.

Thank you for your thoughtful feedback. Please refer to our shared rebuttal (texts and PDF) for additional discussion and results. We address the specific questions below.

Q: The idea of exploiting temporal redundancy via encoding attribute residuals is not new.

Thank you for confirming the value of our exploration on temporal redundancy with 3DGS. While we agree that the idea of residual training has been explored in some way for NeRFs in [75], it is not trivial to adopt residual modeling on 3DGS due to its explicit nature and lack of overall structure. Furthermore, a key contribution of ours comes from not just modeling dynamic Gaussians as residuals but combining it with the optimization of their storage size via our quantization-sparsity framework (as also mentioned by Reviewer 9x86).

Q: The proposed method appears to be highly sensitive to hyperparameter tuning.

Tab. D3: Sensitivity to hyperparameters.

• (a) To test the sensitivity of reconstruction quality to hyperparameters, we train our method on the N3DV dataset with two sets of hyperparameters. The first configuration uses the original set of hyperparameters while the second utilizes the hyper parameters corresponding to Immersive in Tab. 9 and 10 while also matching the learning rate for the position residuals (0.0005). We show results on N3DV datasets in Tab. D3. We see that the Immersive hyper parameter configuration still achieves similar PSNR as the original set for N3DV. While the model size is higher (1.49 MB) with the Immersive configuration compared to the original configuration (0.60 MB), it is still much lower than the prior state-of-the-art 3DGStream (7.8 MB) while maintaining higher reconstruction quality in terms of PSNR.

• (b) Most of the hyperparameter difference stems from the widely varying amount of scene motion between N3DV and Immersive datasets. The Immersive dataset contains larger and more complex scene motions (e.g., a person entering and leaving the scene) while N3DV contains relatively minor motions. We found that a higher learning rate for the position residuals allows Gaussians to adapt to the highly dynamic scenes.

• (c) The gating hyperparameters in Tab. 9 for N3DV are set to utilize this prior information about the dataset where the stretch hyperparameters $\gamma_0$, $\gamma_1$ are set closer to 0 to enforce more sparsity in the position residuals.

• (d) Additionally, the Immersive dataset itself consists of a wide variety of indoor/outdoor scenes at varying scales/scene motion/illumination. We use the same set of hyperparameters for each scene achieving good reconstruction quality for every scene (Tab. 7) showing its generalization capability.

• (e) Furthermore, we will release the code for the community to experiment.

Thank you for your thoughtful feedback. Please refer to our shared rebuttal (texts and PDF) for additional discussion and results. We address the specific questions below.

Q: 360-degree rendering.

We evaluate on N3DV and Immersive datasets as they are widely-adopted standard benchmarks for various prior works [6,7,8,10,11,12,16] on free-viewpoint videos.

While these two datasets indeed contain forward-facing scenes, there are limited 360-degree datasets in the research community. Most of the 360-degree datasets focus on human characters or single objects (e.g., D-NeRF, Ava-256), which is a different focus from our goal to capture scene-level dynamics. To the best of our knowledge, N3DV and immersive datasets are among the best benchmarks for real-world scene details and dynamics.

Note that our approach makes no assumptions on a forward-facing setup and can directly be applied to 360-degree scenes. In the revision, we will evaluate our method on the CMU Panoptic studio datasets (http://domedb.perception.cs.cmu.edu/), which covers 360-degree and contains some dynamics beyond a single human character.

Q: No evaluation or demonstration of the "streaming" feature.

The streaming feature is inherently built into our method. Due to the per-frame reconstruction and encoding, at each time-step, we only need to transmit the corresponding residuals with no dependence on future frames. Furthermore the high compression rate and high training/rendering efficiency are essential features for streaming.

Q: No information on how does the entropy coding work.

Our entropy coding approach flattens our integer latent matrix for each attribute before encoding. For example, for L-dimensional latent attributes for N gaussians, we flatten the matrix to obtain a vector with L*N elements. This integer vector is then encoded using standard entropy coding approaches such as arithmetic coding [C]. These standard entropy coding approaches have been used in a large number of works [79, 81]. We will clarify and add an explanation in the revision.

[C] Langdon, Glen G. "An introduction to arithmetic coding." IBM Journal of Research and Development 28.2 (1984): 135-149.

Q: Confusing notations.

$\mathbf{l}_i$ is a generic symbol for the quantized integer latent for any of the four categories: rotation, scale, opacity and color. $\mathbf{l}_{p_i}$ is the pre-gated residual for position attributes. $\alpha$ (without subscript, in bold font) is the set for all per-Gaussian alphas ($\alpha_i$). Each $\alpha_i$ corresponds to a $g_i$. We will improve our notation and add clarifications.

Note: some symbols are not rendered correctly in the open review system, so we wrote it as plain code.

Q: More detailed explanation on the challenges of per-frame training.

Unlike offline reconstruction, per-frame training does not have access to future-frame information. This setup limits the capability to effectively reason about large scene changes (topological changes and highly varying appearance) [D]. Suppose an object suddenly appears or disappears, it is more difficult to (de-)allocate and update scene parameters to capture such changes. In the context of Gaussian splatting, it would be tricky to schedule densification and pruning of the Gaussians. We will add a more detailed explanation in the revision.

[D] Richard A. Newcombe, Dense Visual SLAM, PhD Dissertation, 2012

Thank you for your thoughtful feedback. Please refer to our shared rebuttal (texts and PDF) for additional discussion and results. We address the specific questions below.

Q: Experimental settings.

• (a) The number of bits is dependent on the scene content as it relies on the amount of motion. This number can be fractional, on average, which is the standard for any entropy coding algorithm like arithmetic coding/Huffman coding. For example, for the Sear Steak scene in the N3VS datasets, on average, we require 0.68 bits for all the quantized attributes (corresponding to 0.5MB/frame). This depends on the entropy of the latents itself, which varies with each scene (Figure 7).

• (b) For sparse position residuals, we use fp32 as full precision.

• (c) For position residuals, we utilize sparsity as it has better compression ratios than fixed quantization like bfloat16, fp8/int8 and so on while still maintaining reconstruction quality (Table 2). As seen in Figure 5, only ~2% of gates are active, which corresponds to 98% sparsity and ~15x compression factor, (with additional storage costs for index locations) for the position residuals. In contrast bfloat16 would lead to only 2x reduction while still reducing the residual precision which is important for high reconstruction quality.

Q: Evaluation analysis.

Please see the detail discussion in the shared responses for all reviewers.

Q: Perception of quality.

We conduct an extensive A/B user study. For each vote, we show a pair of randomly chosen rendering results (from a test view that is not used in training) by our method and one of the baseline methods (3DStream and TeTriRF). We also show the ground truth video as a reference for the participants to make the decision. We ask the participant to choose the method that more faithfully matches the reference video. In total, we collected 285 responses from 15 participants within the timeline of the rebuttal. Tab. D2 summarizes the preference percentage of our method over the baseline methods. On both the N3DV and Google immersive datasets, the participants strongly prefer our results in comparison to the baseline methods.

Tab. D2: User preference on visual results.

Q: Performance evaluation on a more reasonable client device.

For fair comparison, we use the same A100 GPU for a consistent benchmark for measuring our rendering speeds (and training times) for our method as well as prior works such as 3DGStream [16]. Our rendering process is similar to the original work of 3D-GS [9]. The original paper of 3DGS and many recent follow-up work (e.g. [B]) demonstrate that Gaussian splatting rendering can work efficiently on a consumer-grade device (e.g., A6000 or RTX 2080). Due to the similarity of the rendering implementation, we believe our rendering speed performance would be at a similar level to what was shown in these 3DGS papers.

[B] Xu et al. "Splatfacto-W: A Nerfstudio Implementation of Gaussian Splatting for Unconstrained Photo Collections." arXiv preprint arXiv:2407.12306 (2024).

Q: Accuracy-memory trade-off.

Please see the shared rebuttal response.

Dear reviewers, as we approach the final two days, please take a moment to review the author's responses and join the discussion. Thank you!