HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis

We sincerely thank the reviewer for their thorough and insightful review. Please find our point-by-point responses below.

[W1, Q1] Hyperparameter Sensitivity

We present sensitivity analysis of the key hyperparameters in our two-stage training process. Our framework consists of a Component Pre-training stage to establish a robust geometric and motion foundation, followed by a Joint Fine-tuning stage to refine the appearance and inter-object interactions. The analysis is therefore split into two parts. The results are averaged across all sequences in our dataset. Our chosen default parameters are highlighted and represent the best-performing configuration.

Stage 1: Component Pre-training Sensitivity

In this initial stage, the goal is to learn a stable canonical shape (SDF) and a coherent deformation field. The most critical hyperparameters are the weights for the depth, color, and optical flow losses.

The results show that a careful balance is required during pre-training.

• Depth loss ($\lambda_\text{depth}$). This is the most critical parameter in Stage 1. A lower weight (1.0) fails to establish a strong geometric foundation, resulting in significantly worse depth accuracy. Conversely, a very high weight (10.0) can cause the model to overfit to potentially noisy pseudo-ground-truth depth, creating geometrically rigid but less visually coherent surfaces, which slightly hurts all metrics. Our default of 5.0 provides the best balance.
• Color loss ($\lambda_\text{color}$). We found that a relatively small color weight (0.1) is crucial in this initial stage. A higher weight (1.0) can cause the model to "bake in" appearance and lighting details prematurely, which conflicts with the primary goal of learning a clean, underlying geometry and leads to degraded performance.
• Flow loss ($\lambda_\text{flow}$). The flow loss acts as a regularizer for motion. A low weight (0.1) results in less temporally consistent reconstructions. A very high weight (5.0) can cause overfitting to the estimated optical flow, introducing jitter and artifacts.

Stage 2: Joint Fine-tuning Sensitivity

Using the best model from Stage 1, this stage refines the entire scene with a focus on high-fidelity appearance and correct object interactions. The key parameters are the weights for the final photorealistic, normal, and depth losses.

In this refinement stage, the balance shifts towards achieving high visual quality while preserving the strong geometry from Stage 1.

• Photometric and normal losses ($\lambda_\text{photo}$, $\lambda_\text{normal}$). We observe that an equal weighting of 1.0 for the photometric and normal losses provides the best trade-off. Over-emphasizing the photo loss (5.0) can introduce minor visual artifacts as the model sacrifices geometric smoothness for pixel-perfect colors. Over-emphasizing the normal loss (5.0) can create overly sharp, non-photorealistic surfaces that hurt the LPIPS score.
• Depth loss ($\lambda_\text{depth}$). While appearance is key in this stage, maintaining a strong depth weight (5.0) is still vital. A lower weight (1.0) allows the geometry to drift from the metric ground truth, significantly impacting depth accuracy. A higher weight (10.0) offers diminishing returns and can slightly compromise the visual metrics by making the surfaces overly rigid.

In conclusion, our chosen hyperparameters represent a carefully tuned balance between geometric and appearance-based losses, tailored to the specific goals of each training stage. While the model shows some sensitivity, the performance remains robust within a reasonable range of our selected default values.

[W2, Q2] Generalization without Extensive Supervisions

We conducted an ablation study to validate the effectiveness of supervision components, with results below. Note that semantic masks are indispensable, as the pipeline fails without them. The results demonstrate that all supervision components are crucial for achieving our state-of-the-art performance, as ablating any of them leads to a significant degradation in quality. This analysis clarifies that our invertible deformation model and the geometric supervisions (like depth) play distinct, complementary roles: the depth loss establishes the scene's foundational 3D structure, while the invertible model ensures that the learned motion is stable and consistent over time. The large drop from ablating depth supervision does not mean our invertible model is ineffective. Rather, it shows that even a perfect motion model can't fix a scene with fundamentally incorrect geometry. You need both: correct geometry (from depth) and consistent motion (from our invertible deformation model). While our method relies on these priors, this is a standard and necessary approach (also adopted in MoSca, Shape-of-Motion, etc.) to constrain the fundamentally ill-posed problem of monocular dynamic reconstruction, and our ablation shows how each component contributes to the final high-quality result.

[M3] Static Component Initialization

Our method first decomposes the scene into static and dynamic components using foreground masks from SAM. The static background is then initialized by creating a 3D point cloud from initial camera poses (e.g., from ARKit) and estimated depth. The static components are initialized from this point cloud.

We sincerely thank the reviewer for their diligent feedback and for engaging with our previous rebuttal. We address each of your points below and will incorporate them into the revised manuscript.

How is the number of elements N (L168) determined for different scenes?

We follow Total-Recon to manually choose the number of scene elements ($N$) as a scene-specific hyperparameter. It is equal to the number of dynamic objects (the foreground) plus 1 (the static background).

While the authors provide a detailed ablation on loss weights, the sensitivity of these hyperparameters across scenes raises concerns about the method’s generalizability and ease of transfer to new scenarios.

We respectfully disagree with the assessment of hyperparameter sensitivity, as our experimental results demonstrate remarkable robustness rather than sensitivity. As shown in following per-scene ablation studies, a single, fixed set of hyperparameters consistently achieves the best or near-best performance across all 11 diverse test sequences for both training stages. This consistency shows that per-scene tuning is not required, confirming our method's strong generalizability and ease of transfer to new scenarios. Even when deviating from these optimal values, the performance degrades only mildly.

Additionally, echoing points raised by other reviewers, I believe that several key technical components are not clearly explained.

We appreciate your feedback on clarity. We will integrate all clarifications from this rebuttal into the revised manuscript. To ensure our method is unambiguous and fully reproducible, we will also open-source the complete codebase upon acceptance.

Please also let us know if there is any aspect remaining unclear to you. We would be happy to provide further explanation.

We sincerely thank the reviewer for their thorough and insightful review. Please find our point-by-point responses below.

[W1, Q1] Handling Higer-resolution Inputs

We initially used a lower resolution to ensure fair comparisons with baselines like Total-Recon. To address your point, we re-trained our HoliGS on the original high-resolution (1K) inputs, which improves performance across all metrics. This confirms that our model successfully leverages the high-frequency details available in the 1K inputs to produce more accurate reconstructions. Further, we believe the more modest gains in structural metrics (SSIM and depth) are primarily limited by the significant motion blur present in the casually captured source videos, which serves as an imperfect ground truth.

[W2] Floater-like Artifacts

We agree with the reviewer that the floater artifacts are an inherent limitation of our current design. Our analysis points to three root causes, each suggesting a clear direction for future work.

Skinning weight discontinuities. Artifacts appear at articulation boundaries (e.g., tails, hands) where conflicting motion signals from different bones pull Gaussians into physically implausible positions. This could be addressed with adaptive, time-varying skinning weights that adjust based on local motion.

Discrete primitive limitations. Unlike continuous fields, discrete Gaussians cannot smoothly "stretch" across high-deformation regions. During rapid motion, they can land in mathematically averaged, non-physical intermediate states. A promising solution is targeted densification, spawning new Gaussians in high-deformation zones to better capture the surface.

Hierarchical motion dominance. Large global motions can sometimes dominate the local deformation field, especially at extremities, as the model struggles to fully counteract the root transformation. Introducing physics-informed constraints could penalize non-physical states and help disentangle global and local motion.

[Q2] Citing SMPL

Thank you for the suggestion. We will ensure a proper citation for SMPL is included in the limitations section of the revised manuscript.

We sincerely thank the reviewer for their thorough and insightful review. Please find our point-by-point responses below.

[W1, Q1] Key Insights

While with a piepline similar to Total-Recon, HoliGS does not simply replace the NeRF representation in Total-Recon with GS. A naive swap fails to converge because GS is notoriously sensitive to its starting configuration. Based on this observation, we introduce a pre-training stage that optimizes a Neural SDF proxy. This step, which is not present in Total-Recon's pipeline, serves two crucial purposes: it first optimizes our deformation networks to a robust state, and then produces a high-quality surface from which we sample points to provide an effective initialization for the Gaussians. As our ablation study confirms, removing this initialization causes a severe drop in performance, validating it as a necessary and novel component of our method.

[W2] Efficiency Comparison

We benchmarked all methods on a single NVIDIA RTX A6000 GPU. As summarized in the table below, HoliGS offers a significant training efficiency advantage, especially when compared to high-fidelity NeRF-based methods. This improved efficiency makes the reconstruction of long, dynamic scenes practical, avoiding the prohibitive training times of previous state-of-the-art approaches.

[W3] Reliance on Off-the-shelf Models

We would like to clarify that our methodology is common practice. The use of external priors from networks for depth, flow, and segmentation is a cornerstone of today's leading monocular 4D reconstruction pipelines, including MoSca (CVPR 2025) and Shape-of-Motion (ECCV 2025). We chose this established, tracker-free design for its efficiency, which allows HoliGS to scale to minute-long videos where dense-tracking methods like G-Marbles (Siggraph Asia 2024) would fail due to memory constraints. Although fully self-supervised methods are a valuable research goal, this pragmatic design is what currently makes high-quality, long-duration dynamic reconstruction a tractable problem. While a purely self-supervised approach is an important future goal, this pragmatic design is what enables HoliGS to achieve the high-quality, minute-long reconstructions demonstrated in our work.

[W4] Key Mathematical Details

We agree that adding more mathematical details will improve the paper's clarity, and we will incorporate them into the main paper in the revised manuscript.

[Q2] Downsides of Replacing NeRF with GS

We acknowledge there are several downsides of GS compared to NeRFs. GS is less robust, being highly sensitive to initialization (requiring our SDF bootstrap to prevent a 35% PSNR drop) and demanding more complex optimization heuristics. As an explicit representation, it also faces memory scalability issues when the scene is complex and can struggle with the topology of fine structures like fur. However, we found these challenges to be a worthwhile trade-off for the paramount benefits of 10$\times$ faster training and real-time rendering capabilities.

[L1] Generative Priors

Integrating generative priors represents an promising future direction for this field. Such models could be used to create truly holistic scene reconstructions by completing geometry for unobserved regions, like the back of a sofa or the underside of a table, thus enabling true free-viewpoint rendering. Furthermore, generative priors could operate in the temporal domain to synthesize plausible dynamics during moments of heavy motion blur or occlusion, leading to smoother, more realistic animations. One could even leverage cross-modal, text-conditioned models to inject high-level semantic details, such as using the prompt "wooden table" to realistically generate its unseen legs and texture. We are excited to explore it in the future.

We sincerely thank the reviewer for their thorough and insightful review. Please find our point-by-point responses below.

[W1] Clarity

Thank you for your detailed comments. We hereby address all points raised and will revise the manuscript accordingly to ensure a thorough explanation of our method.

[W2] Input Camera Pose

We initialize our model using camera-to-world poses from onboard motion sensors (e.g., ARKit), which are then jointly refined during optimization by predicting per-frame delta SE(3) twists. Background Gaussians are initialized from these initial poses. For the foreground, we first pre-train a neural Signed Distance Function (SDF) and then initialize Gaussian centers by sampling directly on its learned surface. Since we lack independent ground-truth extrinsics beyond the noisy ARKit data, we evaluate performance based on reconstruction quality, such as Novel View Synthesis and depth accuracy, rather than a standalone pose-error metric.

[W3] PoseNet

We use PoseNet in an off-the-shelf manner, while further training it during optimization.

[W4] A Detailed Breakdown of Figure 3

Our two-stage pipeline uses two different types of renderers. The "differentiable rendering" in Stage 1 is volumetric rendering used to learn the geometry; it is completely separate from the Gaussian rasterization in Stage 2. Here is a breakdown of the inputs and outputs for each module.

Stage 1: Geometry & Deformation Initialization

This stage learns the object's shape (as a Neural SDF) and its deformation model. It does not use Gaussian Splatting.

Pose Estimation Network

• Input: A single video frame at time $t$.
• Output: The object's initial root pose and skeleton parameters for that frame.

Articulation & Soft Deformation Networks

• Input: A 3D point $X^*$ in the canonical (rest) space, plus the root pose and skeleton from the pose network.
• Output: The warped 3D point $X^t$ in the live camera space for that frame.

Camera Intrinsics MLP

• Input: The frame index $t$ and initial camera intrinsic values (e.g., focal length derived from image dimensions).
• Output: Refined camera intrinsics for that specific frame (focal lengths $f_x, f_y$ and principal point $c_x, c_y$). This allows the model to learn and correct for minor inaccuracies in the initial camera parameters, leading to a more consistent 3D reconstruction.

Neural SDF Network

• Input: A 3D point $X^*$ in the canonical space.
• Output: A signed distance value, which defines the object's base shape.

Differentiable Rendering (Volumetric Renderer)

• Input: Camera parameters (from pose estimation + camera MLP), the learned deformation networks, and the Neural SDF.
• Output: A rendered RGB image, depth map, and mask. These are compared against the ground-truth video to train the SDF and deformation networks.

Sample GS (Final step of Stage 1)

• Input: The fully trained Neural SDF.
• Output: A point cloud of 3D points sampled from the SDF's surface. These points become the initial centers for the Gaussians in the next stage.

Stage 2: Deformable Gaussian Splatting

This is the main optimization stage that uses the efficient differentiable Gaussian rasterizer.

• Input: The initial Gaussians from the "Sample GS" step, the pre-trained deformation networks from Stage 1, and a set of static background Gaussians.
• Output: The final, high-fidelity rendered image, depth map, and normals, which are used to optimize the properties of the Gaussians (color, scale, rotation, opacity).

[W5] Memory and Time Comparison

Unlike other state-of-the-art 4DGS methods (e.g., MoSca, Shape-of-Motion) that rely on dense point-tracking, our approach avoids its prohibitive resource costs. A dense tracker can exhaust even an 80GB A100 GPU on sequences with over 300 frames, making minute-long video reconstruction infeasible. In contrast, HoliGS employs a pipeline of lightweight networks, reducing memory and per-frame overhead to less than 25% of a dense tracker’s. This efficient design enables scalable reconstruction on long videos, as detailed by the modest VRAM usage shown in the table below.

[W6] Shorter Sequence Comparison

To ensure fair comparisons with baselines that fail on long sequences due to memory constraints, we evaluated all methods on shorter, 200-frame segments with 100-frame strides for long sequences that originally lead to OOM. The experimental results reveal that HoliGS consistently outperforms or remains highly competitive with specialized short-clip methods across nearly all evaluated sequences and metrics. We analyze the results as follows.

Strength of HoliGS. The results do not show a trade-off. Instead, they highlight the comprehensive strengths of our method. Specifically, across all six test sequences, HoliGS wins or ties on 22 of the 30 primary metric comparisons (~73%). The superior performance of HoliGS even on these shorter clips underscores the benefits of our core design. By building a globally consistent canonical model and jointly optimizing camera poses, our method offers greater robustness to varied motion and complex object interactions. This validates that HoliGS is not only more scalable but also generally more accurate, making it a more reliable and powerful solution for "in-the-wild" dynamic scene reconstruction.

Specific strengths of baselines. The baseline method MoSca consistently achieves the best LPIPS score, suggesting it excels at producing perceptually plausible results. However, this advantage in perceptual quality does not translate to better geometric or colorimetric accuracy, where HoliGS is demonstrably stronger.

[W7] Supplemental Video

Depth map quality. While overall error metrics are similar, our method produces qualitatively superior depth maps with noticeably sharper and more coherent details. For instance, in our supplement (Sec. S3), the results show crisper definition on the animal’s legs (Fig. S5) and cleaner surfaces on the table legs (Fig. S6) compared to Total Recon.

Video synchronization. Both HoliGS and Total-Recon refine their initial camera poses during training. With different architectural biases, HoliGS and Total-Recon converges to a slightly different optimal camera trajectory. This "representation-dependent drift" means the novel-view videos, which are derived from these diverged paths, are not perfectly spatially aligned, leading to the illusion of a timing mismatch. The videos are temporally synchronized by frame count, which can be confirmed by observing specific actions within each render.

[L1] Handling Rigid Objects

We did not explicitly analyze performance on rigidly moving objects such as cars, furniture, or other non-articulated foreground elements due to the scope of this work. While our primary focus was not on rigid objects, our hierarchical framework inherently handles them as a simplified case of articulated motion. Our model can represent a rigid object by using a single-bone skeleton and disabling non-rigid deformations, letting the global SE(3) transformation capture all movement. Preliminary tests on a sequence with a rigidly moving car confirm this works well, achieving over 30 PSNR on training views. We agree that a full evaluation incorporating more diverse forms of objects is a valuable next step, and we look forward to exploring this promising direction in future work.