We thank the reviewer for appreciating our framework design and recognizing our videos as impressive results.

We now address the concerns.

Robustness of HoloScene:
HoloScene exhibits robustness to individual foundation model failures by jointly optimizing multiple energy terms and selecting the best among sampled solutions. Failures in one module typically introduce inconsistencies, which are reflected as higher energy, making such samples less likely to be selected. This contrasts with traditional multi-stage pipelines, where a single module failure can propagate downstream and compromise the entire output. This robustness is reflected in both our real-world examples—where foundation models are imperfect—and in benchmark metrics.

That said, this sampling-based robustness is not a guarantee against all types of failures. Catastrophic errors or insufficient sampling coverage, as shown in some of our failure cases, can still lead to imperfect final results.

Outdoor scenes:
HoloScene, like other compared methods, such as DP-Recon[1] and ObjSDF++[2], mainly focuses on and conducts experiments on indoor scenes. Outdoor scene setting features dynamic motions and unbounded geometry, which is out of our claimed scope. We will leave the outdoor simulation-ready environment reconstruction for future work.

Training details:
We will open-source our full codebase upon acceptance to support reproducibility. For the reviewer's reference, we provide additional training details in the table below and will include these parameters in the final version.

Gradient-based optimization (stage 1):

Sampling-based optimization (stage 2):

Gradient-based Refinement (stage 3): (for Gaussian refinements)

Average Training time (one A6000 GPU):

Models used:

SDF penetration term details:
In each iteration, we sample 4096 points uniformly in the 3D space, and calculate the SDF values of all instances at those points. We then apply $E_{\mathrm{pene\_sdf}} = \sum_{\mathbf{x} \in R_i}\sum_{k\neq i}\max\bigl(0,\,-g_k(\mathbf{x}) - g_{i}(\mathbf{x})\bigr)$ on these points as the penetration loss to optimize. By uniformly sampling points, we can reduce the calculation complexity.

Scene graph inference, writing, and novelty:
Thank you for the suggestion. We will expand the related work section to include a discussion of prior work on scene graphs, including Neural Scene Graphs [Ost et al., CVPR 2021], and cite relevant references.

We do not claim the use of a VLM for scene graph inference as our main contribution. Rather, we emphasize that the scene graph plays a critical role in HoloScene for enforcing physically plausible interactions (see Tab. 3, L280–L286). While the idea of a scene graph itself is not novel, our energy-based formulation and sampling-based inference framework built on top of the scene graph is a core contribution of HoloScene. This integrated approach enables robust and physically consistent scene reconstruction from video.

Evaluation results analysis:
Here we explain our evaluation results in Tab. 2. We will include a more detailed explanation in the final version.

• Real-time PSNR gap: We adopt the Gaussian on Mesh (GoM) from DRAWER[4]. Compared with standard 3DGS, the position, orientation, scale, and other properties of every Gaussian are regularized, so they can align better with mesh surfaces. While this strategy limits the rendering quality, the real-time Gaussian rendering aligns with the accurate geometry, improving visual quality in downstream simulation applications. This is a key trade-off for attaining a simulation-ready environment. It is noticeable that the gap happens in DRAWER[4], GaMeS[5], and other GoM papers as well.

• Correlation of PSNR/SSIM/LPIPS: PSNR and SSIM/LPIPS are correlated when we compare the real-time rendering quality in Tab. 2. But those three metrics sometimes are not necessarily correlated. For example, we can witness uncorrelated PSNR and SSIM/LPIPS in Table 1 of the Mip-NeRF 360[6] paper.

• iGibson results: The object-level evaluation on iGibson scenes shows low PSNR values because the evaluation is performed on the whole object, including unseen regions, making it especially challenging. (L274-L275) Therefore, the PSNR/SSIM/LPIPS are lower than common reconstruction task. We will add more qualitative examples in the camera-ready version.

Diverse results of resampling:
Thank you for pointing this out — there may have been a misunderstanding. By "diverse virtual observations," we refer to the diversity in invisible regions when generative resampling with different seeds. This diversity helps us explore multiple plausible completions and select the one that best minimizes our regularization terms.

Importantly, multi-view consistency is preserved through the use of an explicit 3D representation (GoM), ensuring that renderings from different viewpoints remain consistent with each other.

Code open source:
We will open-source all the code, data, and running scripts upon acceptance, and we believe this will encourage future research from the community.

[1] Ni, Junfeng, et al. "Decompositional neural scene reconstruction with generative diffusion prior." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[2] Wu, Qianyi, et al. "Objectsdf++: Improved object-compositional neural implicit surfaces." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Ke, Bingxin, et al. "Repurposing diffusion-based image generators for monocular depth estimation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

[4] Xia, Hongchi, et al. "Drawer: Digital reconstruction and articulation with environment realism." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[5] Waczyńska, Joanna, et al. "Games: Mesh-based adapting and modification of gaussian splatting." arXiv preprint arXiv:2402.01459 (2024).

[6] Barron, Jonathan T., et al. "Mip-nerf 360: Unbounded anti-aliased neural radiance fields." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

We thank the reviewer for appreciating HoloScene as a solid integrated system with visually compelling results and various downstream applications.

We now address the concerns.

Comparison with CAST [1]:
HoloScene differs from CAST in problem setting, inference stage, and overall purpose. We provide a detailed comparison in Table 1 and the related work section, and summarize key differences below:

• Problem setting:
  CAST generates component-aligned 3D scenes from a single image, and its outputs do not necessarily match the input image in appearance or geometry (see CAST’s overview figure). In contrast, HoloScene reconstructs simulation-ready digital twins from a single video, aiming to replicate geometry, appearance, and physical plausibility from the input video.

• Inference stage:
  CAST relies on a generative model and uses differentiable optimization refining objects to avoid object collisions. HoloScene combines reconstruction, generation with sampling-based optimization, minimizing both observation energy (to match input video appearance) and regularization energy (for physical plausibility and object completion). While it leverages Wonder3D [3] for generation, reconstruction is guided by observed data.

• Purpose:
  CAST focuses on content creation with clean, component-aligned 3D assets from single images. HoloScene instead targets realistic, faithful, physically consistent reconstruction from video for simulation-ready environments. Both realism and fine-level fidelity are our key goals.

Comparison with DRAWER [2]:
We discussed DRAWER[2] in the related work section (L69-L71). Though HoloScene adapts Gaussians on Mesh from DRAWER[2], we have distinct differences with them. Here we will explain the detailed comparisons of HoloScene and DRAWER[2]. We also include a table below for the reviewer's reference.

• Focus:
  DRAWER[2] focuses on the articulated object reconstruction and lacks the shape completion and stable physics for some rigid-body objects. However, HoloScene focuses on all rigid-body objects in the input video, and reconstructs them with invisible region completion, collision avoidance, and physical stability in the simulation.
• Approach:
  In DRAWER, rigid-body objects are decomposed following Video2Game[4]. From the paragraph of Rigid object decomposition in DRAWER's supp material page 9, we can see that, although rigid-body objects in DRAWER can be segmented from the SDF field using 3D bounding box, DRAWER doesn't decompose every rigid-body objects and the objects lack the completion for invisible regions, so that they can not be guaranteed with physical stability in the simulation. Unlike DRAWER[2], HoloScene decomposes every rigid-body object in the video. Instead of direct segmentation from the SDF field, every object is modeled by its own SDF for convenient 3D decomposition. HoloScene also adapts a generative sampling approach to optimize the completion and stability of every rigid-body object. We show our quantitative comparison results with DRAWER in the table below. We compare on all the rigid-body objects in the scene.
• Experiments:
  We compare the scene-level and object-level performance on the iGibson dataset with DRAWER[2]. We provide the ground-truth object 3D bounding boxes to DRAWER. We show the evaluation results in the table below. From the results, we can witness that DRAWER[2] fails to get complete geometry (according to object-level CD/F1/NC and PSNR/SSIM/LPIPS evaluation) and plausible physics from the input video, and can not ensure stability even for objects on the ground (according to the stability evaluation).

Technical contribution and system design novelty:
We agree that system integration is a key part of HoloScene’s technical contribution. However, the novelty and intellectual contribution go beyond integration. Our energy formulation and inference design are both novel, enabling HoloScene to jointly optimize for appearance-geometry alignment and physical-geometric plausibility. This combination makes HoloScene an effective and principled framework for simulation-ready scene reconstruction.

[1] Yao, Kaixin, et al. "Cast: Component-aligned 3d scene reconstruction from an rgb image." arXiv preprint arXiv:2502.12894 (2025).

[2] Xia, Hongchi, et al. "Drawer: Digital reconstruction and articulation with environment realism." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[3] Long, Xiaoxiao, et al. "Wonder3d: Single image to 3d using cross-domain diffusion." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

[4] Xia, Hongchi, et al. "Video2game: Real-time interactive realistic and browser-compatible environment from a single video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We thank the reviewer for recognizing our unified framework and energy-based optimization design for simulation-ready environment reconstruction.

We now address the concerns.

Literature review:
We discuss all the papers mentioned in the review, and will include them in the final version.

• Digital Cousins [1]:
  In the first row of Table 1, we include Digital Cousins (ACDC) [1], a single-image reconstruction-by-retrieval approach that operates under a fundamentally different setting than HoloScene. It retrieves plausible 3D assets based on similar semantics and layouts rather than precisely reconstructing the actual scene geometry and appearance. As such, it is not directly comparable to our method under the rendering or reconstruction metrics we report, since it does not aim to faithfully replicate the original scene. In contrast, HoloScene is designed to balance both simulation capability (as explored in Digital Cousins) and scene fidelity (as targeted by reconstruction methods).
• DeepPriorAssembly [2]:
  DeepPriorAssembly [2], like Digital Cousins [1], is a single-image reconstruction method and is not designed to faithfully reconstruct scenes consistent with image sets or video input—the primary focus of HoloScene. In Table 1 and the related work section, we already discuss methods with similar settings, including Gen3DSR and CAST. We will include DeepPriorAssembly [2] in the final version for completeness.
• Scenethesis [4]:
  Scenethesis is a text-conditioned scene generation method, with a goal fundamentally different from ours. HoloScene focuses on reconstructing simulatable digital twins from images or videos. We also note that Scenethesis appeared on arXiv on May 5, 2025. According to the NeurIPS 2025 FAQ, “Papers appearing online after March 1st, 2025 are generally considered concurrent to NeurIPS submissions. Authors are not expected to compare to those.”
• Generation / retrieval-based scene generation:
  Methods such as Holodeck [3], Digital Cousins [1], Scenethesis [4], and Architect [5] generate 3D scenes from text or sparse inputs. While their outputs are also object-centric scene assets, their primary goal is to create diverse and plausible layouts—not to reconstruct high-fidelity digital twins faithful to input images in appearance or geometry. Consequently, these methods are not directly comparable to HoloScene under reconstruction or rendering metrics. In contrast, HoloScene integrates generative priors into a reconstruction pipeline that accurately replicates input videos and produces simulation-ready environments. It further leverages IsaacSim to ensure physical stability (see Eq. 5), as demonstrated in Fig. 4.

We will make these distinctions and contributions more explicit in the final version.

Baseline comparisons:
Thank you for the suggestions. We respectfully disagree with this concern. The methods mentioned in the reviews—Digital Cousins [1], Holodeck [3], Scenethesis [4], and Architect [5]—are retrieval or generation-based and not designed to produce 3D scenes that align accurately with input images or videos at pixel-level appearance and centimeter-level geometry, as elaborated in the previous response. These approaches target a different goal: generating diverse and plausible virtual scenes. While they reflect the state of the art in that domain, they are not representative of the current state of high-fidelity, simulation-ready reconstruction and twinning. As such, they are not directly comparable using our reconstruction or rendering metrics.

In contrast, our selected baselines are task-aligned and up-to-date. DP-Recon [6], included in our experiments, is a recent state-of-the-art method published at CVPR 2025 (June). ObjSDF++ [7], while introduced earlier, shares the same multi-view reconstruction setting as HoloScene and remains a strong and relevant baseline. Along with PhyRecon, these methods reflect the state of the field at the time of submission. We do not believe ObjSDF++ loses its relevance simply because it has been previously compared with DP-Recon.

Overall, we believe our evaluation setup and baseline choice are fair, representative, and meaningful for assessing progress on this task.

Failure cases analysis:
Our paper has shown failure cases caused by imperfections in foundation models, including inpainting from LaMa[8] and completion from Wonder3D[9], which can be found in Fig. 2, 3 in the main paper and Fig. 8 in the supplementary material and videos. We can witness that although Holoscene has achieved superior results over baselines, it still suffers from imperfect foundation model results. We will include a section on failure case analysis. Here is a brief analysis of failure cases:

• Inpainting:
  The inpainting results with LaMa are not necessarily consistent with the background. (See the background inpainting color in the bottom of Fig. 2, and the sofa inpainting color in our object appearance results of Fig. 3)
• Completion:
  The generation results of Wonder3D are not necessarily perfect as well. From Fig. 3, we can see the geometry of the back of the sofa has some artifacts (somewhat concave but not flat), which is due to the imperfect predictions from Wonder3D.
• Physics:
  From Fig. 8 in the supplementary material and the simulation videos in the supplementary material, in the experiments with the iGibson[10] dataset, we observe that some objects still deviate from original poses after applying gravity, which is due to imperfect predictions by LaMa and Wonder3D.
• Conclusion:
  The failure cases are commonly from the foundation model's imperfect results. Holoscene could keep improving performance by leveraging more advanced foundation models.

Reliability against foundation model failures:
HoloScene exhibits robustness to individual foundation model failures by jointly optimizing multiple energy terms and selecting the best among sampled solutions. Failures in one module typically introduce inconsistencies, which are reflected as higher energy, making such samples less likely to be selected. This contrasts with traditional multi-stage pipelines, where a single module failure can propagate downstream and compromise the entire output. This robustness is reflected in both our real-world examples—where foundation models are imperfect—and in benchmark metrics.

That said, this sampling-based robustness is not a guarantee against all types of failures. Catastrophic errors or insufficient sampling coverage, as shown in some of our failure cases, can still lead to imperfect final results.

Computation efficiency:
Simulation and rendering runtime: Our simulation-ready environments support high-fidelity physical simulation and rendering at over 100 FPS on a single A6000 GPU. Reconstruction runtime The HoloScene reconstruction stage takes an average of 8 hours per scene. For comparison, DP-Recon [6], a state-of-the-art baseline, requires over 14 hours on average for similar reconstruction tasks.

Regularization terms:
We discuss completeness energy (L159-L167), geometry energy (L168-L175), and physics energy (L176-L183) in Sec 3.2. Because of the limit of maximum characters, we will include the intuition and a more detailed formulation for each energy term in the final version.

Criteria for foreground and background objects:
We define foreground objects as those that can be affected by external forces, such as gravity. For example, walls and wall-mounted structures (e.g., ceiling lights) are considered background objects, as they are fixed and unlikely to move under simple external forces. In contrast, objects merely leaning against walls—such as bookshelves—are considered foreground, as they can respond to physical interactions.

Object Reconstruction (OR) metric:
Thank you for pointing this out — there may have been a misunderstanding. The OR metric quantifies how many objects are present in the final reconstruction, regardless of their completeness. This metric is not designed to evaluate object completeness quantitatively; instead, the object-level CD, F1, and NC metrics in Table 2 serve that purpose.

In our baseline experiments, some small objects are missing from the reconstructions because no valid SDFs exist for these objects, preventing mesh extraction via marching cubes. This occurs due to their limited appearance in training images and the constraints of certain loss terms (such as the SDS loss in DP-RECON). In contrast, HoloScene addresses this issue by implementing balanced sampling across instances, ensuring robust reconstruction of all objects.

Examples of missing objects in baseline reconstructions can be found in Figures 5-8 in the supplementary material. Figure 7 (the "no physics" column) shows DP-Recon reconstructing only one object on the shelf, while Figure 8 demonstrates that all baselines fail to reconstruct the small vase.

[1] Dai, Tianyuan, et al. "Automated creation of digital cousins for robust policy learning."

[2] Zhou, Junsheng, et al. "Zero-shot scene reconstruction from single images with deep prior assembly."

[3] Yang, Yue, et al. "Holodeck: Language guided generation of 3d embodied ai environments."

[4] Ling, Lu, et al. "Scenethesis: A language and vision agentic framework for 3d scene generation."

[5] Wang, Yian, et al. "Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting."

[6] Ni, Junfeng, et al. "Decompositional neural scene reconstruction with generative diffusion prior."

[7] Wu, Qianyi, et al. "Objectsdf++: Improved object-compositional neural implicit surfaces."

[8] Suvorov, Roman, et al. "Resolution-robust large mask inpainting with fourier convolutions."

[9] Long, Xiaoxiao, et al. "Wonder3d: Single image to 3d using cross-domain diffusion."

[10] Li, Chengshu, et al. "igibson 2.0: Object-centric simulation for robot learning of everyday household tasks."

I appreciate the author's effort to differentiate the current generative method-based SOTA from Holoscene. While I remain somewhat unconvinced by the method's reliability and robustness, the authors have demonstrated its advancements through comparisons with several baselines and a discussion of CAST. Since these methods aim to achieve similar goals, it is important to include a statistical analysis of Holoscene’s failure rate and a direct comparison with the current reconstruction-based SOTA. Additionally, considering that Holoscene requires a significant amount of time to reconstruct a scene (averaging 8 hours per scene), it would be helpful to clarify its advantages over mentioned generative 3D scene methods.

We thank the reviewer for appreciating the elegant and comprehensive design and superior performance of the simulation-ready environment.

We now address the concerns.

Why is an SDF representation needed?
Directly optimizing surface meshes from video is challenging due to their complex and discrete topology, especially in large indoor scenes. Neural SDFs provide a flexible, continuous representation that simplifies optimization and has demonstrated superior geometric quality [6, 7]. Once optimized, we extract meshes via marching cubes for simulation and attach Gaussians for high-quality rendering. This design balances reconstruction accuracy, topological flexibility, and downstream usability.

Are SDFs and meshes synchronized?
Yes. Meshes are extracted from the SDF using marching cubes (L199–L206), ensuring they are fully synchronized.

Is geometry evaluated with meshes or SDFs?
In Table 2, we evaluate geometry quality using the final reconstructed mesh, following standard protocols used by our baselines. This ensures fair comparison across diverse approaches and accurately reflects downstream performance in simulation and other mesh-dependent tasks.

Could you show example input videos (how dense/sparse they are)?
Yes! We will include example input videos in the supplementary video and project website to illustrate capture density. Real-world inputs from ScanNet++ [2] are 1–2 minute sequences at 30 FPS with smooth handheld trajectories. Our synthetic datasets (Replica and iGibson) follow the same setup with comparable frame density and motion patterns.

What is the typical length?
The table below summarizes the average number of frames across different datasets. Our collected videos follow the capture speed and density of ScanNet++ [2]. Typical sequence lengths are several hundred frames for small indoor scenes (Replica, ScanNet++) and over a thousand frames for larger scenes (iGibson). For data we collected ourselves, we used handheld RGB cameras with smooth trajectories at 30 FPS, consistent with ScanNet++ practices. HoloScene is designed to handle input videos of varying lengths with ease. For additional statistics and visualizations, please refer to the ScanNet++ website.

Can HoloScene handle unstructured image collections?
Yes! HoloScene supports unstructured image collections as input. The main difference lies in scene graph construction (L193–198): for videos, we sample keyframes and estimate relationships using a VLM, while for unstructured image collections, we can compute pairwise camera distances and incrementally build the scene graph. The remaining inference pipeline remains unchanged (L198).

Why is the reconstruction error at object level lower than scene level on iGibson?
This difference is due to scale and sampling resolution. Our evaluation follows standard geometry protocols [8, 9], which compute metrics by sampling points on surface geometry. At the object level, denser sampling within smaller regions increases the likelihood of finding closer correspondences, leading to lower errors. In contrast, scene-level evaluations span larger and more complex spaces, resulting in higher average error. Therefore, object- and scene-level errors are not directly comparable. That said, within each setting, the relative rankings across baselines still reliably reflect reconstruction quality.

Explanation of the idea of balancing samples across instances:
In Stage 1 (L199–L206), we sample an equal number of pixels for each instance in every optimization iteration. This ensures that smaller objects receive sufficient attention during training and are not overshadowed by larger instances (see L203 and L268).

How is occlusion inpainting done?
We first obtain reconstructed geometry for each object after Stage 1 inference (L199). Then, LaMa takes the rendered instance masks as input to inpaint occluded regions (L164). The inpainted images are subsequently passed to Wonder3D to generate novel views for invisible regions.

Question about rendering quality and potential improvements:
We adopt the Gaussian-on-Mesh (GoM) approach from DRAWER [4]. Unlike standard 3D Gaussian Splatting, GoM constrains each Gaussian’s position, orientation, and scale based on the underlying mesh. While this better supports physical simulation and improves scene editability, it reduces the flexibility of Gaussian optimization compared to unconstrained 3DGS, which often trades geometry accuracy for rendering quality. This trade-off explains the slightly blurrier appearance, as also reflected in the rendering metrics in Table 2. Similar limitations are observed in other GoM-based methods such as DRAWER [4] and GaMeS [5]. Rendering quality can be improved by densifying the Gaussians or relaxing mesh constraints, though this may reduce the alignment between appearance and geometry—potentially harming simulation fidelity.

[1] Straub, Julian, et al. "The replica dataset: A digital replica of indoor spaces." arXiv preprint arXiv:1906.05797 (2019).

[2] Yeshwanth, Chandan, et al. "Scannet++: A high-fidelity dataset of 3d indoor scenes." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Li, Chengshu, et al. "igibson 2.0: Object-centric simulation for robot learning of everyday household tasks." arXiv preprint arXiv:2108.03272 (2021).

[4] Xia, Hongchi, et al. "Drawer: Digital reconstruction and articulation with environment realism." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[5] Waczyńska, Joanna, et al. "Games: Mesh-based adapting and modification of gaussian splatting." arXiv preprint arXiv:2402.01459 (2024).

[6] Yariv, Lior, et al. "Bakedsdf: Meshing neural sdfs for real-time view synthesis." ACM SIGGRAPH 2023 conference proceedings. 2023.

[7] Li, Zhaoshuo, et al. "Neuralangelo: High-fidelity neural surface reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[8] Wu, Qianyi, et al. "Objectsdf++: Improved object-compositional neural implicit surfaces." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[9] Ni, Junfeng, et al. "Decompositional neural scene reconstruction with generative diffusion prior." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.