LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Control and Rendering

Thank you for reviewing our paper and for the valuable feedback!

#Q1. Enlarge texts in figures.

Thanks. We have carefully revised the manuscript according to your comments.

#Q2. Visualization of some latent features.

We provide additional interaction feature visualization of x-$\boldsymbol{\kappa}$, y-$\boldsymbol{\kappa}$, and z-$\boldsymbol{\kappa}$ in Fig.1(a) of Attached PDF to illustrate latent feature distribution. It can be seen that the features are clustered around the spatial coordinates of interactive objects, corresponding to the local deformable fields in Sec 3.2 of the manuscript..

#Q3. More qualitative results on real-world and public datasets. K-Planes results were missed.

To our knowledge, existing view synthetic datasets for interactive scene rendering are primarily limited to a few interactive objects falling short of scene-level interactive reconstruction. Fig.5 of the Attached PDF provides visualization comparisons with CoNeRF[1] and CoGS[3] in CoNeRF Controllable dataset. As the results show, LiveScene outperforms existing SOTA methods for interactive rendering, achieving higher-quality rendering. More detailed experiments will be provided in the revised version. We must emphasize that K-Planes[2] was originally designed for 4D reconstruction and thus it is not suitable and unfair for controllable rendering. Meanwhile, our enhanced baselines, MK-Planes and MK-Planes* in Line 218 of Sec.5 of the manuscript., require dense interactive variable inputs, which are unavailable in the InterReal and CoNeRF Controllable datasets, making the comparison with K-Planes infeasible.

#Q4. Memory cost of the proposed method

As shown in Table 1, we report GPU memory usage in the seq002_Rs_int sequence of the OminiSim dataset with official parameter settings. Notably, LiveScene (w/o language grounding) requires approximately 8G GPU memory, which is lower than that of the other methods.

#Q5. The finest granularity of interaction can it handle.

The minimum granularity of controllable objects is hard to measure, as it depends on scene complexity, object number, and camera view. Our method still achieves precise control in extreme cases, such as the chest in the anonymous link video demos. In this work, we primarily focus on joint objects in indoor scenes, and finer-grained control is still feasible as long as the dataset provides relevant mask and control variable labels*.

#Q6. Handle repetitive objects.

Good question. In fact, due to the limitations of CLIP's spatial relationships understanding ability, our method does not perform well in distinguishing between repeated objects. This limitation is a commonality among many 3D vision-language field methods, such as LERF[4] and OpenNeRF[5]. We’ll clarify this in the revised version.

#Q7. Annotation error in Supp. Fig16.

Thanks. It’s a typo, and the correct language query is "microwave".

#Q8. More discussion of real-world potential limitations and societal impact.

Beyond the existing discussions in Sec.6 of the manuscript., a known limitation in real-world scenarios is that occlusion between objects can affect the interactive rendering effect. Besides, our method currently requires dense GT control variable annotation, which can be time-consuming and labor-intensive to obtain in real-world scenarios. We plan to explore sparse GT input methods (e.g., 3-frame annotation) to improve efficiency.

Regarding social impact, our method is committed to building interactive simulators from real-world scenarios, providing real-to-sim interactive environments for Embodied AI, e.g. navigation, grounding, and action. However, as with all work that enables editable models, our method has the potential to be misused for malicious purposes such as deep fakes. We’ll clarify this in the revised version.

Reference

[1] CoNeRF: Controllable Neural Radiance Fields, CVPR2022

[2] K-Planes: Explicit Radiance Fields in Space, Time, and Appearance, CVPR2023

[3] CoGS: Controllable Gaussian Splatting, CVPR2024

[4] LERF: Language Embedded Radiance Fields, ICCV 2023

[5] OpenNeRF: OpenSet 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views, ICLR2024

Sincerely appreciate for your kind comments and voluntary review suggestions! We have carefully reviewed and corrected the entire manuscript to improve the paper's organization and presentation.

#Q1. Improve the presentation of the methods section

Thanks! We have carefully revised the manuscript to eliminate presentation issues.

#Q2. Explain the Interaction Space Factorization

Please refer to Fig.2 and Sec.3.2 of the manuscripts, where we decode and render the interactive space using spatial coordinates $(x,y,z)$ and interactive variables $(\kappa_{1},\kappa_{2},...,\kappa_{\alpha})$ as inputs. The basic idea is to divide the $3+\alpha$ space into $\alpha$ independent local deformable field, each of which interacted with the 4 coordinates $(x,y,z,\kappa_{d})$. Specifically, we use an interaction probability decoder $\boldsymbol{\Theta}$ (MLP with the plane feature) In Fig.2 to predict the probability distribution of the ray samples, thereby determining the local deformable field region to which the sampling points belong. In the interactive ray sampling, each sample is associated with a distinct time coordinate and interaction variable. Therefore, through probability maximization, we can convert this $3+\alpha$-dimensional sampling into a 3+1-dimensional sampling, wherein the index $d$ of interaction variables is derived by maximizing the probability distribution of decoder $\Theta$. In this way, we map the interaction variables $\boldsymbol{\kappa}$ to the most probable cluster region $\mathcal{R} _ {d}$ in the 4D space and accomplish the "projection":

$d = \text{argmax} _ {i}$ \Theta(\boldsymbol{\kappa}, \boldsymbol{\theta} _ s(\mathbf{x})) _ I $$,

where $\boldsymbol{\theta} _ s(\mathbf{x})$ is the probability features at position $\mathbf{x}$ from 3d feature planes.

Additionally, we provide a more detailed figure to illustrate latent feature distribution in Fig.1(a) of the Attached PDF and the learning process of the local deformable field from 0 to 1000 training steps in the Attached PDF Fig.4. Please refer to the PDF for details.

Thank you for the constructive and voluntary review suggestions in both methodology and writing. We have carefully reviewed and corrected the entire manuscript, striving to eliminate any organizational and typos issues. Sincerely hope the proposed method and dataset in this paper will contribute to the field of interactive scene reconstruction.

#Q1. The definition and determinant of disjoint regions 𝑅. How to determine the number of subregions?

As shown in Fig.3(b) of the manuscript, the basic idea of disjoint regions is that we divide a complex interactive space into regions, namely disjoint regions 𝑅, each with an independent local deformable field, where interactions within the local field are manipulated through interaction variables. As the reviewer pointed out, jointly optimizing to form regions is challenging; hence, we utilize mask supervision and focal loss $\mathcal{L}_\text{focus}$ in Eq.7 to segment regions during training.

$\mathcal{L} _ \text{focus} = \beta \cdot \left(1 - e^{\sum _ {i=1}^{\alpha} \tilde{\mathbf{y}} _ {m}^i \log(\mathbf{q} _ i)}\right)^{\gamma} \cdot \left(- \sum _ {i=1}^{\alpha} \tilde{\mathbf{y}} _ {m}^i \log(\mathbf{q} _ i)\right)$,

where $\tilde{\mathbf{y}}_{m}^i$ is the ground truth label, $\mathbf{q}_i$ is the predicted probability $\Theta(\boldsymbol{\kappa}, \boldsymbol{\theta}_s(\mathbf{x}))$ rendering from the interactive probability field, $\beta$ is the balancing factor, and $\gamma$ is the focusing parameter.

The number, shape, and density of regions are gradually established in training by maximizing the probability distribution outputs of the interaction probability decoder shown in manuscript Fig.2. We provide additional experiments in Fig.4 of the Attached PDF to illustrate the learning process of disjoint regions from 0 to 1000 training steps. The results demonstrate a clear trend that, as training advances, the proposed method is able to progressively converge to the vicinity of the interactive objects, thereby establishing interactive regions. Furthermore, we provide interaction feature visualization of x-$\boldsymbol{\kappa}$, y-$\boldsymbol{\kappa}$, and z-$\boldsymbol{\kappa}$ in Fig.1(a) of Attached PDF to illustrate latent feature distribution. It can be seen that the features are clustered around the spatial coordinates of interactive objects, corresponding to the disjoint regions 𝑅.

#Q2. What does Θ in Eq.3 mean? Presentation of notations and paper organization.

$\Theta$ in Eq.3 represents the projection operation in Eq.2 to map the interaction variables $\boldsymbol{\kappa}$ to the most probable cluster region. It is implemented with the interaction probability decoder (MLP with the plane feature) in Fig.2 of the manuscript. In addition, all the region is split by the probability distribution yielded with the only one interaction probability decoder according to Eq.3. The interaction probability decoder is primarily trained with the constraints of the mask (focal loss $\mathcal{L} _ \text{focus}$ in Eq.7) and RGB (rendering loss $\mathcal{L} _ \text{MSE}$ in Eq.6). We have carefully revised our manuscript to clarify the notations.

#Q3. Will there be multiple objects in each data sample?

Yes. Our method and dataset are designed for complex scenes containing multiple objects. Please refer to the anonymous link in supplemental Sec.C and Tab.5 for details.

#Q4. Any priors were added to the implemented baselines?

Yes, we introduce extensions of K-Planes[1], namely MK-Planes and MK-Planes*, which enable their control capabilities by generalizing them from $C^2_4$ planes to $C^2_{3+α}$ and $3+3α$ planes, as elaborated in Sec.5 of the manuscript. In addition, CoGS[2] is able to control multiple object elements according to the original paper. We emailed authors and extended the CoGS based on Deformable Gaussian[3] to tackle multiple interactive objects. We’ll clarify this in the revised version.

#Q5. Future explorations on extending it to 3D meshes or simulated environments.

Thanks. Building interactive simulations from real-world scenarios holds great promise, particularly in Embodied AI. In the future, we will further explore this area, including explicit scene representation such as 3DGS and Mesh, as well as interactive generation, .etc.

Reference

[1] K-Planes: Explicit Radiance Fields in Space, Time, and Appearance, CVPR2023

[2] CoGS: Controllable Gaussian Splatting, CVPR2024

[3] Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction, CVPR2024

Thank the authors for the clarifications. Now that it's been clear about the formulations, I'm willing to increase my scores to positive hoping that the authors could refine the manuscript in future iterations.

#Q1. How to cope with time variations? Are the time dimension encoded within control variables?

Yes. The timestep variables, 3D features, and interaction variables are hybridized and fed into the interaction probability decoder to yield the probability distribution in implementation, as shown in Fig.2 of the manuscript. It’s critical to distinguish between 4D reconstruction and our interactive scene modeling. Unlike 4D reconstruction, where all properties synchronously vary over time, interactive scene modeling involves independent changes to individual objects, which is precisely why we require additional $\alpha$ dimensions to model the scene. Hence, each object forms its own local 4D scene, where interactions within a local scene are manipulated through interaction variables $\boldsymbol{\kappa}$.

#Q2. Potential maximum number of objects and limitations within the scene.

In the following table and Fig.1(b) of the Attached PDF, we validate the performance of LiveScene in scenarios with up to 10 complex interactive objects. Notably, our method demonstrates robustness in rendering quality, which does not degrade significantly as the object number increases. The number of objects is not a major limiting and our method is still feasible as long as the dataset provides mask and control variable labels. In contrast, the occlusion and topological complexity between objects do affect the reconstruction results, which will be discussed in the limitations section.

#Q3. Is it possible to control more complex or fine-grained objects? The interpolation and generalization capability of the interaction-aware feature.

Yes. In Fig.1(c) of the Attached PDF, we demonstrate the fine-grained control capability of LiveScene on a refrigerator and cabinet dataset without part-based labels. Our method can control a part of the object even though there are no individual part-based interaction variable labels. However, the effect is not entirely satisfactory, due to the lack of labels and CLIP's limited understanding of spatial relationships.

Additionally, we conducted an experiment to examine the method's interpolation capability for unseen yet correlated states. As shown in Fig.2 of the Attached PDF, we mask the camera view and control variable labels from 100% to 30% to increase the unseen yet correlated states. The results show that our method achieves good interpolation and generalization performance, as the image rendering quality remains stable between 100% and 40%. However, the algorithm deteriorates and causes artifacts, as the perspective and labels are increasingly missing in the last column of the table and picture in Fig.2 of the Attached PDF. Hence, the proposed feature interpolation can only maintain interaction and view consistency but does not completely address the extreme view missing issue. We’ll clarify this in the limitation section of the revised version.

#Q4. GT labels during inference control and performance degradations without GT interaction values.

The interactive variable labels are only required during training, not during inference control. As shown in Fig.2 of the Attached PDF, the decrease in interactive variable density only starts to have an adverse impact once it reaches a certain threshold (40%~30%), which has been clarified in the limitation Sec.6 of the manuscript. Additionally, we provide an experiment in Fig.3 of the Attached PDF which compares the rendering results with and without GT interaction variables. Note that we only mask the GT interaction variables but provide the supervision of RGB in both setting. According to the results, even without the interaction variable supervision, the proposed method can still achieve satisfactory rendering quality (31.45 vs 31.58 in PSNR) but loses the ability to control. Specifically, our method is unable to open the dishwasher and degenerates into 4D reconstruction without any interaction variable supervision. We have carefully revised our manuscript and will provide more detailed experiments to illustrate.

#Q5. More diverse demos to highlight and more clarifications on details of the paper.

Thanks for your kind comment. We have provided a project page in supplemental Sec.C with an anonymous link to demonstrate the interactive scene reconstruction and multimodal control capabilities. Please refer to the anonymous link for more information. Besides, we have carefully revised our manuscript to eliminate all typos and presentation issues.