Neural Atlas Graphs for Dynamic Scene Decomposition and Editing

We appreciate the reviewer's comprehensive and constructive assessment of our paper.

1. Distinguishing Originality

The reviewer sought clarification on the originality of our method, specifically asking whether it is simply a combination of existing approaches or includes novel contributions, and requested originalty details on stated equations.

We introduce NAGs as a novel hybrid scene representation method, positioned between Neural Scene Graphs (objects collectively form a scene graph structure) and Neural Atlases. The new representation yields an editable atlas specifically designed for video editing. While our method is inspired by various concepts in the field, our core contribution is a new representation and learning method (see Q2-Q4 below): to the best of our knowledge, there does not exist a method that models dynamic 3D scenes using 2D planar fields placed on 3D rigid motion paths in the manner we propose.

To explicitly clarify, if parts of equations are derived from others in the literature, we have explicitly cited their source directly along the equation:

• Eq. 1: Describes the color and alpha formation for our novel method's rendering process.

• Eq. 2: Represents known alpha-compositing, which is standard in neural rendering (e.g., NeRFs and 3DGS), and we have explicitly cited the work of [17] for reference.

• Eq. 3 and 4: Describe our dedicated spline-based rotation model for objects. While inspired by [6], [6] primarily focused on static binary 2D layer separation and modeled smooth camera movements, whereas our formulation is specifically adapted for per-object dynamic rigid transformations.

• Eq. 5 and 6: Detail the color and alpha formation for every planar node. Their form, utilizing differently parameterized and regularized neural fields (one purely position-dependent, the other both position- and view-dependent) and their unique interplay, is, to the best of our knowledge, unique to our method.

• Eq. 7: Our flow field parameterization is also inspired by [6]; however, in our method, the flow is uniquely parameterized for every object and learned within its own frame of reference, whereas [6] applies optical flow in the camera coordinate system.

2. Missing Training Details

The reviewer noted that key training details are missing. Due to NeurIPS paper size limitations, we had to move all detailed training specifics into the Experimental Details (Sec. 2.) of our supplement, as also noted within our main paper. We tried to cover all information within this dedicated section, and further have uploaded all code in an anonymous github repository (Sec. 4.1 Ln. 245) to circumvent any missing details.

3. Comparison with Atlas-based Methods

The reviewer remarked that the experiments were missing atlas-based approaches. We would like to clarify that we compare against established video editing and matting methods, addressing this point directly. As reported in Table 3 of our main paper, NAGs demonstrate significant performance advantages (e.g., a 7.3 dB PSNR improvement) over these baselines, specifically Layered Neural Atlases [3] (LNA)—a recent texture editing method—and OmnimatteRF [4] (ORF)—a SOTA matting approach. This validates NAGs' robust decomposition and reconstruction capabilities even without relying on explicit 3D supervision, while simultaneously enabling flexible editing. The evaluation confirms that NAGs offer substantial benefits in terms of consistency, object decomposition, and general video editing over existing methods.

To further improve clarity, we will add a paragraph elaborating on these recent video matting methods and their specific usage.

Questions

Q1

For the Waymo dataset, we rely on the camera poses provided directly by the dataset, consistent with the handling by our baselines. For the DAVIS dataset, we leverage RodynRF [7] for camera pose estimation, also consistent with our baseline OmnimatteRF [4]. Further comprehensive details regarding our camera modeling approach can also be found in the Method Details section (Section 1.1) in our supplement.

Q2

All objects are used to initialize the system, as each object has its dedicated neural fields. We leverage the input masks to initially project the color and alpha information onto our base color and alpha estimates for each object. For every object (excluding the global background), we establish its initial 3D trajectory, which will be further optimized. For the Waymo dataset, this initialization utilizes the provided 3D bounding boxes, similar to [1]. For scenes where 3D bounding boxes are not available (e.g., DAVIS), object positions are estimated based on image-to-image homographies derived from sequential frames and an initial monocular depth estimation.

Q3

Our training strategy employs a multi-phase optimization approach combined with a coarse-to-fine learning strategy to effectively optimize the various components of each node. The training is subdivided into 3 phases over 80 epochs:

• 1. (Epochs 0-5): Only the positional parameters ($\mathcal{P}^{T}\_i$,$\mathcal{P}^R\_i$,$\mathcal{P}^{T}\_{cam}$,$\mathcal{P}^R\_{cam}$) are optimized to correct for initial positional errors of objects and the camera.
• 2. (Epochs 5-20): The color and opacity fields ( $\mathcal{F}\_{i, c}$,$\mathcal{F}\_{i, \alpha}$) are additionally optimized, alongside the positional parameters.
• 3. (Epochs 20+): All parameters ($\mathcal{P}^{T}\_i$,$\mathcal{P}^R\_i$,$\mathcal{P}^{T}\_{cam}$,$\mathcal{P}^R\_{cam}$,$\mathcal{F}\_{i, c}$,$\mathcal{F}\_{i, \alpha}$,$\mathcal{F}\_{i, f}$,$\mathcal{F}\_{i, \phi}$) —are optimized together. Now additionally including flow, and view-dependent fields.

In conjunction with these phases, we apply a coarse-to-fine learning strategy by masking encoding dimensions for the view-dependent and planar flow fields, gradually activating more encoding dimensions as training progresses. This approach limits the initial expressiveness of the view-dependent field and encourages the planar flow field to learn coarse alignments first, before enabling finer details and view dependencies to emerge, thus guiding the overall optimization process. Further Details can be found in the supplementary in Sec. 1.2 and 2.2.

Q4

Our model distinguishes between fixed initial conditions and learnable neural fields:

• Fixed Initial Conditions: $\tilde{C}_i$ (base color) and $\tilde{A}_i$ ​ (base alpha) are not learnable parameters of the network. They are initialized by an initial forward projection of a reference frame onto the position-initialized plane, based on the provided masks (Sec. 3.3 Ln 181f.). This step serves to pre-condition each atlas node to a dedicated region, providing a good starting point.

• Learnable Neural Fields: The core appearance and motion are then captured by our learnable neural fields: the color field $\mathcal{F}\_{i, c}$​, opacity field $\mathcal{F}\_{i, \alpha}$, flow field $\mathcal{F}\_{i, f}$, and view-dependent field $\mathcal{F}\_{i, \phi}$​. These fields are explicitly designed and optimized to serve distinct roles:

  • The color field $\mathcal{F}\_{i, c}$ and opacity field $\mathcal{F}\_{i, \alpha}$ are primarily responsible for modeling the view-agnostic, canonical appearance of the object in its atlas space.

  • The flow field $\mathcal{F}\_{i, f}$ warps this canonical appearance across time, enabling the representation of non-rigid motion and facilitating better editability by keeping the base texture consistent.

  • The view-dependent field $\mathcal{F}\_{i, \phi}$ is designed to capture subtle view-dependent effects (e.g., specularities, reflections) that cannot be explained by the canonical appearance or flow alone.

The optimization process distinguishes between these components through our phase-based learning and coarse-to-fine learning strategy. By initially limiting the expressiveness of the view-dependent field (via sparse encoding) and progressively activating it, we implicitly regularize the model to put the majority of the information into the non-view-dependent fields ($\mathcal{F}\_{i, c}, \mathcal{F}\_{i, \alpha}, \mathcal{F}\_{i, f}$). This forces the learnable non-view-dependent components to capture the primary appearance and motion, which are then effectively mappable by the flow for enhanced editability. The view-dependent field then only incorporates additional, subtle changes, preventing conflation and ensuring a disentangled representation.

We will add these clarifications to the revised work.

References

[1] Chen, Ziyu, et al. "OmniRe: Omni Urban Scene Reconstruction." The Thirteenth International Conference on Learning Representations. 2025

[2] Yang, Jiawei, et al. "EmerNeRF: Emergent spatial-temporal scene decomposition via self-supervision." arXiv preprint arXiv:2311.02077 (2023).

[3] Kasten, Yoni, et al. "Layered neural atlases for consistent video editing." ACM Transactions on Graphics (TOG) 40.6 (2021): 1-12.

[4] Lin, Geng, et al. "Omnimatterf: Robust omnimatte with 3d background modeling." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[6] Chugunov et al. “Neural Spline Fields for Burst Image Fusion and Layer Separation”. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 25763–25773, 2024.

[7] Liu et al. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[17] Kopanas et al. “Neural Point Catacaustics for Novel-View Synthesis of Reflections”. ACM Trans. Graph., 41(6), 2022. Publisher: Association for Computing Machinery.

We sincerely thank the reviewer for the thoughtful and detailed review.

1. Number of Baselines

The reviewer considered the number of baselines low compared to prior work. We would like to clarify that we actually compared our method against four distinct baselines: OmniRe [1] (a Gaussian Splatting Method), EmerNeRF [2] (a Dynamic NeRF Method), Layered Neural Atlases [3] (a texture editing-capable method), and OmnimatteRF [4] (a recent matting method). Note that we selected baselines for which code was available and that ideally supported per-object removal, allowing for meaningful comparisons in our hybrid setting. We find that comparing across these diverse fields (Gaussian Splatting, Dynamic NeRF, Texture Editing, and Matting) provides strong empirical support for the broad applicability and competitive performance of NAGs, rather than limiting the comparison to a single sub-domain. Given that OmniRe demonstrated improved performance compared to the other five Gaussian Splatting methods, we focused our comparisons on it rather than including additional similar approaches. We will include a brief explanation of these baseline methods in the appendix to enhance clarity and completeness, similar to [1] and add additional automotive baselines within the revised version of our work.

2. Comparison of Editing Results to Prior Work / Question 3

The reviewer suggested including comparisons of editing results with prior work, specifically mentioning [1]. We do not find that OmniRe [1] states direct editing comparisons against baselines. To our knowledge, neither their main paper nor supplementary material explicitly presents quantitative or qualitative comparisons of edited scenes against other methods. While they do show their own scene decomposition in Figure 1 and a qualitative result comparison of the complete scene in Figure 4, these do not allow for a direct comparison of editing capabilities against baselines.

In our paper, we explicitly compare the visual results of scene decomposition in Figure 3 against OmniRe on both a populated car scene and an adverse weather scene, confirming that our method excels at extracting objects and produces fewer artifacts. Furthermore, we demonstrate our scene decomposition and visual quality on two DAVIS dataset scenes in Supplementary Figure 6 (and the supplementary videos) compared against recent video editing and matting methods, again showing superior visual quality.
The inherent difficulty in finding ground truth for arbitrary semantic edits (e.g., changing a car's color, adding a speed limit sign) makes direct quantitative comparisons challenging for any method in this domain.
While we acknowledge this challenge, we have computed the Temporal (frame-by-frame) LPIPS and Fréchet inception distance (FID) [5] for our editing figures to quantify their temporal consistency and overall perceptual impact, providing an objective measure where possible. We find that the consistency is comparable to the ground truth (GT) or respective baselines.

For T-LPIPS we report the mean and standard deviation over all image-pairs and objects in case of Fig. 3. For FID we report the standard deviation only for the per-object decomposition evaluation, given that it is computed as a distributional measure.

3. Evaluation on Stronger Ego Motion / Question 2.

The DAVIS dataset has more camera motion than the presented Waymo sequences, but does not contain the high rotational ego motion of some autonomous driving scenes. As suggested, we add an ablation on this category of scene below:

For foreground objects, managed by dedicated nodes, our approach achieves substantial improvements (up to 8 PSNR for Vehicles/Humans) compared to all baselines. This is primarily attributed to our robust rigid motion model, which, being independent of global camera motion, is also well-suited for large ego-motion environments. The overall image scores are comparable to OmniRe and EmerNeRF (second table). As mentioned in the Limitations section, this score is not substantially higher, as we observe more artifacts in the background, as this large motion has to be generated through learning large flow vectors based on photometric loss (rather than from explicit 3D projective geometry).

We emphasize that our proposed Neural Atlas Graph Model is fundamentally “2.5D” (planes + flow), with a large inductive bias which helps regularize solutions of low-parallax / sparsely observed scene elements – e.g., a vehicle that quickly drives through the scene while we only observe one side of it – but is not as well suited to model large 3D structures with large amounts of self-occlusion. However, we believe our findings in the work indicate that this is an interesting direction for future research, and hope our work can be extended as a vehicle/object-centric graph model (combining both 2D and 3D primitives in one render graph).

We will add a discussion for the newly added evaluation sequences to the revised work, along with a discussion on this direction of future research.

Given that we are not allowed to upload visual content in the rebuttal given the newest NeurIPS policies, we include a short text description of the added scenes:

The added sequences s-1918764220984209654 (191) are showing a drivepath into a suburban pedestrian crossing, where the ego-vehicle stops shortly, vehicles and a pedestrian are crossing, after which the ego vehicle drives further into the scene.

The sequence s-2547899409721197155 (254) shows the ego vehicle also on a pedestrian crossing, where a pedestrian with stroller and vehicles passing by, after which the car performs a right turn and drives into the scene, faced by a streetcar.

4. Reliance on Coarse Masks & Sensitivity

The reviewer inquired about the method's reliance on and sensitivity to coarse masks. We initialize each node's scale using the largest mask observed across all frames, incorporating an empirically found 80% margin. This margin is specifically designed to encompass object-associated effects such as shadows and reflections, which typically extend beyond the immediate object silhouette and are of low occupancy. The approach preconditions the node's spatial extent, but ensures only weak reliance on the pixel-level quality of individual frame masks for precise sizing. To further test robustness to varying input mask qualities, we will conduct experiments on perturbed masks for the final version.

5. Clarity Suggestion

The reviewer suggested adding a comparison between neural atlases, neural scene graphs, and NAGs to enhance clarity and self-containment. We will do so and add a discussion of the key ideas, design choices, strengths, and weaknesses of (Layered) Neural Atlases, Neural Scene Graphs, and Neural Atlas Graphs (NAGs).

References

[1] Chen, Ziyu, et al. "OmniRe: Omni Urban Scene Reconstruction." The Thirteenth International Conference on Learning Representations. 2025

[2] Yang, Jiawei, et al. "EmerNeRF: Emergent spatial-temporal scene decomposition via self-supervision." arXiv preprint arXiv:2311.02077 (2023).

[3] Kasten, Yoni, et al. "Layered neural atlases for consistent video editing." ACM Transactions on Graphics (TOG) 40.6 (2021): 1-12.

[4] Lin, Geng, et al. "Omnimatterf: Robust omnimatte with 3d background modeling." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Heusel, Martin, et al. "Gans trained by a two time-scale update rule converge to a local nash equilibrium." Advances in neural information processing systems 30 (2017).

We thank reviewer HUMq for their analysis of and feedback on our work.

1. Practical Significance in Autonomous Driving and Comparison to NeRF/3DGS

The reviewer raises a point regarding the practical significance of Neural Atlas Graphs (NAG) editability in autonomous driving scenarios, especially compared to NeRF/3DGS-based approaches, which prioritize consistent 3D rendering. Our intent with NAGs is to offer a novel hybrid representation that excels in intuitive, object-centric manipulation, which is distinct from purely volumetric or splatting-based 3D reconstruction. While NeRF/3DGS methods excel at novel view synthesis and high-fidelity rendering from dense captures, their texture editability is challenging and often requires re-training or complex parameter adjustments to the implicit volume or point cloud, especially for complex non-rigid motion and semantic object manipulation.

NAGs, by explicitly decomposing the scene into editable planar objects, offer a unique advantage in applications requiring semantic understanding and direct editing of texture.

For a concrete application example in autonomous driving, we demonstrated in Figure 5 (supplementary material) a texture editing where we retextured the street to include a speed limit sign. As this texture editing interacts realistically with dynamic objects in the scene, forming realistic occlusions, such a technique could be highly valuable for generating realistic data for driving simulators or for training automotive image recognition agents. This demonstrates a practical application where explicit object manipulation and view-consistent texture changes are essential, complementing, rather than directly replacing existing 3D methods.

We will improve the manuscript to clarify this.

2. Evaluation on Stronger Ego Motion / Question 1.

The DAVIS dataset has more camera motion than the presented Waymo sequences, but does not contain the high rotational ego motion of some autonomous driving scenes. As suggested, we add an ablation on this category of scenes below:

For foreground objects, managed by dedicated nodes, our approach achieves substantial improvements (up to 8 PSNR for Vehicles/Humans) compared to all baselines. This is primarily attributed to our robust rigid motion model, which, being independent of global camera motion, is also well-suited for large ego-motion environments. The overall image scores are comparable to OmniRe (ORE) and EmerNeRF (ERF) (second table). As mentioned in the Limitations section, this score is not substantially higher, as we observe more artifacts in the background, as this large motion has to be generated through learning large flow vectors based on photometric loss (rather than from explicit 3D projective geometry).

We emphasize that our proposed NAG Model is fundamentally “2.5D” (planes + flow), with a large inductive bias which helps regularize solutions of low-parallax / sparsely observed scene elements – e.g., a vehicle that quickly drives through the scene while we only observe one side of it – but is not as well suited to model complex 3D structures with large amounts of self-occlusion. However, we believe our findings in the work indicate that this is an interesting direction for future research, and hope our work can be extended as a vehicle/object-centric graph model (combining both 2D and 3D primitives in one render graph).

We will add the evaluation of the newly introduced sequences into the revised work, along with a discussion on this direction of future research.

As pointed out by the reviewer, our main priority is not novel view synthesis, but rather in the video editing, which we address in the following point 3.

3. Comparison to Video Editing Methods

We note, that we did compare our method to established video matting methods on the DAVIS dataset. Specifically, in Table 3 of our paper, our NAG demonstrates clear performance gains (7.3 dB PSNR improvement) over these video editing baselines (Layered Neural Atlases [3] (LNA) - most recent texture editing, OmnimatteRF [4] (ORF) - most recent matting), validating our ability to achieve robust decomposition and reconstruction even in the absence of 3D supervision, while simultaneously enabling flexible editing. This evaluation also confirms that even without a primary focus on novel view synthesis, NAGs offer advantages over traditional video editing approaches in terms of consistency, decomposition and editing.

To make this clearer, we will add a paragraph in the revised work, elaborating on these recent video matting methods and their usage. Yet, we remain open to further baseline suggestions for our revised work.

4. Training Time / Question 3.

The reviewer finds the training time of 2 to 6 hours per scene on an NVIDIA L40 GPU as a potential barrier to practical adoption. We agree that shorter training times are highly desirable for practical applications. However, we would like to contextualize these times within the field of dynamic neural scene representation. Other state-of-the-art methods, such as OmniRe [1], also report comparable training durations (e.g., approximately one hour for a single scene at 960x640 resolution), higher on larger resolutions, while the proposed method runs for frames with 4x more pixels.

Furthermore, highly optimized methods like Gaussian Splatting, while achieving impressive speeds, have benefited from dedicated native CUDA implementations and extensive optimization efforts over recent years [5, 6]. Below, we provide observed training times for the full resolution (1920 x 1280 for Waymo and 1920 x 1080 for DAVIS) in all experiments:

For Waymo (W), we used a subsequence from 975 with a length of 68 frames, and for DAVIS the bear sequence with 74 frames in their respective resolution. Our training time is significantly shorter than the other atlas methods on DAVIS (D). On Waymo, we are slower than the OmniRe (ORE) [1] given its efficiency in rendering. EmerNeRF (ERF) [2] is faster, given its not a scene graph method, and does not have a dedicated model per object.

Below, we elaborate on three key components that contribute to our training time while being essential for a high-quality and editable NAG representation:

1. Extensive Ray-Casting: Necessary for high-resolution (up to 1920x1280), dynamic video learning, training involves extensive ray-casting (2.8×10$^8$ rays/epoch, 80 epochs). Time reduces for lower resolutions/shorter videos (<20 min).
2. Multi-Stage Optimization: A three-phase strategy ensures stable convergence and enhanced texture editability.
3. Per-Object Networks: Training time scales with scene complexity due to dedicated per-object/background neural networks.

Ablation studies on different parameterizations and full training details are available in the Experimental Details (Sec. 2) of our supplementary material.

Questions

Q1:
See above ("2. Evaluation on Stronger Ego Motion / Question 1.")

Q2:
As a 2.5D method, NAGs are designed to infer scene structure and motion primarily from 2D observations. Consequently, novel view synthesis capabilities are inherently bound to the range of view angles seen during the fitting process. While smaller angular and positional variations of objects or the camera are possible and typically yield good results (see Fig. 4 in Supplementary Material), very large camera displacements that are significantly 'out-of-domain' of the original video cannot be handled adequately. This is because our atlas nodes are planar in 3D space, and they mimic depth parallax implicitly via flow and view-dependent fields. Therefore, large view changes can lead to perspective errors, such as looking orthogonally onto plane edges or causing objects to disappear. This limitation is inherent to our design choice: we modeled the method as 2.5D to facilitate texture edits, rather than prioritizing novel view synthesis.

Q3:
See above ("4. Training Time / Question 3.")

References

[1] Chen, Ziyu, et al. "OmniRe: Omni Urban Scene Reconstruction." The Thirteenth International Conference on Learning Representations. 2025

[2] Yang, Jiawei, et al. "EmerNeRF: Emergent spatial-temporal scene decomposition via self-supervision." arXiv preprint arXiv:2311.02077 (2023).

[3] Kasten, Yoni, et al. "Layered neural atlases for consistent video editing." ACM Transactions on Graphics (TOG) 40.6 (2021): 1-12.

[4] Lin, Geng, et al. "Omnimatterf: Robust omnimatte with 3d background modeling." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Kerbl, Bernhard, et al. "3D Gaussian splatting for real-time radiance field rendering." ACM Trans. Graph. 42.4 (2023): 139-1.

[6] Wu, Guanjun, et al. "4d gaussian splatting for real-time dynamic scene rendering." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

We appreciate the thorough and constructive feedback provided by reviewer WUKh.

1. Quantitative Evaluation of Temporal Consistency

The reviewer suggested evaluating temporal consistency with metrics such as Temporal-LPIPS or Fréchet Video Distance (FVD). We report these scores in the following and will include them in the revised work, as we agree that this strengthens the overall quantitative evaluation:

In the above table, we report the FVD scores as the mean over the evaluated sequences and the Temporal (frame-by-frame) LPIPS, including their respective standard deviation. The scores have been calculated on the ablation sequences for Waymo (W) and blackswan, bear, and boat sequences for Davis (D). For FVD, lowest is best. We note that T-LPIPS is not a quality but rather a consistency metric, given its frame-by-frame application, and report the ground truth (GT) here as comparison. We find that our model outperforms baselines on FVD, for comparable image consistency scores, which aligns with the claims of the main work.

We would also like to clarify that we computed the inter-frame standard deviation and reported the findings in the supplementary material (Tables 2, 3, and 4) due to space constraints in the main paper. For instance, the inter-frame standard deviation of PSNR for our method is very competitive at ±1.15 (mean PSNR 41.85), compared to OmniRe's ±2.31 (mean PSNR 36.92) and EmerNeRF's ±0.65 (mean PSNR 34.93). We will highlight these statistics in the revised main text, and further emphasize the importance of temporal consistency in addition to individual frame fidelity.

2. Evaluation on Stronger Ego Motion

The DAVIS dataset has more camera motion than the presented Waymo sequences, but does not contain the high rotational ego motion of some autonomous driving scenes. As suggested, we add an ablation on this category of scene below:

For foreground objects, managed by dedicated nodes, our approach achieves substantial improvements (up to 8 PSNR for Vehicles/Humans) compared to all baselines. This is primarily attributed to our robust rigid motion model, which, being independent of global camera motion, is also well-suited for large ego-motion environments. The overall image scores are comparable to OmniRe and EmerNeRF (second table). As mentioned in the Limitations section, this score is not substantially higher, as we observe more artifacts in the background, as this large motion has to be generated through learning large flow vectors based on photometric loss (rather than from explicit 3D projective geometry).

We emphasize that our proposed Neural Atlas Graph Model is fundamentally “2.5D” (planes + flow), with a large inductive bias which helps regularize solutions of low-parallax / sparsely observed scene elements – e.g., a vehicle that quickly drives through the scene while we only observe one side of it – but is not as well suited to model complex 3D structures with large amounts of self-occlusion. However, we believe our findings in the work indicate that this is an interesting direction for future research, and hope our work can be extended as a vehicle/object-centric graph model (combining both 2D and 3D primitives in one render graph). We will add the evaluation of these sequences into the revised work, along with a discussion on this direction of future research.

Given that we are not allowed to upload visual content in the rebuttal given the newest NeurIPS policies, we include a short text description of the added scenes:

The added sequences s-1918764220984209654 (191) are showing a drivepath into a suburban pedestrian crossing, where the ego-vehicle stops shortly, vehicles and a pedestrian are crossing, after which the ego vehicle drives further into the scene.

The sequence s-2547899409721197155 (254) shows the ego vehicle also on a pedestrian crossing, where a pedestrian with stroller and vehicles passing by, after which the car performs a right turn and drives into the scene, faced by a streetcar.

3. Editing under Motion

The reviewer raised a question regarding how camera motion is handled during scene editing to ensure accurate and clean edits. Edits are projected back to the planar nodes based on one or multiple edited frames, forming an "overlay-layer" on the object plane (not the image plane), which is therefore largely independent of the camera's ego and object-motion. However, they are dependent on the viewing angle with respect to the object. Our coarse-to-fine strategy aims to learn most of the view-dependent transformations in the planar flow field, which also affects the overlay-layer, resulting in deformations that can be perceived as view-consistent editing. For example, see Fig. 4., Sup. Fig. 8, 9, or the corresponding videos in the supplementary. In the boat and blackswan scene (Sup. Fig. 8, 9), the camera and object are moving independently, yet show a visually consistent texture edit.

The frame of reference is used to enable intuitive editing in image space by robustly projecting edits to the planar domain via numerical inversion of the learned flow. This significantly simplifies the editing process compared to direct planar manipulation (as in [3]) and allows seamless integration with off-the-shelf image editing tools and improves editing comparability by making the texture flow-independent. We will further clarify this process in the revised work.

4. Quantitative Scores for Editing

Establishing ground truth for arbitrary edited frames on real data is challenging, making the quantitative evaluation of (texture) editing quality particularly difficult. This is a common issue in the field, and we observe that existing baseline methods [3, 4] also primarily rely on qualitative assessments. Nevertheless, we evaluated the Temporal (frame-by-frame) LPIPS and Fréchet inception distance (FID) [5] metrics to quantify the consistency and similarity of the decomposed objects, or edited frames versus their succeeding frames, for all our provided editing figures. We find that the consistency is comparable to the ground truth (GT).

For T-LPIPS we report the mean and standard deviation over all image-pairs and objects in case of Fig. 3. For FID we report the standard deviation only for the per-object decomposition evaluation, given that it is computed as a distributional measure. For all stated edits, our consistency results are comparable to our reference method or the respective ground truth (GT).

References

[1] Chen, Ziyu, et al. "OmniRe: Omni Urban Scene Reconstruction." The Thirteenth International Conference on Learning Representations. 2025

[2] Yang, Jiawei, et al. "EmerNeRF: Emergent spatial-temporal scene decomposition via self-supervision." arXiv preprint arXiv:2311.02077 (2023).

[3] Kasten, Yoni, et al. "Layered neural atlases for consistent video editing." ACM Transactions on Graphics (TOG) 40.6 (2021): 1-12.

[4] Lin, Geng, et al. "Omnimatterf: Robust omnimatte with 3d background modeling." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Heusel, Martin, et al. "Gans trained by a two time-scale update rule converge to a local nash equilibrium." Advances in neural information processing systems 30 (2017).

I find the review satisfactory. I would request the authors to put special focus on "Editing under Motion" use case - where the editable objects as well as the frame of reference are in motion, analysis in the supplementary. The real world use cases would strongly demand this feature. Adversarial robustness is a field where motion is important while dealing with adversaries. A detailed analysis of this direction is important for a strong real-world applicability of the research.

It is also expected that the code for this scenario be made available in the final code.