DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation

Dear Reviewer ovuc,

Many thanks for your valuable comments and questions. We hereby address your concerns as follows.

---

$W1$ : Limited performance gain.

• RVT-2 attains high performance partly due to its two-stage prediction strategy, while our current framework adopts a simpler single-stage design. For a fair comparison, our results should be viewed against other single-stage baselines. Even so, our method surpasses RVT-2 under comparable conditions, indicating strong effectiveness. Importantly, our architecture is fully compatible with a two-stage extension like RVT-2, which could further increase performance if desired. To verify this, we additionally implemented a two-stage variant of our framework and evaluated it on 18 RLBench tasks, where it achieved clear improvements over the previous SOTA.

  	Avg. S.R. $\uparrow$
  RVT	62.9
  RVT-2	81.4
  DynaRend	83.2
  DynaRend-2	87.8

• The RLBench 18-task suite contains relatively simple tasks, where many methods already approach saturation, leaving limited headroom for large absolute gains. Following prior work, we train on the PerAct-generated dataset (100 demos/task), which inevitably omits some task variations and constrains achievable performance. To better measure generalization, we evaluate on all executable RLBench tasks(71 tasks) and compare against RVT-2 variants with different pretraining strategies. In this broader setting, our method improves success rate by 8.2%, underscoring its robustness beyond the standard benchmark.

$W2$ : Generalizability in out-of-distribution cases.

Overfitting in imitation learning often arises when policies memorize scene layouts or latch onto spurious correlations, especially given noisy action labels. This results in poor transfer to novel conditions. Our approach addresses this by using self-supervised pretraining via 3D future prediction, enabling the model to capture how the environment evolves and to develop dynamics-aware features that support inverse dynamics reasoning. These features mitigate reliance on brittle visual-to-action mappings.

To evaluate generalization, we adopt the Colosseum[1] benchmark, which is explicitly designed to evaluate robot policies under 12 distinct types of out-of-distribution environment perturbations, such as object size, unseen distractors, lighting shifts, novel textures and so on. Each perturbation type targets a different failure mode in manipulation policies, making the benchmark a comprehensive proxy for real-world distribution shifts. Across all perturbation types, our method achieves an average success rate improvement of 18.2% over the previous SOTA, 3D-MVP (in Fig. 4).

Additionally, To further validate robustness in real-world deployment, we conduct experiments under a challenging setting where novel, unseen distractor objects are introduced into the workspace during policy execution. Under this setting, the success rate of RVT-2 drops sharply by 57%, indicating a strong reliance on memorized scene layouts or specific visual cues. In contrast, our policy—initialized with 3D-aware masked future prediction pretraining—shows only a 21% drop in success rate.

These results confirm stronger robustness to both synthetic and real-world distribution shifts. In future work, we plan to pretrain on large-scale multi-view datasets (e.g., ScanNet[2], DROID[3]) to further expand coverage of real-world variability.

$W3$ : Reliance on extensive camera viewpoints.

We appreciate the reviewer for the comment but respectfully note that our method does not require many cameras for deployment. In real-world experiments, we use only two fixed cameras and outperform RVT-2 under the same configuration. Four cameras are used only in simulation for fair comparison with prior work. Our method also remains effective with single-view setups. As shown below, performance drops only slightly when using a single camera.

Table: Real-world performance.

Table: Real-world performance with additional distractors.

Crucially, our claim of practicality refers specifically to prior rendering-based pretraining methods (e.g., GNFactor), which rely on rendering large amounts of novel views in simulation—a process that is infeasible in the real world. Instead, we leverage a visual-conditioned diffusion model to generate novel views from limited input views, which substantially reduces the need for dense multi-view data and makes rendering-based pretraining applicable in real-world settings.

$W4$ : Action engagement in predictive network.

While we do not feed explicit actions into the predictive network, it is designed to learn dynamics-aware features directly from visual evolution. Introducing action inputs during pretraining risks information leakage, potentially causing the model to overfit to specific action-conditioned cues rather than learning transferable spatial-temporal structure.

Robotic actions are inherently multi-modal; different actions can yield diverse future states. To control this uncertainty, we predict semantically meaningful keyframes (following PerAct) rather than fixed-interval future frames. These keyframes correspond to stable intermediate states (e.g., pre-grasp and post-grasp), enabling the network to focus on informative state transitions that align with decision points, rather than noisy intermediate frames.

$W5$ : Novelty of DynaRend.

1. DynaRend introduces a new paradigm for rendering-based representation learning in robotic manipulation by jointly modeling spatial geometry and future dynamics within an explicit 3D triplane representation. Unlike prior approaches (e.g., SNeRL, RVT) that focus solely on static scene representation, DynaRend couples masked reconstruction with future prediction through differentiable volumetric rendering. This 3D-aware future-prediction objective is directly aligned with the demands of manipulation tasks, where reasoning about how the scene will change after an action is as critical as perceiving its current structure.

2. DynaRend integrates visual-conditioned diffusion-based view synthesis (See3D) into the rendering-based pretraining pipeline. This design enables high-quality novel-view generation from a minimal number of real-world cameras, eliminating the dependence on dense multi-view rigs or simulator-rendered images that limit prior methods’ applicability outside simulation. By reducing the hardware and data requirements, DynaRend makes 3D rendering-based pretraining practically deployable in real-world robotic systems.

3. Extensive evaluations on RLBench, the Colosseum benchmark (12 perturbation types), and six diverse real-world manipulation tasks consistently show that our approach yields higher success rates and substantially improved robustness to appearance, geometry, and viewpoint shifts compared to state-of-the-art baselines. Notably, on Colosseum, DynaRend improves over the previous best (3D-MVP) by 18.2%, and in real-world unseen distractor tests, our policy’s performance drops by only 21%, compared to a 57% drop for RVT-2. These results confirm that the combination of 3D-aware masked future prediction and diffusion-based view augmentation provides a meaningful and transferable advance in representation learning for robot manipulation.

---

Thank you again for your constructive and detailed feedback. We hope our responses have addressed your concerns, and we will incorporate the above clarifications and improvements in the final version. Please let us know if you have further questions — we will be actively available throughout the rebuttal period and would greatly appreciate your reconsideration of our work.

references:
[1] THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation, 2024.
[2] DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, 2024.
[3] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations, 2025.

Dear Reviewer xmna,

Many thanks for your valuable comments and questions. We hereby address your concerns as follows.

---

$W1$ : Justification of future prediction.

• The 3D-aware future prediction component in our pretraining is a core element of our method, intended to learn dynamics-informed representations rather than purely static scene encodings. In robotic manipulation, the ability to anticipate future states inherently captures future dynamics and the consequences of interaction. By predicting such future keyframes during pretraining, we obtain features that support inverse dynamics reasoning during downstream policy learning.
• While prior works such as VPP have shown the value of 2D future prediction for improving policies, our framework extends this to 3D by jointly modeling scene geometry and temporal evolution using triplane-based prediction with volumetric rendering. This results in a unified representation that captures both the spatial structure of the scene and how it changes over time. In our ablation (Table 3), including future prediction yields a 4.3% higher success rate compared to static reconstruction alone, confirming its effectiveness.

We will further elaborate in the final version on how future prediction specifically benefits downstream performance.

W2 & Q3 : End-to-end and continuous control.

We agree that the current implementation executes predicted keyframes via an external motion planner, and thus does not yet achieve fully end-to-end trajectory generation. However, the learned triplane representation itself imposes no such restriction. It can readily serve as input to end-to-end control frameworks such as Diffusion Policy[1] or Trajectory Autoregressive Models[2], enabling direct generation of continuous action sequences conditioned on our 3D-aware, dynamics-informed features. We view integrating such models into our framework as a natural next step to achieve continuous, planner-free execution.

$Q1$ : Temporal prediction horizon.

Thank you for raising this question. We would like to clarify that we do not define the future keyframe based on a fixed temporal horizon. Instead, following prior work (PerAct), we adopt a heuristic-based segmentation of the trajectory: (1) Joint velocities are near zero, and (2) The gripper’s open/close state remains unchanged. This procedure partitions the end-effector motion into smooth and coherent sub-actions, each representing a motion primitive. This segmentation reduces learning complexity by focusing prediction on meaningful transitions rather than low-level time-stepped actions. Moreover, it removes the need to tune a predefined horizon, improving robustness across diverse task durations and speeds.

$Q2$ : Performance when view augmentation fails.

We would like to clarify that view augmentation (via See3D) is applied only during pretraining to compensate for the limited and fixed viewpoints in real-world data. During downstream fine-tuning and inference, no view synthesis is used. By introducing spatially consistent RGB-D novel views into pretraining, we enhance viewpoint invariance and prevent overfitting to specific camera configurations. Importantly, this process does not rely on semantic labels; instead, the model distills high-level semantics from pretrained visual foundation models. Our ablations (Table 3) show a 3.4% downstream performance gain with view augmentation.

We acknowledge that synthesizing high-quality novel views becomes more challenging with large viewpoint shifts. To ensure fidelity, we restrict augmentation to within ±30° on a spherical surface (Supplementary L43). Qualitative examples are provided in Supplementary Fig. C and Fig. E. To quantify view quality:

1. In simulation: We compare 1000 See3D-generated views with ground-truth novel renderings from the simulator, reporting metrics in the following table.

   Max angle	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
   ±15°	26.58	0.842	0.155
   ±30°	24.26	0.776	0.193
   ±45°	20.43	0.714	0.258

2. In real-world: As ground-truth is unavailable, we perform a human evaluation with five independent raters scoring 200 generated views on a 0–4 scale (0: very poor – 4: excellent), obtaining an average score of 3.27, indicating satisfactory perceptual quality.

---

Thank you again for your constructive and detailed feedback. We hope our responses have addressed your concerns, and we will incorporate the above clarifications and improvements in the final version. Please let us know if you have further questions — we will be actively available throughout the rebuttal period and would greatly appreciate your reconsideration of our work.

references:
[1] Visuomotor Policy Learning via Action Diffusion, 2023.
[2] Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation, 2025.

Dear Reviewer Jtz2,

Many thanks for your valuable comments and questions. We hereby address your concerns as follows.

---

W1 & Q1 : Unclear motivation and positioning.

We respectfully believe there may be a misunderstanding regarding the nature of our approach. Our framework is explicitly designed around a two-stage pipeline:

1. Pretraining on multi-view robot data without actions, where we learn a unified 3D representation that captures both spatial geometry and future dynamics via masked reconstruction and future prediction with volumetric rendering.
2. Policy fine-tuning with actions, where the policy network predicts actions from these pretrained triplane features, effectively performing inverse dynamics reasoning.

This predictive pretraining significantly benefits downstream policy learning, as shown by our ablations (Table 3), where pretraining improves success rate by 8.4% compared to training from scratch.

Our method is fundamentally different from 2D pretraining approaches such as MVP[1] or VPP[2], which operate on images or videos. In contrast, our pretraining learns 3D-aware triplane representations that explicitly encode scene geometry and dynamics. Moreover, our view augmentation strategy enables rendering-based pretraining even with limited camera viewpoints, making the approach practical in real-world setups. Looking forward, we plan to extend this framework to larger-scale multi-view or robotic datasets such as DROID[3] and ScanNet[4].

$Q2$ : Limited performance gain.

We note that RVT-2 employs a two-stage prediction strategy, which gives it a structural advantage over single-stage baselines like RVT. Our method, while remaining single-stage, still outperforms RVT-2, showing strong effectiveness despite the absence of this extra stage. Our framework could incorporate a similar two-stage design to further improve fine-grained manipulation performance, as confirmed by our additional two-stage experiment in the following table.

We also highlight that performance gains on the standard 18-task RLBench benchmark are inherently limited due to the relative simplicity of the tasks and the potential upper bound in achievable success rate. Following prior work, we use datasets generated by PerAct, which include only 100 demonstrations per task. This limited coverage makes it difficult to capture all intra-task variations, naturally constraining improvement margins. To better assess generality, we evaluate our method on all executable RLBench tasks(71 tasks), comparing against RVT-2 variants trained with different pretraining strategies. These results (Table 2) show that our method achieves an improvement of 8.2%, confirming the robustness and scalability of our approach beyond the standard benchmark setting.

$Q3$ : Insufficient real-world experimental evidence.

We would like to clarify that our current real-world experiments already provide strong empirical validation. Specifically, we pretrain on multi-view RGB-D data without actions, then fine-tune on human demonstrations. Compared to RVT-2, our method achieves a 20% absolute performance gain in success rate.

We also evaluate robustness under real-world perturbations by adding unseen distractor objects. Under these conditions, RVT-2 suffers a 57% performance drop, whereas our method shows only a 21% drop, indicating significantly better generalization. This improvement is largely due to our view augmentation strategy, which synthesizes spatially consistent novel views from a small set of fixed cameras, providing diverse and semantically consistent supervision signals during pretraining.

We perform additional real-world comparisons between a policy trained from scratch and one initialized with our pretrained features, keeping the architecture and demonstrations identical. As shown below, the pretrained variant consistently outperforms the scratch-trained variant, particularly in scenarios with unseen distractors, confirming the value of our pretraining pipeline in practical deployment.

Table: Real-world performance.

Table: Real-world performance with additional distractors.

$Q4$ : Lack of detailed precision demonstrations.

We agree that our current work does not emphasize high-precision manipulation demonstrations, as our main objective is to improve generalization across diverse tasks via transferable triplane representations. While RVT-2 achieves high precision partly due to its two-stage strategy, our single-stage framework could be extended with a similar design for fine-grained control if needed. However, we view such precision-focused extensions as complementary to our primary contribution, which lies in representation learning for multi-task policy improvement rather than optimization for specific precision-oriented tasks. To further address this point, we additionally implemented a two-stage prediction variant of our framework and evaluated it on three precision-oriented manipulation tasks. Results show that with the two-stage extension, our method surpasses the previous SOTA by a clear margin, highlighting its potential for high-precision applications.

---

Thank you again for your constructive and detailed feedback. We hope our responses have addressed your concerns, and we will incorporate the above clarifications and improvements in the final version. Please let us know if you have further questions — we will be actively available throughout the rebuttal period and would greatly appreciate your reconsideration of our work.

references:
[1] Masked Visual Pre-training for Motor Control, 2022.
[2] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations, 2025.
[3] DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, 2024.
[4] Richly-annotated 3D Reconstructions of Indoor Scenes, 2017.

Thanks for the further clarification. I now better understand the core issue that led to my initial review comment on “Unclear Motivation and Positioning.” In the context of representation learning, pretraining is commonly done on out-of-domain data — most notably using in-the-wild datasets like MVP or 3D-MVP. In contrast, RVT-2 only trains on in-domain action data, making it essentially a multi-task reinforcement learning policy.

This paper proposes something in between: pretraining on in-domain actionless data followed by fine-tuning on in-domain action data. While this framework is indeed interesting and has the potential to extend to in-the-wild data, the current setup introduces ambiguity in the experimental setting. Specifically, it complicates the fairness and clarity of comparisons, at least from my understanding. If I get something wrong, please point out.

So my key question is: what was the reason behind not pretraining on out-of-domain data from the start? Understanding this design choice would help clarify the motivation and better position the contribution relative to existing work.

Dear Reviewer KFSY,

Many thanks for your valuable comments and questions. We hereby address your concerns as follows.

---

$W1$ : Limited real-robot experiments.

We appreciate the reviewer’s feedback on the real-robot evaluation. Following the suggestion, we have conducted additional real-world experiments and expanded the set of baseline methods for comparison, including RVT, 3D Diffuser Actor and an ablated variant of our method without pretraining. To ensure a fair comparison, all methods were trained on the same set of demonstration data and evaluated under the same hardware setup. As shown in the following table, our approach achieves a 20% absolute improvement in average success rate over the strongest baseline and significantly outperforms the no-pretraining variant by 19%, highlighting the critical role of the pretraining in capturing 3D geometry and dynamics for real-world manipulation. Notably, our method shows particularly strong performance in the presence of unseen distractor objects, where it maintains high success rates while other methods degrade substantially, demonstrating its robustness to real-world scene variations.

Table: Real-world performance.

Table: Real-world performance with additional distractors.

$Q1$ : Moving camera calibration.

Thank you for raising this point. While our real-world experiments utilize fixed cameras for simplicity, the proposed DynaRend framework is fully compatible with moving camera setups commonly used in robotics, such as wrist-mounted or head-mounted configurations. In such cases, standard eye-in-hand calibration techniques can be applied to estimate the transformation between the camera and the robot link to which it is attached. Given this transformation and the robot’s joint state, the camera pose can be accurately computed at any time. We will clarify this general applicability in the final version to emphasize that the framework is not limited to static-camera deployments.

Q2 & Limitation : Explicit geometry and collision avoidance.

We appreciate the reviewer’s thoughtful observation. Our current policy predicts only the next best end-effector pose and gripper state via behavior cloning, and does not explicitly encode the full-body geometry of the robot arm or perform collision checking during policy inference. To ensure safe execution in practice, we rely on sampling-based motion planners such as RRT* at deployment time to generate collision-free trajectories from the current arm configuration to the predicted end-effector pose.

While this strategy works reliably in most scenarios, we acknowledge that it can occasionally fail in cluttered scenes if the motion planner cannot find a valid path within the allocated time. We recognize this as a limitation of the current design. Looking forward, we believe this can be addressed by integrating our method with end-to-end motion generation frameworks[1][2] that account for full-body kinematics, or by incorporating explicit 3D scene geometry representations such as occupancy grids. These directions would enable collision-aware action generation without relying on an external planner, and we will mention them explicitly in the discussion of future work.

$Q3$ : Clarify the use of DINOv2.

Thank you for raising this question. In our framework, we leverage features from vision foundation models such as DINOv2 and CLIP as additional supervision signals during rendering-based pretraining, with the goal of distilling high-level semantic information into our learned representation. Since the choice of foundation model is not the primary focus of our work, we adopt a simplified approach by using RADIO[3] as the teacher network. RADIO provides a unified distillation interface by aggregating features from multiple foundation models (including DINOv2 and CLIP) into a single representation. We will revise the manuscript to make this design choice clearer.

$Q4$ : Clarify how language condition is integrated.

We apologize for the confusion. As described in the method section, language instructions are incorporated during pretraining by concatenating CLIP-based language embeddings with the triplane features, which are then fed into both the reconstructive and predictive networks. In Figure 2, we omitted the language branch for visual clarity and to better highlight the core architectural components. We will update the figure in the final version to explicitly depict the language-conditioning pathway.

---

Thank you again for your constructive and detailed feedback. We hope our responses have addressed your concerns, and we will incorporate the above clarifications and improvements in the final version. Please let us know if you have further questions — we will be actively available throughout the rebuttal period and would greatly appreciate your reconsideration of our work.

references:
[1] Visuomotor Policy Learning via Action Diffusion, 2023.
[2] Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation, 2025.
[3] AM-RADIO: Agglomerative Vision Foundation Model - Reduce All Domains Into One, 2024.